Performance metrics when using balance_classes

[hasithjp]

Motivation

When balance_classes is used in H2O modeling, the performance metrics will not match the performance metrics constructed during training.

An example is shown below:

# Build model with balanced classes
gbm_model <- h2o.gbm(x, y, training_frame, balance_classes = TRUE)
h2o.performance(gbm_model, train = True)@metrics$AUC
[1] 0.7248632
h2o.performance(gbm_model, newdata = training_frame)@metrics$AUC
[1] 0.7248632

Solution

When balance_classes is enabled in H2O modeling, the model is built on a balanced version of the training data frame. The performance metrics constructed during training are based on this balanced version of the training data frame. Therefore, when performance is calculated on the unbalanced training data frame, the metrics will be different.

To determine the performance metrics on the unbalanced training data frame use the following:

h2o.performance(gbm_model, newdata = training_frame)

JIRA Issue Migration Info

Jira Issue: TN-9
Assignee: Megan Kurka
Reporter: Megan Kurka
State: Open