Fix Issue #2577 - Zero Division Error in diagnostics.performance_metrics() causing failed assertion #2578
Conversation
| @@ -630,6 +630,7 @@ def smape(df, w): | |||
| Dataframe with columns horizon and smape. | |||
| """ | |||
| sape = np.abs(df['y'] - df['yhat']) / ((np.abs(df['y']) + np.abs(df['yhat'])) / 2) | |||
| sape = sape.fillna(0) | |||
There was a problem hiding this comment.
are there tests covering these methods?
There was a problem hiding this comment.
There was a test looking at the mape metric underneath the wider test_performance_metrics() method, so I have extended that to test for zero values across all other metrics.
| @@ -234,13 +234,13 @@ def test_performance_metrics(self, ts_short, backend): | |||
| metrics=["coverage", "mse"], | |||
| ) | |||
| assert set(df_horizon.columns) == {"coverage", "mse", "horizon"} | |||
| # Skip MAPE | |||
| df_cv.loc[0, "y"] = 0.0 | |||
| # Handle zero y and yhat and skip MAPE | |||
There was a problem hiding this comment.
You can leave the MAPE skip test as-is and do the test for 0.0 y and 0.0 yhat afterwards.
I guess mdape doesn't have the same problem because it uses the rolling_median function?
There was a problem hiding this comment.
Updated!
Yep, rolling_median doesn't get rid of the np.nan values, they're still included with the groupby.
| @@ -236,11 +236,13 @@ def test_performance_metrics(self, ts_short, backend): | |||
| assert set(df_horizon.columns) == {"coverage", "mse", "horizon"} | |||
| # Skip MAPE | |||
| df_cv.loc[0, "y"] = 0.0 | |||
There was a problem hiding this comment.
You'll still want to keep these calls:
df_horizon = diagnostics.performance_metrics(
df_cv,
metrics=["coverage", "mape"],
)
assert set(df_horizon.columns) == {"coverage", "horizon"}
There was a problem hiding this comment.
Ah thanks, missed that! Added back in now.
This PR resolves Issue #2577
This issue is caused by zero division leading to
np.nanvalues, so this PR adds a fix by fillingnp.nanvalues with zero during the calculation of the SMAPE metric.This PR also updates the
test_performance_metrics()method in theTestPerformanceMetricsclass intests.test_diagnostics.pyto cover all metrics handling zero values foryandyhat.