-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Lack of reproducibility between TPOTRegressor and .fitted_pipeline_ attribute #1305
Comments
Succinctly, the problem is that you expected: I doubt what you got is intended for at least for some datasets; the first step would be consolidating your demonstration code into an automated test. def test_accuracy_scoring(...):
...
some_tpot_classifier.score() == sklearn.metrics.accuracy_score(some_tpot_classifier)
some_tpot_regressor.score() == sklearn.metrics.accuracy_score(some_tpot_regressor) |
The default scoring for TPOTRegressor is 'neg_mean_squared_error'. so tpot.score will return the neg_mean_squared_error. But you are comparing it to mean_absolute_error. If you want to optimize mean absolute error, you can pass that in as a scorer. If you change your estimator to the following, you get the same results in your example. |
It is currently not possible to reproduce the results of regression performed with the TPOTRegressor class with the resulting pipeline.
Context of the issue
Currently, the accuracy score from the .score() method of a TPOTClassifier instance and the output of sklearn.metrics.accuracy_score on the best pipeline are identical. This is not the case with pipelines from TPOTRegressor instances.
Process to reproduce the issue
Classifier (correct/reproducible results)
The following code is used to create a TPOTClassifier, train it on the iris dataset and then return the accuracy
When the sklearn.metrics.accuracy_score function is called on the y_test data and the predictions from the best pipeline created by the TPOTClassifier instance, the result is identical:
Regressor (incorrect/nonreproducible results)
With TPOTRegressor, the results are not identical.
Unfortunately, the results of rerunning the call to .predict with test data for the best pipeline from the TPOTRegressor object, are not identical to this step:
I would have expected the last line to return 2572.133297426151.
The text was updated successfully, but these errors were encountered: