The discrepancy between the exported best model and training log

[pancookie]

Hi Haifeng,

Thank you for developing this powerful autoML tool and I have been using it for a while.
And I would appreciate it a lot if we can discuss some of the issues to provide useful information for other users to refer to and to make this package better.

Background, I am using the AutoKeras, StructuredDataRegressor().

1. The best model cannot be exported?
   I checked the discussion thread and only found this could be related Get model structure from trial folder #1627.
   While my judgment is based on the following:

Please check the highlighted parts: at the trial 56 the best val_loss is 0.186, however, at epoch 1993 the MAE (mean absolute error) reached 0.0887. And for the StructuredDataRegressor the two arguments are `loss='mean_absolute_error', metrics='mean_absolute_error',`.

So, does it mean that during the search, the model has ever reached MAE as 0.0887? If so, why the best model is showing MAE much higher than 0.0887?

Besides, I read through all those 56 json files in those trial folders and cannot find any "value" even less than 0.1.

2. Is it possible to customize the search in a more "flexible" level?
   Interestingly, there is no any trail with over 3 hidden layers.
   So can we customize it, such as to let the model with exactly 4 layers?
   And to refrain the number of neurons at each layer? Such as only allowing [32, 128] neurons for each layer.

Again, I made this judgment based on the oracle.json:

Again, thanks so much and look forward to discussing it.

Regards,
Yang

[pancookie]

Alright, so after tons of time reading the code line by line.

Here is the what I have found,
the best model is saved in <AK_run_projectname>/structured_data_regressor/best_model.

Regarding the discrepancy, the logic of this algorithm is below:
The very important concept is "trial", which contains a unique ID, network hyperparameters (trial_obj.hyperparameters.values).
Each trial in the terminal progress bar stands for certain neural network architecture.
While the weights are updated and the epoch goes on as it meets the criterion of early stopping.

After the condition to stop the trial search, the best trial is selected based on the "optimal(max or min)" objective metric.
There is a final fit after the trial search, where the best trial is loaded.
After the final fit the architecture of the best trial and the updated weights are saved in /best_model.