Tracking with aim, python api

[andife]

Hello,

I wonder how to track the experiments with aim using the python api?

Thank you

Andreas

[brightsparc]

Please see the integrations section of documentation that also includes a link to the aim ludwig demo: https://github.com/aimhubio/aim-ludwig-demo

[andife]

Hi, thank your for the answer, but for me it is still not quite clear how to integrate it. But the links show how to use aim with the cli
"!ludwig train --dataset final_train.csv --config train_conf.yaml -g 0 --aim --experiment_name "Classification"
I'm looking for how to use that with the python api.

Maybe adding to model.train(dataset=train_df) in (MNIST Image Classification with Ludwig - Python API.ipynb)

[brightsparc]

If you are using the python sdk you need to register the AimCallback when creating the LudwigModel:

from ludwig.api import LudwigModel
from ludwig.contribs.aim import AimCallback

callback = AimCallback()
model = LudwigModel(config="./model1_config.yaml", callbacks=[callback])
(
    train_stats,  # dictionary containing training statistics
    preprocessed_data,  # tuple Ludwig Dataset objects of pre-processed training data
    output_directory,  # location of training results stored on disk
) = model.train(
    training_set=training_set,
    test_set=test_set,
    experiment_name="simple_experiment",
    model_name="simple_model",
    skip_save_processed_input=True,  # Don't write intermediate files
)

The results are then saved to the hidden .aim folder under the results directory, which can visualize passing the repo arg like so:

aim up --repo results/simple_experiment_simple_model

Under the hood the library writes metrics like so:

from ludwig.utils.package_utils import LazyLoader
aim = LazyLoader("aim", globals(), "aim")

repo = 'results/simple_experiment_simple_model'
run = aim.Run(repo=repo, experiment="simple_experiment")
run.track(1, name='metric_name')

I think there could be value in the AimCallback not setting the repo for every run, but re-using a top level folder, so I might make a PR so that any subsequence run can then include the model_name as a run hash.

[andife]

Hi @brightsparc ,
Although I have used Aim before, and also the callback it didn't occur to me that I could have used it to solve my question. Maybe it would also be useful to add the point to the doc.

[brightsparc]

Yes, this is a good call out, I will look to see if the docs can get updated with this.