-
-
Notifications
You must be signed in to change notification settings - Fork 130
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Further training with heterogeneous datasets #423
Comments
Specifying UPDATE: Actually pinning [
('change', ['head.layers.0.weight'], ((34, 512), (21, 512))),
('change', ['head.layers.0.bias'], ((34,), (21,)))
] (still recommend saving model as inference only as it would otherwise bundle the server-side train dataset which we don't need on clients): model_config: DictConfig = OmegaConf.load(f"{model_path}/config.yml")
datamodule: TabularDatamodule = joblib.load(f"{model_path}/datamodule.sav")
model_config.embedding_dims = copy.copy(datamodule._inferred_config.embedding_dims)
del datamodule # since we don't need it anymore |
For now you need to extends |
I guess the library was designed to make it easy for a non-DL user also and so we infer a lot of things from the data. While this is useful for many usecases, in use cases such as yours it's definitely a problem. I did introduce a low-level API to give more flexibility to users where the Also, there are ways to save and load the model (different ways). Can you check the tutorials and documentation for those methods? But in the long run, there may need some tweaks in the library code for this to work seamlessly. If you can figure out a neat way(without disrupting existing workflows), then nothing like it! |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Is your feature request related to a problem? Please describe.
Alternative title: Federated Learning usecase with the same model but different parameters shape due to training with heterogeneous datasets
I'm using Flower, and my workflow is as below:
I'm used to train on simple models with simple data preprocessing (sklearn's
StandardScaler
) and their parameters shape always remains the same regardless of the dataset. This is not the case for Pytorch Tabular as whenever I save a model trained on a dataset and reuse it to train another slightly different one i.e missing all rows of a label, both share the same layers but are not the same shape-wise.Describe the solution you'd like
I hope that model parameters shape can be consistent between training with different datasets, so that aggregation works without needing to pad model parameters with smaller shape to fit bigger ones.
Describe alternatives you've considered
Padding only works with aggregation, and while restoring parameters on clients works, training on it would cause shape mismatch as the original model parameters shape is different from the aggregated ones.
Additional context
Training with client mismatch causes:
RuntimeError: The size of tensor a (85) must match the size of tensor b (84) at non-singleton dimension 1
I used the following to restore parameters to model:
Reproducing notebook: https://colab.research.google.com/drive/11BD1XqSgc5a4k0YxxfyeFGfr0_KOUj0h?usp=sharing
The text was updated successfully, but these errors were encountered: