How to improve the accuracy of the model

[ZKNB]

I am training the quaternionic potential function [Fe,C,O,H],I have more than 38000 single point energy data,each data has only one frame structure.I used the following two ways to train:
1, Fine-tuning training on the basis of OC_10M.pb big model
2, Training from scratch based on descriptor 'se_atten_v2'
My input file is as follows:

OC_10M.pb finetune training
{
    "model": {
        "type_embedding":{"trainable": true},
        "descriptor": {"trainable": true},
        "fitting_net": {"trainable": true},
        "type_map": [
            "H",
            "C",
            "O",
            "Fe"
        ]
    },
    "learning_rate": {
        "type": "exp",
        "start_lr": 1e-03,
        "decay_steps": 80000,
        "stop_lr": 3.51e-08
    },
    "loss": {
        "type": "ener",
        "start_pref_e": 0.02,
        "limit_pref_e": 1,
        "start_pref_f": 1000,
        "limit_pref_f": 1,
        "start_pref_v": 0,
        "limit_pref_v": 0
    },
    
    "training": {
        "training_data": {
            "systems": [
"./training_data/training_data1/44",
...it include 38000 datasets
            ],
            "batch_size": 1
        },
        "validation_data": {
            "systems": [
 "./validation_data/validation_data9859/55"
...it include 200 datasets
                ],
            "batch_size": "auto"
        },
        "numb_steps": 10000000,
        "seed": 3982377700,
        "_comment": "that's all",
        "disp_file": "lcurve.out",
        "disp_freq": 1000,
        "numb_test": 1,
        "save_freq": 10000,
        "save_ckpt": "model.ckpt",
        "disp_training": true,
        "time_training": true,
        "profiling": false,
        "profiling_file": "timeline.json"
    }
}

DPA-2 se_atten_v2 descripter
{
  "_comment1": " model parameters",
  "model": {
    "type_map": [
        "H",
        "C",
        "O",
        "Fe"
    ],
    "descriptor": {
      "type": "se_atten_v2",
      "sel": 120,
      "rcut_smth": 0.50,
      "rcut": 6.00,
      "neuron": [
        25,
        50,
        100
      ],
      "resnet_dt": false,
      "axis_neuron": 16,
      "seed": 1,
      "attn": 128,
      "attn_layer": 0,
      "attn_dotr": true,
      "attn_mask": false,
      "precision": "float64",
      "_comment2": " that's all"
    },
    "fitting_net": {
      "neuron": [
        240,
        240,
        240
      ],
      "resnet_dt": true,
      "precision": "float64",
      "seed": 1,
      "_comment3": " that's all"
    },
    "_comment4": " that's all"
  },

  "learning_rate": {
    "type": "exp",
    "decay_steps": 100000,
    "start_lr": 0.001,
    "stop_lr": 3.51e-8,
    "_comment5": "that's all"
  },

  "loss": {
    "type": "ener",
    "start_pref_e": 0.02,
    "limit_pref_e": 1,
    "start_pref_f": 1000,
    "limit_pref_f": 1,
    "start_pref_v": 0,
    "limit_pref_v": 0,
    "_comment6": " that's all"
  },

  "training": {
    "training_data": {
      "systems": [
     "./training_data/training_data1/44",
...it include 38000 datasets
      ],
      "batch_size": "auto",
      "_comment7": "that's all"
    },
    "validation_data": {
      "systems": [
     "./validation_data/validation_data9859/55"
...it include 200 datasets
      ],
      "batch_size": 1,
      "numb_btch": 3,
      "_comment8": "that's all"
    },
    "numb_steps":10000000,
    "seed": 10,
    "disp_file": "lcurve.out",
    "disp_freq": 5000,
    "save_freq": 10000,
    "_comment9": "that's all"
  },

  "_comment10": "that's all"
}

Their convergence curves are as follows：


Their test results are as follows：


I have the following questions:

1. The convergence curves can see that they RMSE can go to the order of 1e-05, but the accuracy of the test is not there, what is the reason for this?
2. The average target accuracy of energy I want to train is 1e-03 ev/atom, and the average target accuracy of force is 1e-02 ev/atom, the test found that the energy accuracy of some structures can even go up to 1e-07 ev/atom, but the overall average results are all an order of magnitude less, how should I adjust the parameters to make the model more accurate?
3. When I train the model, I found that the time of fine-tuning training is longer than the time of training from scratch. Fine-tuning training costs 45s per 1000 steps, while training from scratch costs 7s per 1000 steps, which is a difference of 6-7 times, what is the reason for this?
   Thank you for reading.I am looking forward to your replying.

[haitaier]

Have you solved your problem?If so, by what means was it resolved? Thanks.

[ZKNB]

Not yet, if you have some solutions and ideas we can discuss them together

[Guancred]

The same situation also exist for my trained potential? Have any suggestions for this ? Thanks very much!

[ZKNB]

I'm trying to change the force and energy weights right now, so if you have other ideas you can discuss them together

[haitaier]

Changing the force and energy weights probably can't help you,I have tryed that before.

[iProzd]

Hi @ZKNB, I may be able to offer some assistance on the following points:

1. The convergence curves display results from on-the-fly validation. While some samples may exhibit lower energy RMSE values like 1e-5, the average energy RMSE of the curve (located at the center of the red and green curves) hovers around 1e-2, aligning with the dp test results. This result is within the expected range.

2. Have you calculated the standard deviation of your energy and force data distribution? Additionally, have you trained other models using your data? These two analyses can provide a deeper understanding of the training difficulty associated with this potential. Because scenarios vary across different kind of data systems. To enhance training accuracy, you can consider adjusting the force and energy prefactors as you mentioned. You can also explore using more powerful architectures, such as DPA2 in PyTorch backends. You can start from scratch using the input example here or fine-tune from the recently released DPA2 multitask pretrained model accessible here. For further insights, refer to the DPA2 paper. It's worth noting that the second script you utilized, se_atten_v2, is not DPA2 but a different version of DPA1.

3. Regarding training efficiency, when utilizing fine-tuning, the settings from the pretrained model are automatically applied. In this scenario, the attn_layer parameter in OC_10M.pb is set to 2, whereas in your from-scratch script, it's set to 0. This parameter determines the number of attention layers, enhacing accuracy but potentially reducing efficiency. This difference appears to be the primary contributing factor.

I hope this information is helpful to you!

[ZKNB]

Thanks for your reply! I'm going to try the dpa2 descriptor, does deepmd v2.2.7 support that descriptor now? Do I need to install the latest deepmd version and which command should I use to install the latest deepmd version in conda environment?

[iProzd]

DPA2 is available on built-from-source installation now, you can see How to use section here , and install the 2024Q1 branch of DeePMD-kit, which is a stable and higher version than v3.0.0a.

[Guancred]

Hi, the 2024Q1 is also a gpu version?