Use Neuromancer to learn the unmodeled dynamics and combine it with MPC

[gy2256]

I am fortunate enough to discover Neuromancer. The software is great and handy.

For my current project, I would like to learn the unmodeled dynamics using Neuromancer and combine it with my known nominal system dynamics (A and B matrices shown in the picture above). The \phi(x,u) represents the unmodeled system dynamics which can be captured by a Neural Network. After obtaining \phi with offline training, I will then use it to solve an MPC problem with a solver, such as Casadi. The constraints look like the following:

Is this feasible?

(I am aware of the DPC problem formulation, but it seems like it includes everything in the loss function and trains the policy network offline. If I understand it correctly, there is no optimization to be solved online, as the trained policy can be used directly to infer the desired control. For my application, I still want to solve the constrained optimizations (MPC) online but with the inferred \phi(x,u). )

[gy2256]

I have figured it out. It requires me to design the loss function to account for the discrepancy between my nominal dynamics prediction and real states. And I can save the trained model using torch.save() and then use it later in Casadi.

[drgona]

Hi @gy2256 . Yes this problem is feasible in Neuromancer.

The system identification of unmodeled dynamics conceptually falls in the gray-box modeling task with few continuous-time example in our library:
https://github.com/pnnl/neuromancer/blob/master/examples/ODEs/Part_2_param_estim_ODE.ipynb
https://github.com/pnnl/neuromancer/blob/master/examples/ODEs/Part_3_UDE.ipynb

Modification of these in the context of state space models with neural components should be straightforward.
See this example on neural state space models, where instead of black-box models for A, B, you can use pre-defined matrices as inputs to the model and learn only the unknown part \phi(u,x).
https://github.com/pnnl/neuromancer/blob/master/examples/ODEs/Part_5_nonauto_NSSM.ipynb

You can even design your custom callable to represent your model architecture. For instance, here I am using simple lambda function to instantiate state space model with fixed A, B, matrices representing my dynamics for control learning task.
https://github.com/pnnl/neuromancer/blob/master/examples/control/Part_1_stabilize_linear_system.ipynb

For the second part, correct DPC is an offline policy optimization algorithm that does not solve online optimization problem like MPC does. Instead, you obtain explicit control policy parametrizing the solution to parametric optimal control problem (explicit MPC).

For importing trained neural network in online MPC using CasADi, there is a handy library that allows you to load trained pytorch models into CasADi. We use this for designing learning-based safety filters ourselves.
https://github.com/Tim-Salzmann/l4casadi
https://github.com/TUM-AAS/ml-casadi

hope this helps.

[gy2256]

Thank you very much! I will try to design my custom callable. Also, I confirm that the output model (from Neuromancer) can be converted using l4casadi and solve the optimization with Casadi.

Additionally, it's worth noting that several techniques enable the extraction of layer parameters from a trained neural network. These methods directly apply matrix multiplication to the input data with the layer parameters, effectively conducting a 'manual forward pass' of the network. This is a handy technique when the model is being used on embedded systems.

[gy2256]

Just an update: I have successfully trained the model for \phi using a toy example. First, I created a state space model based on nominal dynamics (defined by A, B matrix)

class State_Space_Model(torch.nn.Module):
    def __init__(self, A, B, phi, nx, nu):
        super().__init__()
        self.A, self.B = A, B  # Nominal dynamics
        self.nx, self.nu = nx, nu  # Number of states and controls
        self.phi = phi  # Unkown dynamics (approximated by NN)
        self.in_features, self.out_features = nx + nu, nx  # Input and output dimensions of the NN

    def forward(self, x, u):
        x_dot = [email protected] + [email protected] + self.phi(x, u)
        return x_dot

Then, I feed the state space model into the integrators

phi = nm.blocks.MLP(n_states + n_controls, n_states, bias=True,
                       linear_map=torch.nn.Linear,
                       nonlin=torch.nn.Tanh,
                       hsizes=[64, 64, 64])

    ssm = State_Space_Model(A=torch.tensor([[0, 1, 0, 0], [0, 0, 0, 0], [0, 0, 0, 1], [0, 0, 0, 0]],
                                           dtype=torch.float32),
                            B=torch.tensor([[0, 0], [1, 0], [0, 0], [0, 1]], dtype=torch.float32),
                            phi=phi,
                            nx=n_states,
                            nu=n_controls)

    # Adjoint based solver
    fxRK4 = integrators.DiffEqIntegrator(ssm, h=dt, method='rk4')

Note that this is a bit different from my formulation above as I am learning the continuous dynamics using \phi, instead of the discrete dynamics. The final result is pretty good on the testing data: