Implement SolverInterface for Clarabel solver

[RussTedrake]

Based on our slack discussion, we would like to move forward with adding Clarabel to our list of supported solvers. In an earlier discussion we converged on thinking that wrapping the Rust interface is the most straight-forward.
https://github.com/oxfordcontrol/Clarabel.rs

@jwnimmer-tri -- I believe you said that you were willing to give us a proof of life on the rust-bazel integration? Then @hongkai-dai and/or I can work on the solver interface.

cc @goulart-paul . (sorry for the internal slack links above! we'll track the work here moving forward)
cc @TobiaMarcucci

[goulart-paul]

Wonderful. I suggest that the cleanest approach is for us to make a generic looking FFI in either C or C++, and then call that using a wrapper similar to those that you already have here.

That's probably more maintainable than making direct calls into the compiled rust libraries. We'd like to have something like that anyway since it would other language interfaces (e.g. for Matlab for Fortran) easier to support.

[PTNobel]

@goulart-paul would it make sense to try and emulate the SCS C interface since Clarabel solves problems with the same structure? https://www.cvxgrp.org/scs/api/c.html#c-interface

[goulart-paul]

@PTNobel probably there is a lot of the SCS wrapper that could be used, but I would prefer not to directly follow the SCS C interface formatting. The specification of the cone formatting is quite rigid, i.e. conic constraints must appear in the order prescribed here.

Adopting that would mean that any FFI that we implement would have an interface that is quite different from what we provide in Rust / Python / Julia / R. All of those other interfaces allow constraints to appear in arbitrary order.

[jwnimmer-tri]

@jwnimmer-tri -- I believe you said that you were willing to give us a proof of life on the rust-bazel integration? Then @hongkai-dai and/or I can work on the solver interface.

My WIP experimenting is here https://github.com/jwnimmer-tri/drake/commits/bazel-rules-rust. However, as with any other new external, you probably don't need to block on the Bazel integration. Run the upstream build system manually, and link against the compiled library manually (something like linkopts = ["/path/to/libclarabel_rs.a"]), and then you can develop the wrapper code without waiting for me. The Bazel build rules just automate (and cache) things that can already happen by hand.

I suggest that the cleanest approach is for us to make a generic looking FFI in either C or C++, and then call that using a wrapper ...

Yes, I fully agree!

I was originally planning for us to write both parts in separately inside Drake (the FFI for Rust, and separately the drake/solvers/... wrapper), but of course having the FFI layer as part of the upstream project is a much better!

For Drake we would be happy to call into either C or C++ as the FFI. Our build system is able to link a C++ FFI in ways that are ABI-safe for redistribution (hidden, static, with mangling).

[RussTedrake]

I took a first swing at an FFI in C/C++, adding just enough bindings to port the example_lp.rs to c++:
oxfordcontrol/Clarabel.rs@main...RussTedrake:Clarabel.rs:c_ffi

This was my first time touching Rust, and I haven't polished anything yet, so there are a bunch of jagged edges. @goulart-paul , @jwnimmer-tri , @rpoyner-tri any chance you could take a look at let me know if I'm generally heading in the right direction?

To do a nicer job with it, I would probably want to actually hide the existing extern C somewhat mangled API behind a c++ wrapper -- ideally using Eigen (e.g. Eigen::SparseMatrix => CscMatrix, etc).

[RussTedrake]

Ok, I've put a cc wrapper to manage the memory around the raw memory exchanges in the FFI. (at the same link as above).

The lp example now looks pretty good to my eyes:
https://github.com/oxfordcontrol/Clarabel.rs/blob/4e850814ff7b937b6297dd94b583c2d4aeb8cd0b/examples/cc/example_lp.cc

(apart from the include path, which I'll fix with a little more work in bazel).

[goulart-paul]

@RussTedrake We started work about two weeks ago on a generic C wrapper for our Rust code with the idea that dealing directly with Rust would not be necessary for you. Pinging @gaviny82 from my group who is writing that...

He and I will talk this week to see if we can somehow make a coherent set of interfaces for you, using both what you have here and whatever he has done so far.

Apologies - probably should have let you know about progress on our end with this.

[RussTedrake]

I see. I understood that you would do it, but not until fall. But I'm keen to try it much sooner.

[goulart-paul]

I see. I understood that you would do it, but not until fall. But I'm keen to try it much sooner.

I think we will get there a lot faster than that - just didn't want to overpromise.

[jwnimmer-tri]

I've got the minimum viable Drake build system for Clarabel working now on my bazel-rules-rust branch. I still have tons of cleanup to do before we could merge the build system into master, but it might be enough for the rest of the Drake team to start writing the MathematicalProgram <=> Clarabel interface code.

Sample output:

jwnimmer@call-cps:~/jwnimmer-tri/drake$ bazel run //solvers:clarabel_example_lp
INFO: Invocation ID: 76bc3841-29a3-47f7-9582-92e94f272ddb
INFO: Analyzed target //solvers:clarabel_example_lp (26 packages loaded, 779 targets configured).
INFO: Found 1 target...
Target //solvers:clarabel_example_lp up-to-date:
  bazel-bin/solvers/clarabel_example_lp
INFO: Elapsed time: 0.362s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: external/bazel_tools/tools/test/test-setup.sh solvers/clarabel_example_lp
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //solvers:clarabel_example_lp
-----------------------------------------------------------------------------
-------------------------------------------------------------
           Clarabel.rs v0.0.0  -  Clever Acronym              

                   (c) Paul Goulart                          
                University of Oxford, 2022                   
-------------------------------------------------------------

problem:
  variables     = 2
  constraints   = 4
  nnz(P)        = 0
  nnz(A)        = 4
  cones (total) = 1
    : Nonnegative = 1,  numel = 4

settings:
  linear algebra: direct / qdldl, precision: 64 bit
  max iter = 200, time limit = Inf,  max step = 0.990
  tol_feas = 1.0e-8, tol_gap_abs = 1.0e-8, tol_gap_rel = 1.0e-8,
  static reg : on, ϵ1 = 1.0e-8, ϵ2 = 4.9e-32
  dynamic reg: on, ϵ = 1.0e-13, δ = 2.0e-7
  iter refine: on, reltol = 1.0e-13, abstol = 1.0e-12,
               max iter = 10, stop ratio = 5.0
  equilibrate: on, min_scale = 1.0e-4, max_scale = 1.0e4
               max iter = 50

iter    pcost        dcost       gap       pres      dres      k/t        μ       step      
---------------------------------------------------------------------------------------------
  0  +0.0000e+00  -6.0000e+00  6.00e+00  0.00e+00  0.00e+00  1.00e+00  1.40e+00   ------   
  1  -9.1640e-01  -2.0400e+00  1.12e+00  8.16e-17  1.02e-16  1.93e-01  2.63e-01  8.25e-01  
  2  -1.9880e+00  -2.0193e+00  1.58e-02  3.00e-17  4.10e-17  6.96e-03  7.66e-03  9.90e-01  
  3  -1.9999e+00  -2.0002e+00  1.57e-04  1.20e-16  0.00e+00  6.97e-05  7.67e-05  9.90e-01  
  4  -2.0000e+00  -2.0000e+00  1.57e-06  1.83e-13  2.20e-13  6.97e-07  7.67e-07  9.90e-01  
  5  -2.0000e+00  -2.0000e+00  1.57e-08  3.66e-15  4.39e-15  6.97e-09  7.67e-09  9.90e-01  
  6  -2.0000e+00  -2.0000e+00  1.57e-10  8.47e-17  8.20e-17  6.97e-11  7.67e-11  9.90e-01  
---------------------------------------------------------------------------------------------
Terminated with status = Solved
solve time = 71.224µs
Solution (x)	 = [-0.9999999999, 0.9999999999]
Multipliers (z)	 = [0.0000000000, 1.0000000000, 1.0000000000, 0.0000000000]
Slacks (s)	 = [1.9999999999, 0.0000000001, 0.0000000001, 1.9999999999]

[hongkai-dai]

@jwnimmer-tri thanks a lot for the branch, that is super helpful!

May I ask you a few questions?

1. Clarabel often uses enum class, for example its solver status https://github.com/oxfordcontrol/Clarabel.cpp/blob/186d06a329a65e1b0e3baaeefd027a9a3c40912a/include/cpp/DefaultSolution.h#L10-L22. I want to return the solver status in ClarabelSolverDetails struct to the user, as in

   # clarabel_solver.h
   struct ClarabelSolverDetails {
     clarabel::SolverStatus solver_status;
   };

   but this seems a bad approach as solvers/clarabel_solver.h doesn't include or expose Clarabel.h. So should I declare a drake::solvers::ClarabelSolverStatus enum class and try to keep this enum class in sync with clarabel::SolverStatus enum class?
2. Clarabel turns on semidefinite programming support if we define the compile flag https://github.com/oxfordcontrol/Clarabel.cpp/blob/186d06a329a65e1b0e3baaeefd027a9a3c40912a/include/cpp/SupportedConeT.h#L26-L28. Should we set it up in the bazel build?

[jwnimmer-tri]

(1) Would it suffice to provide the solver_status as a string? That's probably the easiest to implement and maintain. Will users need a compile-time-checked list of status codes? If they are only going to print it out, the string would be best. If they are going to branch on it, the strings might be okay but we might want an enum, too.

(2) Sure. Would example_sdp.cpp from the Clarabel.cpp examples be a good demo?

[hongkai-dai]

(1) Would it suffice to provide the solver_status as a string? That's probably the easiest to implement and maintain. Will users need a compile-time-checked list of status codes? If they are only going to print it out, the string would be best. If they are going to branch on it, the strings might be okay but we might want an enum, too.

Using string could work. My only concern is that in IpoptSolverStatus, we have the pattern of returning the int value status, and a function to convert that int value to a string

drake/solvers/ipopt_solver.h

Line 40 in 1d00891

const char* ConvertStatusToString() const;

. I wonder if we should follow the same pattern here, with an enum and a function to convert the Clarabel status to string.

(2) Sure. Would example_sdp.cpp from the Clarabel.cpp examples be a good demo?

Yes, that is a good example, I was also following that example.

[hongkai-dai]

I have a working branch with ClarabelSolver in drake. It works much better (more accurate) than SCS.

Currently this branch doesn't support SDP yet. But I think even without SDP support, this will still be useful, as many of our problem require second order conic constraints, and we do not have a good open-source solver for second order cone problem yet.

@jwnimmer-tri does it make sense to add the Clarabel bazel build file into Drake, and we integrate the ClarabelSolver? We can add the SDP support later.

[jwnimmer-tri]

I'll propose an order of operations something like this:

• Land the build system.
• Land the clarabel_solver, but disabled by default (like with Mosek or Gurobi).
  • Mark it as "experimental" (i.e., unstable).
  • Users building from source can opt-in.
  • In CI only the "Everything" build flavor would test it.
• Continue to experiment with improvements. Both on the build system side (making it suitable / safe to install in our packages) and on the solver side (param tuning, debug options, test coverage, etc.).
• Add it to choose best solver.
• Enable the build by default (make it opt-out instead of opt-in) and ship in our release packages.

(We could add the SDP feature at any point in that sequence.)

[goulart-paul]

Currently this branch doesn't support SDP yet. But I think even without SDP support, this will still be useful, as many of our problem require second order conic constraints, and we do not have a good open-source solver for second order cone problem yet.

We have just released v0.6.0 which includes some speed and stability improvements for SOC handling. Probably worth bumping versions if you use a lot of SOCPs.

[jwnimmer-tri]

I'll propose an order of operations something like this:

• Land the build system.

...

See #20246 for getting the Drake build system off the ground.

We have just released v0.6.0 ...

Thanks! I've used the latest versions of everything in my pull request, and our plan is to have our monthly "upgrade all dependencies" automatically keep up-to-date with the latest numbered releases.

[jwnimmer-tri]

I started investigating how to enable Clarabel SDP in the build system, and ran into oxfordcontrol/Clarabel.rs#61, so I'm putting my work on pause for the moment.

[jwnimmer-tri]

I was able to figure out what's what, so the next PR (to add SDP to the build system) is filed now.

[jwnimmer-tri]

I believe the remaining tasks look like this:

• Add SDP support (with tests) to solvers/clarabel_solver.cc.
• Add ClarabelSolver to the "choose best solver" ladder.
• (Possibly) Invite more early adopters to test ClarabelSolver on their own programs.
• Figure out how to avoid symbol conflicts / ODR violations when shipping Rust code in our installed packages.
• Enable ClarabelSolver by default (make it opt-out instead of opt-in) and ship in our release packages.

[hongkai-dai]

Thanks Jeremy, I will work on adding the SDP support.

[jwnimmer-tri]

For the record, SDP support was added and released in https://drake.mit.edu/release_notes/v1.23.0.html. We invite potential users of this feature to rebuild from source (per the release notes) and try out ClarabelSolver.

I think the remaining tasks look like this:

• Add ClarabelSolver to the "choose best solver" ladder.
• Figure out how to avoid symbol conflicts / ODR violations when shipping Rust code in our installed packages.
• Enable ClarabelSolver by default (make it opt-out instead of opt-in) and ship in our release packages.

[calderpg-tri]

Now that #20572 and #20587 have both landed, are there any blockers to enabling ClarabelSolver by default? Collaborators at Woven are particularly eager to use Clarabel, but also depend on Snopt in the binary releases.

[hongkai-dai]

Now that #20572 and #20587 have both landed, are there any blockers to enabling ClarabelSolver by default? Collaborators at Woven are particularly eager to use Clarabel, but also depend on Snopt in the binary releases.

I am working on a PR to enable Clarabel by default.

[jwnimmer-tri]

I think it's probably ready to be tentatively enabled by default. I still want poll some other users about the integration of Rust into their build systems, but that might be easiest with it on-by-default already. I'll work on a PR.

[hongkai-dai]

One thing to notice is that the default open-source solver for QP is switched from OSQP to Clarabel. This might affect our robot controllers (which solves the differential IK problem as a QP). We will need to test that in anzu.

[goulart-paul]

If you are currently using OSQP for QPs, are you using the OSQP feature that allows for reuse of solver objects when solving similar problems, i.e. by just updating entries of the matrix or vector valued problem data?

We don't currently have that feature implemented in Clarabel, but it is on our road map.

[jwnimmer-tri]

If you are currently using OSQP for QPs, are you using the OSQP feature that allows for reuse of solver objects when solving similar problems, i.e. by just updating entries of the matrix or vector valued problem data?

Not yet.

There has been strong interest in that feature, and a few prototype branches created, but nothing has made it onto Drake's master branch yet.

[jwnimmer-tri]

Also for clarity for anyone watching here... Drake's next release (v1.24.0, scheduled for ~1 week from now) will have ClarabelSolver enabled by default when using solvers::Solve(), motion planning, etc. In the meantime, you can try the nightly builds starting tomorrow (https://drake.mit.edu/installation.html).