Compatibility with qutip v5 #116

BoxiLi · 2024-05-26T12:14:36Z

This is a fix to make krotov compatible with qutip version 5.

I replaced the oct_result.dump with the result from the first Jupyter Notebook example.
@Ericgig Would you have time to double-check if any of the changes I made may affect the performance? Especially the replacement of dense2D_to_fastcsr_fmode and spmvpy_csr. For the latter one, I simply multiply the data layer matrix to the vector and hope the dispatch works.
I have to lose the tolerance for the test of mesolve result from 1e-7 to 5e-4 in tests/test_objectives.py. I think it is just machine precision but I'm not entirely sure.
@goerz I don't know how to create a test file oct_result_incomplete.py for the serialization test, which is now the only test failing on my machine.

goerz

Thanks! This is super helpful!

The automatic tests seem to have had a lot of bitrot in the last couple of years, so it probably makes sense to deal with #117 first, and then rebase this. I'll try to have a look at that soon.

And then, of course, #119 would be the next step

goerz · 2024-05-26T14:01:52Z

src/krotov/objectives.py

@@ -15,6 +15,11 @@

 import numpy as np
 import qutip
+from packaging.version import parse as parse_version
+if parse_version(qutip.__version__) < parse_version("5"):
+ is_qutip5 = False


This is going to go into krotov v2.0, and I think it would be fine to make qutip v5.0 a requirement for that release. So I don't think this is needed. We can drop compatibility with QuTiP 4

Ericgig · 2024-06-03T14:19:37Z

src/krotov/propagators.py

+ return qutip.Qobj(
+ unstack_columns(self._y),
+ dims=state.dims,
+ isherm=True,
+ dtype="csr"
+ )


dtype is not a parameter of Qobj.
Is that output really sparse?
If the output need to be converted, use to:

Suggested change

return qutip.Qobj(

unstack_columns(self._y),

dims=state.dims,

isherm=True,

dtype="csr"

)

return qutip.Qobj(

unstack_columns(self._y),

dims=state.dims,

isherm=True,

).to("CSR")

Just as a note (and this could be a separate PR): The sparse storage of QuTiP objects was probably a major source of slowness in krotov. Most usages of this package would be for small Hilbert spaces – certainly all the examples are for small Hilbert spaces. If I followed QuTiP's development correctly, version 5 should have the ability to use dense storage internally, and it would be good to use that as much as possible here (in the examples). Of course, if people manually define their objectives with sparse objects, those should still be respected.

Yes, with qutip v5 you can do things all dense. One should set the qutip default as dense matrix in krotov and use sparse only if required by users.

BTW, this function seems not tested, since the mistake Eric mentioned doesn't lead to any falling tests. Maybe they are in the notebooks you mentioned?

Quite possibly… the notebooks definitely serve as "integration tests". With my update of the tooling, you should be able to run it locally and/or the CI should run it for you.

Ericgig · 2024-06-03T14:48:30Z

src/krotov/propagators.py

@@ -266,11 +280,17 @@ def _rhs(t, rho, L_list):
 out = np.zeros(rho.shape[0], dtype=complex)
 L = L_list[0][0]
 L_coeff = L_list[0][1]
- spmvpy_csr(L.data, L.indices, L.indptr, rho, L_coeff, out)
+ if is_qutip5:
+ out = L_coeff * L @ rho


rho and out are numpy arrays that need to be wrapped.

Suggested change

out = L_coeff * L @ rho

rho = qutip.data.Dense(rho, copy=False)

out = qutip.data.matmul(L, rho, scale=L_coeff)

With return out.as_ndarray() at the end.

Ericgig · 2024-06-03T14:48:58Z

src/krotov/propagators.py

 for n in range(1, len(L_list)):
 L = L_list[n][0]
 L_coeff = L_list[n][1]
- spmvpy_csr(L.data, L.indices, L.indptr, rho, L_coeff, out)
+ if is_qutip5:
+ out = L_coeff * L @ rho


spmvpy_csr has an implicit out += that was forgotten.

Suggested change

out = L_coeff * L @ rho

out = qutip.data.add(qutip.data.matmul(L, rho, scale=L_coeff), out)

We have imatmul_data_dense(left, right, scale, out) that does the matmul implace like spmvpy_csr, but no python binding. (It's only used in QobjEvo.)

If I keep things in qutip.Qobj, not NumPy array, since we have a much better initialization of Qobj now in v5. Will sparse_csr_qobj * dense_ket_qobj automatically use that imatmul_data_dense?

This will allow the switch between dense and csr.

Since the states, (out, rho) are used by scipy's ODE solver they must enter and exit as numpy's array.
imatmul_data_dense is equivalent to out += left @ right * scale, this can't be represented in Qobj.__matmul__ operation so no. QobjEvo.matmul_data(t, state, out) would use the inplace operation if out is given and dense. But this use the list format directly and rewriting to use QobjEvo is probably not a small task.

qutip.data.add and qutip.data.matmul use qutip's data that can be sparse, dense, jax etc. so it should enough to be able to switch between format.

goerz · 2024-06-04T01:51:30Z

I've updated the tooling for the krotov package (using hatch now) so that the CI testing works again. Also, Python and most non-QuTiP-dependencies have been updated to new versions.

So, the current master would be a good to rebase this one.

I don't know how to create a test file oct_result_incomplete.py for the serialization test

I've added comments in the test file to explain where these files come from, in case they need to be updated. Basically, they're copied from the example notebooks (for one of them, you'd have to temporarily add a oct_result.dump(filename) in the notebook)

which is now the only test failing on my machine.

There's probably also still the example notebooks, and for one of them, the weylcamber package will have to be updated to be compatible with QuTiP 5. Like here, it would be okay to make a breaking version that is only compatible with QuTiP 5.

goerz · 2024-06-04T18:42:21Z

I've released weylchamber v0.6 which supports qutip v5 (and only qutip v5)

goerz · 2024-06-04T19:01:02Z

@BoxiLi I rebased your PR branch on the latest master. For some reason, GitHub wouldn't allow me to force-push that back to your clone. So I've pushed the rebased commits to a quitp5 branch of the main repo.

Maybe you can update your own qutip5 branch with those commits. Alternatively, I've invited you with write access to the krotov repo, so maybe you can continue working on the qutip5 branch here. That would probably require closing this PR and opening a new one. Feel free to do that, if you want to go that way.

BoxiLi · 2024-06-04T20:15:01Z

Thanks @goerz , will have more time later this week on this!

And thanks @Ericgig for the corrections!

goerz · 2024-06-04T21:21:55Z

will have more time later this week on this!

Sounds good! If you run into any technical problems, let me know. You can also hit me up on the Unitary Fund Discord or the Julia Slack if there's something that would benefit from real-time discussion, or we can always hop on a Zoom call or something

BoxiLi · 2024-06-14T10:06:31Z

@goerz All the main tests are passing now on Linux, I think I'll merge to krotov/qutip5 now and address the rest (like notebooks) separately

BoxiLi · 2024-06-14T10:18:18Z

The overhead in the new qutip.Qobj is much smaller compared to the old one, if the propagator is replaced by a hand-written expm, the example in notebook 1 (compared to NumPy in notebook 9) takes:
NumPy Array : 2s
qutip-v5 Dense: 3s
qutip-v5 CSR: 6s
qutip-v4: 21s

The overhead for using the ODE solver is still there, unfortunately, for now.

goerz · 2024-06-14T11:07:07Z

Very nice! And that's even without the further potential improvements from #119 (right?), which I think will make a big difference.

Let me know if you need any help or input with wrapping this up for final review and merging.

BoxiLi · 2024-06-14T11:46:06Z

Indeed, that is not considered, but one has to use a simple self-written expm propagator, not the default krotov propagators to bypass that.

Ok! I'll merge this to the development branch qutip5.

goerz added this to the 2.0 milestone May 26, 2024

goerz mentioned this pull request May 26, 2024

Use stateful solver class in QuTiP 5.0 #119

Open

goerz reviewed May 26, 2024

View reviewed changes

goerz force-pushed the master branch 2 times, most recently from 28e1367 to e9754de Compare June 3, 2024 01:56

Ericgig reviewed Jun 3, 2024

View reviewed changes

BoxiLi force-pushed the qutip5 branch 2 times, most recently from 189663f to 3743055 Compare June 13, 2024 10:39

BoxiLi changed the base branch from master to qutip5 June 13, 2024 13:21

BoxiLi added 5 commits June 14, 2024 11:55

Black style updates

858bef6

Update the serialization test

a4a2a9d

Remove qutip4 code

177348a

Update qutip requirement to 5

c2ac0e4

Fix the bug in propagators.py

e9277af

BoxiLi force-pushed the qutip5 branch from fef2b08 to e9277af Compare June 14, 2024 09:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compatibility with qutip v5 #116

Compatibility with qutip v5 #116

BoxiLi commented May 26, 2024 •

edited

goerz left a comment

goerz May 26, 2024

Ericgig Jun 3, 2024

goerz Jun 3, 2024

BoxiLi Jun 4, 2024

goerz Jun 4, 2024

Ericgig Jun 3, 2024

Ericgig Jun 3, 2024

BoxiLi Jun 4, 2024 •

edited

Ericgig Jun 4, 2024

goerz commented Jun 4, 2024

goerz commented Jun 4, 2024

goerz commented Jun 4, 2024

BoxiLi commented Jun 4, 2024

goerz commented Jun 4, 2024

BoxiLi commented Jun 14, 2024

BoxiLi commented Jun 14, 2024

goerz commented Jun 14, 2024

BoxiLi commented Jun 14, 2024 •

edited

	out = L_coeff * L @ rho
	rho = qutip.data.Dense(rho, copy=False)
	out = qutip.data.matmul(L, rho, scale=L_coeff)

	out = L_coeff * L @ rho
	out = qutip.data.add(qutip.data.matmul(L, rho, scale=L_coeff), out)

Compatibility with qutip v5 #116

Are you sure you want to change the base?

Compatibility with qutip v5 #116

Conversation

BoxiLi commented May 26, 2024 • edited

goerz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BoxiLi Jun 4, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goerz commented Jun 4, 2024

goerz commented Jun 4, 2024

goerz commented Jun 4, 2024

BoxiLi commented Jun 4, 2024

goerz commented Jun 4, 2024

BoxiLi commented Jun 14, 2024

BoxiLi commented Jun 14, 2024

goerz commented Jun 14, 2024

BoxiLi commented Jun 14, 2024 • edited

BoxiLi commented May 26, 2024 •

edited

BoxiLi Jun 4, 2024 •

edited

BoxiLi commented Jun 14, 2024 •

edited