Affine registration

[jboulanger]

Description

Intensity based affine registration of image pairs.

• estimate a transform to register the moving image using ndi.affine_transform
• supports 2d and 3d images
• supports color image using channel_axis parameter (more flexible for 3d images)
• supports affine, euclidean and translation motion models using a string parameter
• supports weights/mask to use only a part of the image. Modifies the normalized_mutual_information to support weights.
• includes 2 solvers: lucas kanade [1] and studholme [2,3] parametrized as functions callback
• related to PR 3544 and 7050. The studholme solver is inspired heavily from PR 3544.
• include an quantification of alignment error (target registration error)

The API tries to take inspiration from the PR3455, however I found it not very user friendly to pass callback for handling euclidean or translation motion model. Moreover the logic is different between the two solvers and could not find a way to conciliate the previously discussed API with the added solver. PR3455 was relying on transforms in the example to handle restricted model (euclidean) which is not compatible with ndi.affine_transform, here we stick to ndi.affine_transform to ba able to handle 3D images.

Comparison bewteen the solvers and cv2.findTransformECC. The 2nd row display the registration error map.

Registration of a brain MRI volume

TODO

• euclidean is not implemented for lucas kanade solver for dimension > 2.

References

1. algo_affine
2. Studholme C, Hill DL, Hawkes DJ. Automated 3-D registration of MR and CT images of the head. Med Image Anal. 1996 Jun;1(2):163-75.
3. J. Nunez-Iglesias, S. van der Walt, and H. Dashnow, Elegant SciPy: The Art of Scientific Python. O’Reilly Media, Inc., 2017.

Checklist

• A descriptive but concise pull request title
• Docstrings for all functions
• Unit tests
• A gallery example in ./doc/examples for new features
• Contribution guide is followed (as much as possible)

Release note

For maintainers and optionally contributors, please refer to the instructions on how to document this PR for the release notes.

...

[stefanv]

Nice example; something to note for maintainers, looks like this exposes new API.
Ignore that comment, GH hid the new file from diff view, so I misread this as a gallery example.

[stefanv]

@jboulanger This is a huge amount of work; thank you very much! @jni (who guided #3544) and I have been discussing this feature ever since writing Elegant SciPy, and this puts us significantly closer to that goal. I would like for @jni to review, and @JDWarner has kindly offer to as well. This change is significant enough that we may want to dedicate the next developer call to it, and perhaps you want to join to discuss.

[mkcor]

Just took a quick, superficial look at this huge PR!

[mkcor]

When did the syntax/convention change? Just curious. 🙏

[jboulanger]

Hello, Thank for the modifications. I am not totally sure which syntax/convention you are referring to. Is it in the docstring? I propose a change in simple_metrics which is a bit a side effect of the main proposal. This is to have a uniform support of weights in the registration code across the two implemented methods. I hope this is backward compatible and doesn't break anything.

[mkcor]

I'm referring to the extra indent, which I also noticed lines 18-19 of doc/examples/registration/plot_register_affine.py. Perhaps the formatting got fixed/taken care of by the pre-commit hook, in which case you didn't actually decide on it. I was just curious about the change because, up until now(?), we would write:

.. [1] Juan Nunez-Iglesias, Stefan van der Walt, and Harriet Dashnow. Elegant
       SciPy: The Art of Scientific Python. 1st. O'Reilly Media, Inc.,
       2017. isbn: 1491922877, 9781491922873.

and not

.. [1] Juan Nunez-Iglesias, Stefan van der Walt, and Harriet Dashnow. Elegant
        SciPy: The Art of Scientific Python. 1st. O'Reilly Media, Inc.,
        2017. isbn: 1491922877, 9781491922873.

[jboulanger]

I have been exchanging a few messages with @jni on zulip before putting this together. However I am aware that the proposition diverged significantly from the original discussion that happened during #3544. I tried to balance usability and flexibility and also wanted to keep the multi-resolution factorized across the two methods.

[JDWarner]

Thank you for the contribution @jboulanger! What an awesome first PR!

This feature is one I have long anticipated and am excited to see in scikit-image.

My review will be in 2 parts; this first set of comments is mostly stylistic from a detailed read-through to get the ball rolling. In the near future, I will follow up after digging in further with testing on real images in both 2 and 3 dimensions, as well as some performance profiling.

Thanks again!

[JDWarner]

For both solver functions, from a forward-looking naming convention standpoint, we likely want to tweak the names. To optimize the discoverability and intuitiveness of the namespace, perhaps we should consider function names beginning with either their linked registration type or purpose.

So for 'affine' as here, we could go with either:
affine_solver_studholme (linking the function to affine), or
solver_affine_studholme (making it clear this function is a solver, then that it's linked to affine)

Either would be reasonable at the moment and I'd welcome input from the team as to their preference. I have a small preference to starting with solver_ so it's clear the group of functions is specific to that purpose.

[jboulanger]

I'll wait a bit for more feedback on this. I have no big preference.

[JDWarner]

The docstrings for each solver specify that we can use the model options "affine", "translation" or "euclidean", but most users would interact with this option at the level of this function so we should make that especially clear in this docstring.

As a lower bound, give them the options: Motion model: "affine", "translation" or "euclidean".

In addition, it might be wise to spend a couple sentences expanding on the difference/constraints of a model limited to translation or euclidean registration alone versus affine registration so a user with less experience could choose without searching elsewhere.

[jboulanger]

I added the following to the affine documentation:

  model: str
        Motion model: "translation", "euclidean" or "affine" where euclidean motion
        corresponds to rigid motion (rotation and translation) and affine motion can
        represent change in scale, shear, rotation and translations.

Let me know if this is the right place and if this is better documentation in general.

[jboulanger]

Thank you for the very informative feedback!