Medical imaging using nerfstudio

[DominikSek]

Hello,

I was wondering would it be possible to use nerfstudio for solving the problem of model reconstruction from X-Ray images? Or nerfstudio does not support this.

Kind regards,
D

[maturk]

Hey @DominikSek , interesting question. I have not seen anyone try nerfstudio for this type of application yet, so I am unsure if it is possible. Nerfs currently are designed to work for 2D images ( with potentially depth information), and my intuition for X-Ray, due to the imaging format, is that you will have to consult more classical ray tracing libraries and computer graphics tools used for volume rendering to get your 3d reconstruction. In addition, I do not think nerfstudio supports any of the industry used x-ray/ct scan data formats since they are not simple pngs/jpgs correct? Let me know what your thoughts are...

[DominikSek]

My idea would be using 2D images taken from an X-ray volume (pair of them from different angles) and try to create a model that works okay

Edit:
The idea is somewhat like MedNerf, where they try to teach a NeRF for medical imaging. https://arxiv.org/abs/2202.01020

[maturk]

Very interesting, thanks for the link, I have not seen nerfs in medical applications before :D How will you get the pose of the 2D images? Note that nerfstudio relies highly on Colmap which is a tool used for RGB image pose estimation. It is unlikely to work for different image formats, like hyperspectral, x-ray etc.

[DominikSek]

Hmmmm... Thanks for the answer, i'll keep in touch if i come up with something

[maturk]

You can give it a try, perhaps xray imaging has enough distinguishable and localizable features for colmap feature extraction/matching to work. You should give it a multiple poses/images to work with though. Good luck.