New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Export Splatfacto-optimized poses #3116
base: main
Are you sure you want to change the base?
Export Splatfacto-optimized poses #3116
Conversation
I see.. When I activate SO3xR3, I checked the image eval result, the rendered image is little bit shifting due to camera optimization vs GT. I think it was cuprit for metrics (PSNR, LPIPS, SSIM) degradation. So, it corrected the eval images aligned with corrected poses. BTW, @jb-ye has overhauled the splatfacto in this draft (#3113). Hopefully it wont break each other. |
I am hesitant about exporting the optimized eval poses. This is very much only for the evaluation metrics reporting for paper write-ups. In general, I'm afraid we don't have a proper framework design in nerfstudio to handle eval pose optimization. I think it would be useful to export both original training poses and optimized training poses. At least this is useful to inspect the results. So just make this PR focus on training poses, no? What do you think? |
@ichsan2895 Sorry, I did not fully get this. Did you mean that this happens in the current Splatfacto or with this PR. What should happen is that in the current main branch, if you activate SO3xR3, you get the misalignment and metric degradation, but this PR fixes the issue.
@jb-ye I agree. Currently, the proper framework for optimizing eval poses (i.e., optimizing their poses while keeping all Gaussian parameters fixed) is missing. For this reason, I have only tested the
Currently, both the original and optmized training poses are available in the saved model. I think exporting the optimized training poses is a more reasonable default for The original poses are also presumably available in the input data, but they do not necessarily match the output of |
Yes, I think keep the PR simple is important for long term maintenance.
That could also work. Export the original poses (post various transforms), so they can be used to compare with the optimized pose. |
a4939f1
to
e123a0e
Compare
Update:
To summarize, the following aspects should now be fixed in Splatfacto pose optimization with the options
The following are left as future improvements:
Also note that the |
from nerfstudio.models.splatfacto import SplatfactoModel | ||
|
||
camera_optimizer = None | ||
if isinstance(pipeline.model, SplatfactoModel): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a custom code for SplatfactoModel. But I also have Nerfacto model which have camera optimizer enabled by default. Can we support both?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm. Good point. In theory, yes, but if we want a generic solution instead of separate ifs for Spaltfacto and Nerfacto, we would also need to fix this: #3073
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we should address #3073 first, and use the same API call for apply_to_camera()? The additional change seems very manageable in my opinion.
keep the current behavior for NeRFs, but match the splatfacto behavior for consistency
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. I'll get back to this a bit later
ac97c2d
to
b454a21
Compare
Currently works correctly only with --eval-mode=all
b454a21
to
0cb5d19
Compare
Currently,
ns-export cameras
does not export the optimized poses even if Splatfacto pose optimization is enabled. Similarly,ns-eval
does not take the pose optimization into account. This is because the "adjustment" to the initial pose is stored as a separate variable in the camera optimizer, which allows properly using the initial poses as priors. The NeRF code uses a different approach.This PR is a slightly cleaned-up version of the approach used in https://github.com/SpectacularAI/3dgs-deblur, which is basically ensuring that the "camera index" and an instance of the CameraOptimizer are available in the relevant places. However, this seems a bit ugly and fragile.
Any ideas of to improve this? Permanently "applying" the adjustments to the
camera_to_worlds
at the end of training could be one option, which would probably help withns-export cameras
(but not the eval PSNR metrics exported to, e.g., tensorboard, which are also fixed in this PR).This is also somewhat related to #3073.
Testing
First, download suitable data and install this branch (similarly to here):
Train (example):
Check metrics:
Then see the metrics in
output.json
. With this PR, should show high PSNR (> 32). Without this, shows something quite low (< 20) due to the misaligned evaluation poses.In addition, the poses exported with the following command are different (with this PR, they are the optimized ones, otherwise the original ones, possibly after some auto-scaling/transformations by the dataparser):