Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Sam integration #7722

Draft
wants to merge 28 commits into
base: gen-ai-dev
Choose a base branch
from

Conversation

vgrau98
Copy link
Contributor

@vgrau98 vgrau98 commented Apr 28, 2024

Fixes #6357 .

Description

Could be splited in smallest pr

  • Image encoder (2D light vision transformer ViTDet). Output consistency checked with official sam weights.
  • Prompt encoder
  • Mask decoder
  • Segment Anything Network
  • Weights mapping/integration
  • Model zoo implementation
  • Monai label apps integration

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@vgrau98 vgrau98 changed the title Sam integration [WIP] Sam integration Apr 28, 2024
@marksgraham
Copy link
Contributor

Hi @vgrau98

Thanks for this - would you be OK to hold off until this PR is merged into the gen-ai branch, which will hopefully be soon?

@vgrau98
Copy link
Contributor Author

vgrau98 commented May 2, 2024

Hi @vgrau98

Thanks for this - would you be OK to hold off until this PR is merged into the gen-ai branch, which will hopefully be soon?

Of course, I think some tasks can be initiated while waiting for the refacto on the transformer blocks as soon as the pr is merged

@@ -24,25 +24,29 @@
from monai.utils import optional_import

einops, has_einops = optional_import("einops")
xops, has_xformers = optional_import("xformers.ops")

TEST_CASE_SABLOCK = []
for dropout_rate in np.linspace(0, 1, 4):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to suggest using itertools.product here rather than the nested for loops

@marksgraham
Copy link
Contributor

Hi @vgrau98
Thanks for this - would you be OK to hold off until this PR is merged into the gen-ai branch, which will hopefully be soon?

Of course, I think some tasks can be initiated while waiting for the refacto on the transformer blocks as soon as the pr is merged

hey @vgrau98 , thanks for waiting - it's now merged into gen-ai-dev so go ahead with this PR :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants