Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

支持多模态大模型需要做的修改 #3891

Closed
dorameng2023 opened this issue Apr 26, 2024 · 2 comments
Closed

支持多模态大模型需要做的修改 #3891

dorameng2023 opened this issue Apr 26, 2024 · 2 comments
Assignees
@zRzRzRzRzRzRzR
Labels
enhancement New feature or request

Comments

@dorameng2023
Copy link

dorameng2023 commented Apr 26, 2024 •
edited

功能描述 / Feature Description
支持多模态大模型，例如chatgpt4v 或者qwen-vl，以及视觉embedding模型，例如multimodal-embedding-one-peace-v1
解决的问题 / Problem Solved
实现图片问答&定位功能

实现建议 / Implementation Suggestions
请问如果想要实现该功能，现有版本具体需要做哪些改动
@dorameng2023 dorameng2023 added the enhancement New feature or request label Apr 26, 2024
@zRzRzRzRzRzRzR zRzRzRzRzRzRzR self-assigned this May 2, 2024
@zRzRzRzRzRzRzR
Copy link
Collaborator

那需要修改的太多了，从模型载入和langchain支持都得改，而且目前embed没办法支持多模态，比较好的方式是做成tools

@dorameng2023
Copy link
Author

@zRzRzRzRzRzRzR 能不能展开说说做成tools指的是什么 多谢

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zRzRzRzRzRzRzR zRzRzRzRzRzRzR

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zRzRzRzRzRzRzR @dorameng2023