Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate embeddings for text sources #105

Open
brainless opened this issue May 4, 2024 · 2 comments
Open

Generate embeddings for text sources #105

brainless opened this issue May 4, 2024 · 2 comments
Assignees
@brainless
Labels
enhancement New feature or request rust Pull requests that update Rust code

Comments

@brainless
Copy link
Owner

As a user I would like to generate embedding for my text sources. Text from Markdown files will have some structure which may be useful internally.

Engineering notes

We should use APIs from OpenAI, Anthropic at the minimum. SentenceTransformers would be a great addition, either running locally or on a hosted server (can be done later).
@brainless brainless added enhancement New feature or request rust Pull requests that update Rust code labels May 4, 2024
@brainless brainless self-assigned this May 4, 2024
@brainless
Copy link
Owner Author

This will depend on #106 if we want to implement support for SentenceTransformers.

@brainless
Copy link
Owner Author

#104 will deal with storage. Perhaps we will work on these two issues together.

@brainless brainless changed the title Generate embedding for text sources Generate embeddings for text sources May 6, 2024
@brainless brainless mentioned this issue May 8, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brainless brainless

Labels
enhancement New feature or request rust Pull requests that update Rust code
Projects
Building Dwata
Status: Done
Milestone
No milestone
Development

No branches or pull requests
1 participant
@brainless