-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
External embeddings #239
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
GPU are most of the time not on the same machine than web crawlers. CPU takes time and is too slow to be use at scale for sentence embeddings.
Describe the solution you'd like
It would be nice to have an external scripts exactly like externalMT. This script will read in stdin and write embedding on stdout.
On the script the user will use whatever he wants to send the sentences to a machine with GPU to run it.
The text was updated successfully, but these errors were encountered: