pgml.embed trust_remote_code

[kczimm]

A user on Discord submitted the following query with subsequent error:

SELECT pgml.embed(
    transformer => 'jinaai/jina-embeddings-v2-base-en',
    text => 'Dynamical Scalar Degree of Freedom in Horava-Lifshitz Gravity"}',
    kwargs => '{"trust_remote_code": true}'
);

ERROR:  ValueError: Loading this model requires you to execute custom code contained in the model repository on your localmachine. Please set the option trust_remote_code=True to permit loading of this model.

We are currently using sentence_transformers and InstructorEmbedding Python packages for the pgml.embed function. This model gets dispatched to the sentence_transformers package which does not appear to support the trust_remote_code keyword.

Huggingface shows, for this model, to use transformers.AutoModel. We should probably add this in, but how do we determine which model should use which package? Can they all use transformers? These questions need answering.

[kczimm]

UKPLab/sentence-transformers#2352