Embeddings support in the SDK

[levkk]

SDK

1. Added pgml.embed() support into Builtins, with single and batch mode.

[kczimm]

You can collect into a Result<Vec<_>> so you can ? the Result instead of panicking if the value is not a str.

[kczimm]

Something like

let texts = texts
            .0
            .as_array()
            .with_context(|| "embed_batch takes an array of texts")?
            .iter()
            .map(|v| {
                v.as_str()
                    .with_context(|| "only text embeddings are supported")
                    .map(|s| s.to_string())
            })
            .collect::<anyhow::Result<Vec<String>>>()?;

I think will work.

[levkk]

I don't think we want that. The API accepts an array of strings, so if someone passes in an array of objects that can be cast to a string, they will get embeddings of integers, or whatever else they passed in, which will make them think that everything is fine.

[kczimm]

We are still calling as_str here so if it was an object, it wouldn't be treated as a string. I was merely suggesting using Result::Err instead of unwrap. But all is good.

[SilasMarvin]

A test for both in Rust, Python, and JavaScript would be nice

[levkk]

A test for both in Rust, Python, and JavaScript would be nice

Do these tests run in CI or only locally? They require a PostgresML installation to be running somewhere right?

[levkk]

I can't seem to find instructions for setting up the JS environment for testing this. I did the Python one from memory, but it would be nice to have a short README for developers. I'm going to skip the JS tests for now and come back to them when I have a moment.