Edge Function CPU Timeouts when generating embeddings

[kevinamick]

Hey folks,
Re: https://github.com/supabase-community/chatgpt-your-files

Firstly, thanks for the great tutorial! Amazing content.

I'm running into an issue with the "embed" edge function both locally and when deployed. When running local I was unable to embed more than ~5 1000 character sections before running into CPU hard limits. This seems to be even worse when deployed. Most of the time the function will crash after the first section. Playing around with the chunk sizes seemed to have no affect. Each invocation is taking around 5 seconds as well.

Is there anything that can be done about this? I've done a ton of digging and I'm not seeing anyone else with this issue so i'm wondering if i'm doing something wrong. But I'm running virtually a 1:1 with this repo other than processing sections by characters instead of markdown headings in the process function.

export function processText(content: string, document_id: number, chunk_size = 1000): Section[] {        
    const numChunks = Math.ceil(content.length / chunk_size);
    const chunks = new Array(numChunks);
    
    for (let i = 0, o = 0; i < numChunks; ++i, o += chunk_size) {  
        chunks[i] = content.substring(o, o + chunk_size);
    }      

    return chunks.map((content: string) => ({ document_id, content }))
}

Example invocation failure:

Log ID
44ffebf4-ff3b-494d-8364-edaa280c4737

Log Timestamp (UTC)
2024-02-18T03:14:13.627Z

Log Event Message
POST | 500 | https://zyzsfcjzezlmdebofdfy.supabase.co/functions/v1/embed

Log Metadata
[
{
"deployment_id": "zyzsfcjzezlmdebofdfy_7b8c1d94-2790-4f4c-b5d7-72249d222742_7",
"execution_time_ms": 5533,
"function_id": "7b8c1d94-2790-4f4c-b5d7-72249d222742",
"project_ref": "zyzsfcjzezlmdebofdfy",
"request": [
{
"headers": [
{
"accept": "/",
"accept_encoding": "gzip",
"connection": "Keep-Alive",
"content_length": "111",
"cookie": null,
"host": "zyzsfcjzezlmdebofdfy.supabase.co",
"user_agent": "pg_net/0.7.1"
}
],
"host": "zyzsfcjzezlmdebofdfy.supabase.co",
"method": "POST",
"pathname": "/functions/v1/embed",
"port": null,
"protocol": "https:",
"search": null,
"url": "https://zyzsfcjzezlmdebofdfy.supabase.co/functions/v1/embed"
}
],
"response": [
{
"headers": [
{
"content_length": "84",
"content_type": "application/json",
"date": "Sun, 18 Feb 2024 03:14:13 GMT",
"server": "cloudflare",
"vary": "Accept-Encoding",
"x_sb_edge_region": "us-west-1",
"x_served_by": "supabase-edge-runtime"
}
],
"status_code": 500
}
],
"version": "7"
}
]

Any pointers here would be greatly appreciated!

[stevengpn]

• Got same issue, the edge function only allows 2 seconds, which is not enough to even download the embedding model from the remote.
• Local hosting the file is also not possible according the supabase egde functino docs

[laktek]

In the last Launch Week, we announced native embedding generation in Edge Functions, which should solve these CPUTime limits https://supabase.com/blog/ai-inference-now-available-in-supabase-edge-functions

Check this guide on how to use this new API within your Edge Functions https://supabase.com/docs/guides/functions/ai-models#how-to-generate-text-embeddings