Fetching model param super slow on Vercel

[louis030195]

for some reason it takes ~3 seconds to load model locally and 30 min on @vercel

stupid question, isn't the model downloaded on client side? so why the network would be different?

PS: similar question, any best practice to avoid loading model every time i switch page? using react hook to load model

[Neet-Nestor]

@louis030195

For the slow loading on Vercel, I'm not sure about the reason and I personally didn't meet the issue. We have WebLLM Chat deployed both on GitHub pages (https://chat.webllm.ai) and on Vercel (https://chat.neet.coffee/) and the loading speed are similar. Could you try it and see whether the same issue?

For you second question, there isn't a good solution since for security and performance reasons browsers typically doesn't allow website state to persist after all pages have been closes. One workaround is using ServiceWorker as WebLLM Chat does right now, which allows the WebLLM engine and its internal state to persist in worker thread even after the pages have been closed, but it is unstable as browser may still kill it at any moment.

For service worker implementation, you can reference examples/service-worker or WebLLM Chat.