What are the models that are suited to run on m1 macbook pro

[Bennoo]

Hey,

I am experimenting for now but I tried a lot of models.. Only one is working, the smallest one OPT-125m
Dolly-v2-3b is running but never answering the requests and others are simply not starting.
I am running them within a dev container (based on python:3.11.3-bullseye) so I assume there are not enough resources for them to run properly.

How can I build myself a rough idea of the suitable size of a model to run on a laptop?

[aarnphm]

I found all OPT,T5-variant, and GPT-neox works on m1

[FotieMConstant]

Hey, i am having issues running it locally on my M1, any dependencies i need to have installed or something?
i am getting an error when i hit the endpoint with the model: opt.

Error: "An error has occurred in BentoML user code when handling this request, find the error details in server logs"

an i get log of something like this:

2024-01-04T18:56:50+0300 [ERROR] [runner:llm-opt-runner:1] Exception in ASGI application

| RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'
+------------------------------------
2024-01-04T18:56:50+0300 [ERROR] [api_server:llm-opt-service:7] Exception on /v1/generate [POST] (trace=9b9fe4e8c4d816bc0f026f8a283b5dbc,span=fcc0918cce1b5ca8,sampled=1,service.name=llm-opt-service)

any idea anyone?

[aarnphm]

You can set DTYPE=float32 in front.