Llama2-7B mobile app crashes on Samsung S23 8GB RAM

[salykova]

Hi all,

I've succefully compiled .pte with default quantization parameters and tokenizer.bin for LLama2-7B according to the tutorial. However, during the inference the android app crashes with no error message (I assume its due to insufficient RAM). Is it currently possible to run 7B models on 8GB RAM phones?

P.S. ~ 4GB RAM out of 8GB is available

[iseeyuan]

@kirklandsign Could you help looking at it?

[iseeyuan]

@salykova could you provide more details on how to reproduce it, including the phone models, the Android version, etc.?

[salykova]

@iseeyuan

Device: Samsung S23, 8GB RAM, Android 14
Pytorch 0.2 stable branch
Java 17 JDK
Android SDK API Level 34
Android NDK 25.0.8775105

Steps to reproduce:

1. Follow the guide https://github.com/pytorch/executorch/tree/v0.2.0/examples/models/llama2 to create .pte and tokenizer.bin models for llama2-7b-chat (default params, default quant 128 groupwise)
2. Build android app following https://pytorch.org/executorch/0.2/llm/llama-demo-android.html
3. The app crashes with no error message immediately after clicking "generate"

[salykova]

@iseeyuan @kirklandsign

I've also tested both LLama2-7B and LLama3-8B models via adb binary-based approach and the inference works. Seems like the problem with the App or it requires much more RAM than adb-binary

[mergennachin]

cc @digantdesai - regarding s23 :)

[kirklandsign]

Hi @salykova the app requires slightly more RAM for Dalvik VM and graphics. However, I think the main issue is for binary, it has a high priority, so when OOM killer kills processes, it's usually killed last. I have seen situations like SystemUI killed before the binary killed.

For the app, it is a normal user app, and usually it's killed before system processes.

Unfortunately I don't have a good solution at the moment. I do see 8GB RAM runs sometimes, and 16GB runs almost all the time.

[salykova]

Hi @kirklandsign thanks for your response! Is it not possible to give the app higher priority? Sorry, I'm not an Android Developer and have no experience with it.

Also, I've found this option in the documentation https://developer.android.com/guide/topics/manifest/application-element.html#largeHeap. Can this increase stability of the app in theory?

[kirklandsign]

Hi @salykova unfortunately I don't have a good way to adjust the priority, especially for non-rooted device. android:largeHeap is for Dalvik heap space, but the RAM consumption happens in native layer so it doesn't help. I also tried with android:persistent but doesn't help either. So I am not sure how to fix this kind of issue at the moment.