Running Local LLMs
Here is a list of models that can potentially run well on the Orange Pi 6 with 16GB of RAM. For example, after testing some of the models on this list, especially the heavier ones with 32 billion parameters, they crash our board due to low available RAM bottleneck. If you plan to run 32 billion parameter models, it is advisable to have 32GB of RAM or more. The 14 billion parameter models probably represent the maximum memory requirements that this board can manage.
- 8B models (like Llama 3.1 8B) use about 5 GB of RAM. (Very fast, no issues).
- 12B models (like Mistral Nemo) use about 8 GB of RAM. (Runs great).
- 14B models (like Qwen 2.5 14B or DeepSeek 14B) use about 9.5 GB of RAM. (Runs perfectly, leaving ~6GB for Ubuntu and the KV cache).
- 32B models require approximately 20 GB of RAM. Therefore, if our board supports only 16 GB, it will cause our OPi board to “crash.”
| # | Model | Size | RAM needed | Safe on your 16GB Pi? |
|---|---|---|---|---|
| 1 | llama3.1:8b | 8B | ~5 GB | ✅ Yes |
| 2 | mistral-nemo | 12B | ~7.5 GB | ✅ Yes |
| 3 | qwen2.5:7b | 7B | ~4.5 GB | ✅ Yes |
| 4 | deepseek-r1:8b | 8B | ~5 GB | ✅ Yes |
| 5 | qwen2.5:14b | 14B | ~9 GB | ⚠️ Only with swap |
| 6 | deepseek-r1:14b | 14B | ~9 GB | ⚠️ Only with swap |
| 7 | phi4 | 14B | ~9 GB | ⚠️ Only with swap |
| 8 | qwen2.5-coder:14b | 14B | ~9 GB | ⚠️ Only with swap |
| 9 | mixtral:8x7b | 46B (MoE) | ~26 GB | ❌ Crashed and didn’t work |
| 10 | qwen2.5:32b | 32B | ~20 GB | ❌ Crashed and didn’t work |
Here are a few additional insights on attempting to run large models locally.
Out of a total of 16GB of RAM, around 4GB is used by the operating system, which essentially leaves you with around 10-12GB depending on what you are running in memory. While this is limiting, it is a factor worth considering. The better method is to run the models exclusively in a CLI environment, without any RAM-consuming desktop environments.
Our tests
Every model we tested processes the same five prompts. While each prompt is running, the script records the CPU percentage, RAM usage, and temperature.
- basic – “Reply with exactly: OK”
- math – “What is 2+2?”
- reasoning – a speed/distance word problem
- json – return valid JSON
- code – write a Python
add(a, b)function
