CPU
This article primarily organizes local AI solutions that can run on CPU only.
These can run on computers without graphics cards, old laptops, or even small embedded devices like Raspberry Pi.
Currently, the preliminary compilation includes:
- Rembg (pure CPU offline image background removal)
- Sherpa-onnx (next-generation pure CPU offline speech recognition and synthesis)
- Piper TTS: A lightweight, offline neural network text-to-speech (TTS) tool that runs in real-time on CPU, quickly converting text into natural speech.
- YOLOv10 / YOLOv8 (ONNX CPU Quantized) Machine Vision
- Marker (fully automatic, high-precision PDF to Markdown pure CPU pipeline)
- llama.cpp: A high-performance local large model inference engine that allows you to run large language models like Llama/Qwen on CPU or GPU.
- whisper.cpp: A Whisper speech recognition engine implemented in C/C++, capable of real-time offline speech-to-text conversion on CPU.
- Stable Diffusion CPP (sd.cpp) (pure CPU pixel-level drawing)
Use Cases
I haven’t deployed llama.cpp yet, but it’s a local LLM, which I primarily intend to use for local large model content translation and organization.
I haven’t deployed whisper.cpp yet; currently, I use fast whisper for speech-to-text on my phone, recording text for my blog. I plan to use whisper.cpp later as a backup for fast whisper, running it on a mini PC as an AI service.
I haven’t deployed Piper TTS yet, and I don’t have any TTS-related business needs at the moment. In the future, I might use TTS to generate audio blogs for blog posts or TTS audio for corresponding videos. This is just an idea for now, noted here.
I haven’t deployed Sherpa-onnx yet, but GPT highly recommends it because it’s a new generation ASR + TTS, a full-link tool with a good ecosystem and high performance. It can stream real-time speech recognition, convert speech to text, and text to speech, across platforms. It’s equivalent to whisper.cpp + Piper TTS but relatively better, suitable for offline voice assistants, such as Raspberry Pi smart homes.
I haven’t deployed YOLOv10 / YOLOv8 yet; it falls under machine vision. I’m not sure how to use it currently, but it’s generally for person recognition, object recognition, etc. I’m noting it down for future reference.
Marker is for document processing; I haven’t deployed it yet, and I don’t have specific needs or use cases for it at the moment. It can convert PDFs to Markdown and JSON, among other formats.
Rembg is for image processing, specifically background removal. I use it when creating YouTube thumbnails, and it’s excellent, highly recommended.
Stable Diffusion CPP is text-to-image generation, which is what I need most right now. For example, I currently use local flux2.0 to generate cover images or illustrations for blog posts. The images generated with 6GB VRAM are barely adequate. I hope CPU can also handle it, but I haven’t deployed this project yet, nor do I expect it to achieve ideal results. It’s noted here for now, waiting for its ecosystem to mature.
Priority
Due to time constraints, I will prioritize trying llama.cpp. It can be combined with local large models to become my AI assistant for tasks like translation and daily document processing and integration. My app translation will use it.
Next, I will prioritize deploying Sherpa-onnx because it supports both ASR + TTS. I only need to deploy one project to get two functionalities. Moreover, it’s more of a tool than a low-level component. My self-developed app’s speech-to-text feature will use it, and it will later be expanded for meeting minutes, offline smart home voice assistants, etc.
Then I will deploy Rembg, because I will later supplement blog posts with YouTube videos, and I’ll need to remove image backgrounds to create video cover images.
whisper.cpp and Piper TTS are reserved projects. If Sherpa-onnx’s performance isn’t ideal, I’ll consider deploying them, or when I have free time, I’ll deploy them to compare their performance with Sherpa-onnx and keep the one that performs better (but only if I have ample free time).
YOLOv10 / YOLOv8, Stable Diffusion CPP, and Marker will not be installed unless absolutely necessary. I don’t want to go looking for problems to solve; this is just a record for those who need to know that they can run on CPU.
UI
llama.cpp is too low-level; we need a simple and easy-to-use UI to use it. Here, I recommend Open WebUI.
With Open WebUI’s iterations, it has listed llama.cpp as one of its officially natively supported Providers (model providers). llama.cpp’s built-in llama-server automatically exposes a standard OpenAI-compatible API, which Open WebUI can directly connect to and recognize.
LocalAI
Unified Solution: LocalAI
Typical inference tools (like llama.cpp) usually only address text generation (Chat) problems. However, a complete AI application often requires multiple AI capabilities.
LocalAI’s strength lies in its ability to achieve a ‘zero-cost freeloading’ full ecosystem within a single Docker container, utilizing pure CPU instruction set acceleration (AVX/AVX-512), simultaneously implementing the following:
- Text Generation: Integrates llama.cpp / ggml at the core, running GGUF models (e.g., Qwen-Coder, DeepSeek) on CPU.
- Audio Transcription: Directly integrates whisper.cpp at the core, completely replacing OpenAI’s Whisper API.
- Text-to-Speech (TTS): Supports local lightweight speech synthesis (e.g., Piper), capable of offline audio generation purely on CPU.
- Image Generation: Can even generate images in the background via the CPU version of Stable Diffusion (sd.cpp).
- Text Embeddings and Vectorization: Runs local BERT models purely on CPU, handling local RAG knowledge bases.
Friendly Reminder
The AI selections above are answers I received from GPT and Gemini.
However, I was very dissatisfied; even with my prompts, their answers were far from what I wanted.
Open WebUI and LocalAI weren’t even mentioned; I had to bring them up myself.
I did some online research and found that LocalAI’s ecosystem is very popular with many stars, but it’s not widely mentioned, possibly because I use Windows.
LocalAI doesn’t seem to support .exe installation, preferring Docker and similar methods. For Windows, this requires installing WSL2 and Docker Desktop, and most people aren’t familiar with docker-compose.
This creates some difficulty in usage.
📊 LocalAI Core Capabilities & Advantages at a Glance
| Dimension | LocalAI’s Core Capabilities & Technical Highlights | Why it’s the ‘No-GPU Independent Development’ Ace (Advantages) |
|---|---|---|
| 💡 Zero Code Modification | 100% compatible with OpenAI REST API (e.g., /v1/chat/completions), with identical interface standards. | When developing Astro blogs or Node.js asynchronous tasks, simply modify the OPENAI_BASE_URL environment variable to switch between local/cloud APIs, without touching business code. |
| ⚡ Max CPU Utilization | Deep integration with llama.cpp, supporting AVX2 / AVX-512 CPU instruction acceleration. | No GPU dependency; ordinary CPUs can run quantized models (e.g., Qwen 1.5B / Mistral 7B quantized versions), enabling low-cost local inference. |
| 🧠 All-in-One Ecosystem (AIO) | One-stop integration of multiple inference backends: LLM + ASR + TTS + Image Generation + Vector Models. | A single Docker container replaces multiple AI services: • Text: llama.cpp • Speech Recognition: whisper.cpp • Text-to-Speech: Piper TTS • Image: Stable Diffusion (CPU/lightweight backend) • Vector: BERT / embedding |
| 💾 Memory Lock Optimization | Supports lock_memory: true to lock model memory and prevent swapping. | Prevents Windows / Linux swap from moving models out of memory, reducing stuttering on subsequent requests (especially in long conversation scenarios). |
| 🏪 Geek Ecosystem (Model Gallery) | Built-in model management and download mechanism (YAML-configured model marketplace). | No need to manually download Hugging Face models; supports one-click pulling of lightweight models and automatic API registration. |
Final Plan
I now plan to prioritize trying LocalAI because my Windows system already has WSL2 and Docker Desktop installed.
LocalAI offers comprehensive capabilities, a corresponding interactive UI, and an API. This means whether I use it on the desktop, through an app via API, or if new capabilities and models emerge later, LocalAI will quickly integrate them and address compatibility issues and performance optimization.
By following LocalAI, I can solve 90% of my problems. If there are particularly important issues that cannot be resolved, I will dedicate time to handle other local AI solutions or use cloud AI as a supplement.