PocketLM
PocketLM is an inference runtime I wrote from scratch to run large language models on ordinary hardware, with no GPU and no cloud. Instead of loading a whole model into memory, it streams in only the parts each token actually needs.
It runs Qwen and Gemma comfortably, and it runs DeepSeek-V3, a 671-billion-parameter model, on a laptop with 16 GB of memory. The largest models are still slow, so throughput is what I'm working on now.
In progress · not ready for public use yet
Why I built it
Running frontier-scale models locally is usually treated as a hardware problem: buy more GPU memory. I wanted to treat it as a systems problem instead. If a model is far too big for memory, can the runtime be smart enough about what it loads, and when, to run it anyway?
PocketLM is my attempt to answer that. I wrote it from scratch rather than on top of an existing runtime, so every layer is something I understand and can change.
What it does today
PocketLM runs large language models entirely on the CPU and streams their weights from disk as they are needed. Compact models such as Qwen and Gemma respond in about a second.
The stress test is DeepSeek-V3: 671 billion parameters, about 641 GB in its original FP8 weights, running on a laptop with 16 GB of memory and no graphics card. It runs end to end at around 34 seconds per token. That is slow, but it is 16.6 times faster than my first working version, which needed more than nine minutes per token.
How it's built
The orchestration layer is Python and PyTorch. The hot paths, such as FP8 matrix multiplication and routing tokens to experts, are native C++. Smaller models in GGUF format run through llama.cpp.
Around the runtime sits a desktop app for loading models, chatting and comparing runs, and more than 560 automated tests that guard correctness while the internals keep changing.
Where it stands
PocketLM is a research runtime, not a product yet. Installation still takes too many steps, and at half a minute per token the largest models prove that it can be done rather than being something you would use every day.
The next phase is throughput: prefetching experts before they are needed, caching the ones that are used most, and moving more of the pipeline into native code. The long-term goal is trillion-parameter models such as Kimi K3, unquantized, on hardware people already own.
How a 641 GB model runs on a 16 GB laptop
- 01Sparse by designDeepSeek-V3 is a mixture-of-experts model. Each layer has 256 small expert networks, and a router picks only 8 of them, plus one shared expert, for every token. So of its 671 billion parameters, only about 37 billion are active at any moment. PocketLM is built around that fact: it never needs the whole model at once.
- 02Streaming instead of loadingThe weights stay on the SSD in their original FP8 format, about 641 GB. For each token, the runtime reads just the experts the router selected, while the shared layers and the most frequently used experts stay in memory. Disk becomes an extension of RAM.
- 03FP8 on a CPUConsumer CPUs have no native FP8 math. I wrote C++ kernels that unpack FP8 weights on the fly and multiply them at higher precision, so the model never has to be converted to a larger format up front, which would need several times the space.
- 04Guessing aheadDeepSeek-V3 ships with a multi-token prediction head. PocketLM uses it for speculative decoding: it drafts the next token cheaply, then checks the draft in the same pass. Every accepted guess saves a full trip through the model.
- 05Measured, not assumedEvery optimization is benchmarked on the same prompts, with a check that the output still matches the reference. A speed-up only counts if the answer stays the same.