The project is an MoE-native serving engine that treats GPU, CPU, host RAM, and PCIe bandwidth as one inference platform. ⚡
Published paper results include:
Qwen3.6-35B-A3B
RTX 4060 Laptop 8GB
39.3 tok/s
DeepSeek-V4-Flash 284B
RTX 5090
22-25 tok/s
The full expert pool lives in system RAM and VRAM acts as an expert cache.
On cache misses, FreeToken can either transfer an expert to the GPU or execute it directly on the CPU, with the split chosen from measured bandwidth.
Important caveat: low VRAM does not mean low total memory. The host RAM still has to hold the expert weights.
https://github.com/FlashML-org/FreeToken
Current support is mainly x86_64 + NVIDIA RTX 30/40/50-series hardware.
Has anyone here benchmarked it against llama.cpp/Ollama on the same checkpoint and hardware? I would be interested in real-world agent workloads rather than short synthetic decode tests.
I've gotten about 12-15 token/sec on a 2070 Super (8GB) with a build called microFlare + llama.cpp, but it has the MTP stripped out. I bet they're using the MTP to double their base performance