AirLLM：让70B/405B/671B超大语言模型在一块4GB显卡上跑起来 AirLLM: Run Massive 70B/405B/671B Language Models on a Single 4GB GPU
无需量化或剪枝，AirLLM 让 70B 模型在 4GB 单卡上运行，并新增支持 2.8T 参数的 Kimi K3，端到端显存仅 3.72GB。 AirLLM runs 70B models on a single 4GB GPU and adds support for 2.8T-parameter Kimi K3 under 4GB via per-expert streaming without quantization.
AirLLM 是一个面向大语言模型推理的开源项目，核心卖点是在极低显存下运行超大模型。根据 GitHub 仓库 README，AirLLM 将 70B 模型推理的内存占用大幅降低，使其可在单块 4GB GPU 上运行，且无需量化、蒸馏或剪枝。项目在 GitHub 已获得 27.1k 星标。
支持更大规模的模型
- 70B 模型：单块 4GB GPU
- 405B Llama 3.1：8GB GPU
- DeepSeek-V3（671B）：约 12GB GPU
- Kimi K3（2.8T）：小于 4GB（端到端实测 3.72GB）
AirLLM 大幅降低推理显存占用，让 70B 大语言模型能在单块 4GB 显卡上运行，而且无需量化、蒸馏或剪枝。
最新更新：支持 Kimi K3（2.8T）
2026 年 7 月的更新显示，AirLLM 已支持 Kimi K3（2.8T），这是目前最大的开源模型。在单张 RTX 6000 Ada 上，端到端显存占用仅为 3.72GB。这一实现得益于 per-expert streaming：模型只加载 token 实际路由到的专家层，而不是整个层。
- 安装 compressed-tensors
- 安装 flash-attn（模型代码强制使用 flash attention）
- 使用 CUDA 12 构建的 PyTorch（没有预构建的 flash-attn wheel）
AirLLM is an open-source project for large language model inference that focuses on running massive models with minimal GPU memory. According to the README, AirLLM reduces inference memory usage enough to run a 70B model on a single 4GB GPU — without quantization, distillation, or pruning. The repository has earned 27.1k stars on GitHub.
Support for larger models
- 70B models: single 4GB GPU
- 405B Llama 3.1: 8GB GPU
- DeepSeek-V3 (671B): ~12GB GPU
- Kimi K3 (2.8T): under 4GB (measured 3.72GB end-to-end)
AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning.
Latest update: Kimi K3 (2.8T) support
A July 2026 update adds support for Kimi K3 (2.8T), the largest open-source model to date. It runs on a single RTX 6000 Ada with only 3.72GB of VRAM, measured end-to-end. This is achieved through per-expert streaming, which loads only the experts a token actually routes to instead of full layers.
- Install compressed-tensors
- Install flash-attn (K3's model code mandates flash attention regardless of request)
- Use a CUDA 12 build of PyTorch (no prebuilt flash-attn wheel is available)
来源
- GitHub Trending · 08-04 08:06