【實戰指南】NVIDIA GB10 (DGX Spark) 本地 AI Agent：vLLM + Qwen3.8-27B-NVFP4 + Hermes Agent 部署調優 (ZH/EN Guide)

🌐 語言模式 / Language Mode:

NVIDIA Grace Blackwell GB10 vLLM V1 + NVFP4 + MTP Hermes Autonomous Agent

【實戰指南】在 NVIDIA GB10 (DGX Spark) 從零打造本地 AI Agent：vLLM + Qwen3.8-27B-NVFP4 + Hermes Agent 完整部署與調優

Building a Local AI Agent on NVIDIA GB10 (DGX Spark): Complete Setup & Tuning Guide for vLLM + Qwen3.8-27B-NVFP4 + Hermes Agent

徹底釋放 121 GB 統一記憶體與 Blackwell SM 12.1 原生 NVFP4 算力——包含 FlashInfer JIT 防 OOM 秘訣、MTP 投機解碼實測數據對決、256K 超長上下文配置與 Systemd 常駐維運指南。

Unleash 121 GB of Unified Memory and native Blackwell SM 12.1 NVFP4 acceleration — covering FlashInfer JIT OOM prevention, MTP speculative decoding benchmarks, 256K context configuration, and systemd production deployment.

Hardware: NVIDIA GB10 (aarch64 / SM 12.1) Engine: vLLM 0.28.1 V1 (CUDA 12.8) Model: unsloth/Qwen3.8-27B-NVFP4

Unified Memory

121 GiB

CPU / GPU 共享記憶體池

Shared CPU/GPU Memory Pool

NVFP4 Weights

22.21 GiB

Qwen3.8-27B 4-bit 浮點量化

4-bit Floating-Point Quantized

KV Cache Pool

64.06 GiB

1,740,978 Tokens (256K × 6.6)

1.74M Tokens Capacity (LBNHC)

MTP (n=3) Speed

13.2 tok/s

較無 MTP 提升 +30% 生成速度

+30% Faster Decode Throughput

01 系統架構與記憶體配置總覽 System Architecture & Unified Memory Allocation

NVIDIA Grace Blackwell GB10 (DGX Spark) 最大的優勢在於擁有 121 GB 統一記憶體 與 Blackwell SM 12.1 原生 NVFP4 硬體加速。在單機環境下，我們不僅能放入 27B 規模的推理模型（僅佔 22.21 GiB），還能切出高達 64.06 GiB 的 KV Cache，讓 Hermes Agent 在長達 256K context 的程式碼庫巡覽與多輪工具呼叫中游刃有餘。

The biggest advantage of the NVIDIA Grace Blackwell GB10 (DGX Spark) is its 121 GB Unified Memory paired with native Blackwell SM 12.1 NVFP4 hardware acceleration. On a single node, we can fit a 27B reasoning model (using just 22.21 GiB) while dedicating 64.06 GiB to the KV Cache, enabling Hermes Agent to handle 256K-context codebase exploration and multi-turn tool calls effortlessly.

▲ 圖 1：NVIDIA GB10 上的 Hermes Agent + vLLM V1 架構與 121 GB 統一記憶體配置比例 / Fig 1: Hermes Agent + vLLM V1 architecture and 121 GB Unified Memory layout on NVIDIA GB10

02 Step 1：使用 uv 建置 vLLM 與 CUDA 12.8 虛擬環境 Step 1: Setting Up vLLM & CUDA 12.8 Environment with uv

在 GB10 (aarch64) 上，強烈建議使用 uv 管理獨立的 Python 3.12 虛擬環境，並安裝 ninja 與 flashinfer-python 以支援 Blackwell FP4 JIT 編譯。

On GB10 (aarch64), we strongly recommend managing an isolated Python 3.12 virtual environment via uv, and installing ninja and flashinfer-python for Blackwell FP4 JIT kernel compilation.

bash — Setup vLLM Virtual Environment

# 1. Install uv / 安裝 uv
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

# 2. Clone vLLM & create Python 3.12 venv / 建立虛擬環境
mkdir -p ~/Projects && cd ~/Projects
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv venv --python 3.12
source .venv/bin/activate

# 3. Install vLLM & JIT build tools / 安裝 vLLM 與編譯工具
VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto
uv pip install ninja flashinfer-python

安裝完成後，執行自檢腳本驗證 Port 8001、GB10 CUDA GEMM、torch.ops._C.rms_norm 原生算子與 V1 AsyncLLM 引擎：

After installation, run the self-verification suite to validate Port 8001 binding, GB10 CUDA GEMM, the native torch.ops._C.rms_norm custom op, and V1 AsyncLLM engine imports:

brian@spark-fb30: ~/Projects/vllm — test_self_check.py

$ .venv/bin/python test_self_check.py
============================================================
🔍 Running vLLM Self-Verification Test...
============================================================
📡 [1/3] Testing Network Interface Binding...
✅ Port 8001 is FREE and ready to bind on 0.0.0.0.
⚡ [2/3] Testing CUDA GEMM & vLLM Custom Ops...
✅ CUDA GEMM passed on NVIDIA GB10.
✅ vLLM custom op (torch.ops._C.rms_norm) matches reference.
⚙️ [3/3] Testing vLLM V1 Engine & Server Components...
✅ vLLM 0.28.1rc1.dev152+g810bc3250: AsyncEngineArgs & AsyncLLM loaded.
============================================================
🎉 ALL SELF-VERIFICATION TESTS PASSED SUCCESSFULLY!

03 Step 2：撰寫 GB10 專屬啟動腳本 (serve_nvfp4.sh) Step 2: Creating the GB10 Launch Script (serve_nvfp4.sh)

⚠️ GB10 必避的三大坑 / Three Critical Pitfalls on GB10

• 坑 1（找不到 ninja）： FlashInfer JIT 需要呼叫虛擬環境內的 ninja。透過 systemd 啟動時，必須把 .venv/bin 明確放進 PATH。
• 坑 2（17 個 nvcc 併發編譯導致 OOM Exit 137）： FlashInfer JIT 預設會開滿 CPU 核心數同時編譯 CUTLASS FP4 範本，瞬間吃光記憶體被 Kill。務必設定 export MAX_JOBS=2，首次編完後即會快取。
• 坑 3（Port 8000 衝突）： 若機器上已有其他 Uvicorn 服務佔用 8000，systemd 重啟循環會瞬間把日誌撐爆數十 MB。此處我們固定使用 Port 8001。

• Pitfall 1 (Missing ninja in PATH): FlashInfer JIT requires ninja from the virtualenv. When running under systemd, explicitly prepend .venv/bin to PATH.
• Pitfall 2 (OOM Exit 137 from 17 parallel nvcc jobs): By default, FlashInfer spawns one nvcc process per CPU core to compile heavy CUTLASS FP4 templates, exhausting memory. Always set export MAX_JOBS=2; compiled kernels are cached afterward.
• Pitfall 3 (Port 8000 Conflict): Avoid colliding with existing local services on port 8000 by binding vLLM to Port 8001.

~/Projects/vllm/serve_nvfp4.sh

#!/usr/bin/env bash
# GB10 vLLM Launch Script — unsloth/Qwen3.8-27B-NVFP4
set -euo pipefail
cd "$(dirname "$0")"

# Key 1: Prepend .venv/bin so FlashInfer JIT finds ninja & compiler tools
export PATH="/home/brian/Projects/vllm/.venv/bin:$PATH"
export VLLM_LOGGING_LEVEL=INFO

# Key 2: Cap parallel nvcc jobs at 2 to prevent OOM (exit 137) on CUTLASS FP4 templates
export MAX_JOBS=2

exec .venv/bin/vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --host 0.0.0.0 \
  --port 8001 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.75 \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice \
  --distributed-executor-backend mp \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

04 實測揭秘：MTP n=3 vs n=5 vs 純 Prefix Cache 效能對決 Benchmark Deep-Dive: MTP n=3 vs. n=5 vs. Prefix Cache Only

Qwen3.8-27B 採用 Attention + Mamba 混合架構 並內建 MTP (Multi-Token Prediction) 預測頭。我們在 GB10 上模擬真實的 Hermes Agent 負載（13,000 tokens 系統提示詞 + 250 個工具定義，連續 6 輪多輪對話），實測發現將 num_speculative_tokens 從 5 調降為 3，切除命中率極低的第 4、5 個預測位置後，生成速度反而再提升 8%（總計較無 MTP 提升 +30%）：

Qwen3.8-27B uses a Hybrid Attention + Mamba architecture with built-in Multi-Token Prediction (MTP) heads. Benchmarking a realistic Hermes Agent workload on GB10 (13K-token system prompt with 250 tool definitions across 6 turns) revealed that tuning num_speculative_tokens from 5 down to 3 eliminates low-acceptance draft positions (Pos 4 & 5), boosting decode speed by another 8% (+30% faster than no MTP):

▲ 圖 2：GB10 上 Qwen3.8-27B-NVFP4 的投機解碼速度對比與各位置 Draft Token 接受率分析 / Fig 2: Speculative decoding throughput comparison and per-position draft acceptance rates on GB10

Configuration (13K Prompt, 6-Turn Agent Test)	Decode Speed (Median)	Later-Turn TTFT	Draft Acceptance Rate	Mean Acceptance Length
No MTP (Prefix Cache Only)	10.2 tok/s	0.82 s	—	—
MTP num_speculative_tokens: 5	12.2 tok/s (+20%)	1.68 s	29% ~ 71% (Volatile)	2.4 ~ 4.5 tokens
MTP num_speculative_tokens: 3 (Optimal ⭐)	13.2 tok/s (+30%)	1.78 s	53% ~ 82% (Stable High)	2.6 ~ 3.5 tokens

05 Step 3：Systemd 常駐服務化與 Logrotate 自動輪替 Step 3: Systemd User Service & Logrotate Automation

建立 ~/.config/systemd/user/vllm.service 讓服務在背景常駐，並搭配 logrotate 限制日誌在 50 MB 內自動壓縮輪替：

Create ~/.config/systemd/user/vllm.service to keep vLLM running in the background with a 900s startup timeout for CUDA Graph compilation, and configure logrotate to cap log size at 50 MB:

~/.config/systemd/user/vllm.service & logrotate

mkdir -p ~/.config/systemd/user ~/.config/logrotate

cat > ~/.config/systemd/user/vllm.service << 'EOF'
[Unit]
Description=vLLM OpenAI-Compatible Server — Qwen3.8-27B-NVFP4 (GB10)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=%h/Projects/vllm
ExecStart=%h/Projects/vllm/serve_nvfp4.sh
Restart=on-failure
RestartSec=15
TimeoutStartSec=900
StandardOutput=append:%h/Projects/vllm/vllm-service.log
StandardError=append:%h/Projects/vllm/vllm-service.log

[Install]
WantedBy=default.target
EOF

cat > ~/.config/logrotate/vllm.conf << EOF
$HOME/Projects/vllm/vllm-service.log {
    size 50M
    rotate 3
    compress
    missingok
    notifempty
    copytruncate
}
EOF

(crontab -l 2>/dev/null; echo '0 * * * * /usr/sbin/logrotate -s $HOME/.config/logrotate/state $HOME/.config/logrotate/vllm.conf') | crontab -
systemctl --user daemon-reload && systemctl --user enable --now vllm.service

vllm-service.log — Production Startup & SpecDecoding Telemetry

(Worker pid=2349427) INFO [model_runner.py:404] Model loading took 22.21 GiB memory and 148.951431 seconds
(Worker pid=2349427) INFO [gpu_worker.py:622] Available KV cache memory: 64.06 GiB
(EngineCore pid=2349333) INFO [kv_cache_utils.py:2020] GPU KV cache size: 1,740,978 tokens, Maximum concurrency for 262,144 tokens per request: 6.64x
(APIServer pid=2490488) INFO: Application startup complete.
(APIServer pid=2490488) INFO [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.45, Per-position acceptance rate: 0.921, 0.842, 0.684, Avg Draft acceptance rate: 81.6%

06 Step 4：串接 Hermes Agent（主力模型 + 本地壓縮 + 自動備援） Step 4: Connecting Hermes Agent (Primary Model + Local Compression + Fallback)

編輯 ~/.hermes/config.yaml，將主力模型（model）與長對話摘要壓縮（auxiliary.compression）全部指向本機 http://127.0.0.1:8001/v1，並設定雲端 Codex 為自動 Fallback：

Edit ~/.hermes/config.yaml to point both the primary model and context compression engine (auxiliary.compression) to your local vLLM endpoint at http://127.0.0.1:8001/v1, with cloud Codex configured as an automatic fallback:

~/.hermes/config.yaml

# 1. Primary Model: Local vLLM (:8001) with full 256K context
model:
  default: unsloth/Qwen3.8-27B-NVFP4
  provider: custom
  base_url: http://127.0.0.1:8001/v1
  api_key: local-vllm
  context_length: 262144

# 2. Auxiliary Models: Local Ollama for Vision & Local vLLM for Context Compression
auxiliary:
  vision:
    provider: custom
    model: qwen2.5vl:7b
    base_url: http://localhost:11434/v1
    api_key: ollama
  compression:
    provider: custom
    model: unsloth/Qwen3.8-27B-NVFP4
    base_url: http://127.0.0.1:8001/v1
    api_key: local-vllm
    context_length: 131072

# 3. Custom Provider Menu & Cloud Fallback
custom_providers:
  - name: Local vLLM (GB10)
    base_url: http://127.0.0.1:8001/v1
    api_key: local-vllm
    model: unsloth/Qwen3.8-27B-NVFP4

fallback_model:
  provider: openai-codex
  model: gpt-5.6-sol
  base_url: https://chatgpt.com/backend-api/codex

07 同場加映：在 GB10 上突破 60+ tok/s 的 MoE NVFP4 模型推薦 Bonus: Achieving 60+ tok/s on GB10 with MoE NVFP4 Models

由於 GB10 的記憶體頻寬約為 273 GB/s，27B Dense 模型每生成 1 個 token 都必須讀取整份 22.2 GiB 權重（物理極限約 13~14 tok/s）。若想體驗 50 ~ 85 tok/s 的極速推論，可改用每次僅激活 3B 參數的 MoE + NVFP4 模型：

Because GB10's unified memory bandwidth is ~273 GB/s, a 27B dense model must read all 22.2 GiB of weights per token step (~13–14 tok/s ceiling). To achieve 50–85 tok/s, switch to an MoE + NVFP4 model that activates only 3B parameters per token:

Model ID (Hugging Face)	Architecture	Weight Size	Active Params	Est. Speed (GB10)	Compatibility & Notes
unsloth/Qwen3.8-27B-NVFP4	Dense + MTP	22.2 GiB	27B (100%)	~13.2 tok/s	Deep dense reasoning, native qwen3 / qwen3_xml
nvidia/Qwen3.6-35B-A3B-NVFP4 ⭐	MoE + MTP	21.8 GiB	3B (A3B)	~50 - 65 tok/s	100% drop-in compatible with existing Qwen flags, 4x faster
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4	Mamba-2 + MoE + MTP	20.1 GiB	3B (A3B)	~60 - 85 tok/s	Fastest architecture on GB10, minimal KV cache footprint
nvidia/Gemma-4-26B-A4B-NVFP4	MoE (Multimodal)	17.5 GiB	4B (A4B)	~35 - 50 tok/s	Smallest memory footprint, multimodal vision support

Built & Benchmarked on NVIDIA Grace Blackwell GB10 (DGX Spark) • Powered by vLLM V1 & Hermes Agent