Prepared by: Aetherix B.V. — Trusted Jetson Hardware Provider for Edge AI

Test platform: 4× NVIDIA DGX Spark (GB10) · Model: DeepSeek-V4.1-Flash (763B) · Report date: September 2026

Contents

1. Introduction

DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters.

Its Hugging Face checkpoint contains 763B parameters: 552B backbone + 196B Engram conditional memory (per the model’s technical report) + ~15B for the vision encoder, MLP projector, and DSpark draft model.

The model ships with MXFP8 dense + MXFP4 expert weight quantization and FP4 (E2M1) KV cache, and supports contexts of up to one million tokens.

This report describes the deployment of this model on 4× NVIDIA DGX Spark (GB10) using tensor parallelism (TP=4).

DGX Spark is a mini supercomputer with 128 GB unified LPDDR5X memory and 273 GB/s bandwidth. Running a 763B data center model on this hardware is made possible by multi-node tensor parallelism and especially the Engram-on-disk technique.

Two aspects make this work worth documenting:

  • The model is data center class. DeepSeek-V4.1-Flash is normally served on DGX-B300 and similar-class hardware. 4× DGX Spark represents the scalable edge of the architecture.
  • 7 SM 12.1a-specific patches. The stock vLLM nightly image has missing or broken code paths for SM 12.1a (GB10) — Engram-on-disk, FlashInfer sparse attention, SWA block size, attention page sizes — which are fixed with community patches.

This is a deployment guide, not a benchmark comparison. Benchmark results are presented in Section 6.

2. Model Architecture

2.1 Overview

DeepSeek-V4.1-Flash introduces an architecture that dramatically reduces the KV cache footprint. Key specifications:

Component Value
Backbone parameters 552B
Total checkpoint parameters 763B (HF verified)
Active parameters per token (prefill) 8B
Active parameters per token (decode) 16B
Engram conditional memory 196B (sparsely accessed via token-based lookup)
Shared / routed experts 1 / 384 (6 active per token)
Maximum context 1,000,000 tokens
Weight quantization MXFP8 dense + MXFP4 experts (block 32×32, ue8m0 scale)
KV cache quantization FP4 (E2M1, runtime — CSA2 architectural feature)
Checkpoint size 510 GB (48 safetensors files — below theoretical 763 GB due to FP4 expert weights and mixed precision)

2.2 Causal Encoder-Decoder (CED) Architecture

The model is a 40-layer Transformer organized as a 20-layer causal encoder followed by a 20-layer decoder. The key advantage of CED: the decoder’s global KV cache is projected from the encoder’s final hidden states rather than derived from each decoder layer’s own hidden states. This allows activating only 8B parameters during prefill and 16B during decode — significant cost efficiency for input-heavy agentic workloads.

SWA Bounded Replay reconstructs missing KV states by replaying only the most recent n_win tokens instead of storing the KV states required for Sliding Window Attention (SWA) in persistent storage. This eliminates the need to write SWA KV states to persistent storage and reduces the persistent KV cache storage footprint to roughly 1/8 of that of DeepSeek-V4-Flash.

2.3 Compressed Sparse Attention 2 (CSA2)

CSA2 assigns each attention layer one of three static modes — Full, Reindex, or Reuse — to share main KV and indexer K across layers and reuse Top-K sparse-attention indices. In the decoder, a Hierarchical Sparse Indexer further restricts later indexing layers to a candidate pool constructed by the first Full Mode layer, bounding deeper indexer cost independently of context length.

Combined with FP4 main KV caching (E2M1 format, one E4M3 scale per 16 channels), these designs reduce the global KV cache footprint to 890 bytes per token — roughly 1/4 of DeepSeek-V4-Flash.

2.4 Engram Conditional Memory

The 196B Engram layer is sparsely accessed via token-based lookup. This deployment uses Engram-on-disk mode: Engram rows are stored on each node’s local disk rather than in GPU memory, and staged into GPU memory on demand. This enables the 763B model to fit within 4× 128 GB (512 GB total) unified memory.

For the embedding lookup mechanism, memory budget and interpretation of the offloading results, see Conditional Memory and Offloading in LLM Inference.

2.5 DSpark Speculative Decoding

DSpark is a speculative decoding mechanism using semi-autoregressive draft generation with confidence-scheduled verification. This deployment uses k=5 depth — 5 tokens are predicted ahead and verified at each decode step. Verified tokens are appended to the output; rejected ones are regenerated.

3. Hardware

Component Value
GPU Blackwell architecture (GB10), SM 12.1a (CC 12.1), 48 SMs
GPU memory 128 GB unified LPDDR5X (shared CPU+GPU)
Memory bandwidth ~273 GB/s (unified)
FP4 peak (with sparsity) ~1 PFLOP
CPU 20-core Arm (10× Cortex-X925 + 10× Cortex-A725)
Node interconnect NVIDIA ConnectX-7, 200 Gb/s RDMA (QSFP)
Storage 4× NVMe SSD (4× 4 TB across 4 nodes)
Network topology 200 GbE switch (no NVLink; all-reduce over Ethernet)

Critical architectural fact: There is no NVLink between DGX Sparks. TP=4 in this deployment is multi-node tensor parallelism over a ConnectX-7 200 GbE network — every all-reduce operation crosses the Ethernet fabric via a 200 GbE switch.

4. vLLM Build and Patch Chain

The stock vLLM nightly image has missing code paths for SM 12.1a (GB10) in the DeepSeek-V4.1-Flash code paths. This section documents the build chain and 7 patches applied to create the custom Docker image (vllm-dsv41:latest, 33.8 GB).

4.1 Base Image and vLLM Branch

  • Base image: vllm/vllm-openai:nightly-8a728663c1c3eeace834a95f5654fa653cc1998c
  • vLLM branch: Checked out via build/fetch_vllm_branch.sh, commit e47aa780b
  • Source repo: tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark (boot10 config)

4.2 Overlay Build Chain

The build consists of 5 overlay layers and a patch layer. Each overlay builds on top of the previous one:

Step Name Content
1 C extension _C_stable_libtorch + _moe_C_stable_libtorch — compiled for SM 12.1a (build_stable_ext.sh in v41build container)
2 overlay1 Base image + vLLM branch tree (vllm/vllm/) + 2 .so files
3 overlay3 FlashInfer 0.7.0rc1 (07869c61) compiled
4 overlay4 CUTLASS mxfp8_gemm_sm120 prebuilt (MAX_JOBS=4)
5 overlay5 sparse_mla_sm120 prebuilt + verify (HIT/HIT)
6 patch layer 7 patch files baked into image + ENTRYPOINT [] added

Important: The Dockerfile must use COPY vllm/vllm/ (not vllm/) — vllm/ is a git repo, the Python package is under vllm/vllm/.

4.3 Patches

The following 7 patches are baked into the image:

# File Mount path Function
1 engram.py models/deepseek_v4_1/common/engram.py Engram-on-disk, rank-offset fix
2 model_state.py models/deepseek_v4_1/nvidia/model_state.py Engram staging before forward (CUDA graph safe)
3 weight_utils.py model_executor/model_loader/weight_utils.py Skip Engram tables on load
4 attention.py models/deepseek_v4_1/attention.py SM 12.1a page sizes
5 flashinfer_sparse.py models/deepseek_v4_1/nvidia/flashinfer_sparse.py 64-state pages
6 sparse_swa.py v1/attention/backends/mla/sparse_swa.py SWA block size hook
7 sparse_attn_indexer.py model_executor/layers/sparse_attn_indexer.py SM 12.1a top_k_per_row_decode

4.4 Optimizations

  • OMP_NUM_THREADS=1 — added to the recipe (reduces spin-wait contention)
  • Build artifacts cleaned — overlay1/3/4/5, base image, build cache deleted (~258 GB reclaimed)

5. Configuration

5.1 sparkrun Recipe

The sparkrun recipe (deepseek-v41-flash-tp4.yaml) contains the following parameters:

Parameter Value Description
tensor_parallel 4 4 nodes × 1 GPU
gpu_memory_utilization 0.80 80% GMU
max_model_len 300000 300K context (below 1M, memory constrained)
max_num_seqs 8 Maximum 8 concurrent sequences
max_num_batched_tokens 8192 Prefill batch size
block_size 128 KV cache block size
distributed-executor-backend mp Multiprocess backend

5.2 Speculative Decoding Configuration 

{
  "method": "dspark",
  "num_speculative_tokens": 5,
  "draft_sample_method": "probabilistic",
  "rejection_sample_method": "block",
  "enable_adaptive_verification": false
}

With DSpark k=5 depth, 5 tokens are predicted ahead at each decode step.

5.3 Engram-on-disk Configuration 

DSV41_ENGRAM_DISK=1
DSV41_ENGRAM_DISK_THREADS=32
DSV41_ENGRAM_DISK_CHUNK=16

Engram rows are stored on disk, read in parallel with 32 threads, and staged into GPU memory in chunks of 16.

5.4 Launch Command 

sparkrun run /home/nvidia/.cordatus-sparkrun/recipes/deepseek-v41-flash-tp4.yaml \
  --hosts 192.168.1.153,192.168.1.147,192.168.1.166,192.168.1.148 \
  --foreground

5.5 Serving Parameters

The model is served with thinking mode disabled ("thinking": false), tool calling and multimodal (vision) support enabled:

  • --default-chat-template-kwargs '{"thinking": false}'
  • --tool-call-parser deepseek_v41 --enable-auto-tool-choice
  • --reasoning-parser deepseek_v41
  • --limit-mm-per-prompt '{"image":4}'
  • --compilation-config with CUDA graphs (FULL_AND_PIECEWISE mode)

6. Performance Results

6.1 Benchmark Table

Measurements were taken with CordatusAI/llm-benchmark (2nd run, after JIT warmup).

Concurrency TTFT (ms) ITL (ms) TPS (tok/s) Latency (s) Throughput (RPS)
1 271.63 32.39 29.48 4.39 0.23
2 395.75 45.12 21.32 6.13 0.16
4 577.41 73.86 13.09 9.96 0.10
8 805.89 110.28 8.79 14.81 0.07

6.2 Charts

TTFT

ITL

TPS

Latency

Throughput

6.3 Assessment

  • 29.5 tok/s at C=1 is sufficient for interactive use, above the reading speed threshold (~15 tok/s).
  • TTFT rises as expected with concurrency (272→806 ms, ~3x) — prefill is compute-bound.
  • TPS decline reflects growing memory bandwidth contention as the GPU approaches saturation (29.5→8.8 tok/s, ~70% drop).
  • Max C = 2 at the Benchmark Explorer’s default targets (TTFT≤1000ms, TPS≥20) — chat capacity 8, agentic capacity 3.

6.4 DSpark Acceptance

The effectiveness of DSpark speculative decoding is measured by acceptance ratio and draft rate:

Prompt type Mean acceptance Draft rate
Prose (bench) 2.1–2.3 22%
Coding 4.9–5.3 80–86%

The high acceptance of DSpark k=5 on coding prompts (4.9-5.3 / 5) is driven by code having more structural, patterned, and predictable token sequences, which contributes to this high acceptance rate. TPS also rises in this type of response.

7. Comparative Analysis

7.1 Comparison with DGX-B300

DeepSeek-V4.1-Flash has also been measured on the LLM Inference Benchmark Explorer on both DGX-B300 (8× Blackwell Ultra, TP=4) and 4× DGX Spark configurations.

These results should not be evaluated as a direct one-to-one hardware comparison. The speculative decoding parameters and supported context lengths differ between the two configurations: the Spark side uses k=5 and 300K context, while the B300 side uses k=3 and 1M context. The measured TPS difference therefore reflects the combined effect of hardware, memory system, and inference server configuration.

Concurrency 4× Spark TP4 TPS B300 TP4 TPS B300 / Spark
1 29.48 284.54 9.65×
2 21.32 294.56 13.82×
4 13.09 252.60 19.30×
8 8.79 209.41 23.83×

In this measurement set, the B300 configuration produced approximately 9.65× the aggregate TPS of the Spark configuration at C=1, and approximately 23.83× at C=8.

The fundamental hardware-level differences are as follows:

Feature DGX-B300 4× DGX Spark
Memory type HBM3e LPDDR5X unified memory
Memory bandwidth Very high, TB/s class ~273 GB/s / system
GPU interconnect High-bandwidth NVLink-based connection 200 GbE RDMA
GPU architecture Blackwell Ultra GB10
Scaling target Data center / high-density inference Desktop/edge and development

Especially in large MoE models, not only total memory capacity but also memory bandwidth and inter-GPU communication capacity significantly affect performance. Therefore, the fact that the model can run on 4× DGX Spark does not mean that the same model will reach a similar throughput level on DGX-B300.

Conclusion: 4× DGX Spark can be used to run large-scale MoE models and can be considered for development, prototyping, model validation, and deployment scenarios requiring low/medium concurrency. However, the measured throughput results should not be interpreted as replacing data center class systems such as DGX-B300. These two platforms have different usage targets in terms of memory bandwidth, interconnect, and scaling capacity.

8. SLO and Capacity

At the Benchmark Explorer’s default targets (TTFT≤1000ms, TPS≥20 tok/s), this configuration yields Max C = 2:

SLO Threshold C=2 status
TTFT ≤ 1000 ms 396 ms ✓
TPS ≥ 20 tok/s 21.32 tok/s ✓

Warning: At C=4, TPS drops to 13.09, falling below the 20 tok/s threshold. For interactive chat services, 1-2 concurrent users are recommended; for higher loads, data center hardware (B300/GB300) should be preferred.

9. Conclusion

  1. DeepSeek-V4.1-Flash (763B) can run on 4× DGX Spark — demonstrating the scalability ceiling of office-friendly mini supercomputers.
  2. 7 SM 12.1a patches are mandatory — the stock vLLM nightly image has missing code paths for GB10 (SM 12.1a) in Engram-on-disk, FlashInfer sparse attention, and SWA paths.
  3. Engram-on-disk enables the model to fit in memory — offloading the 196B conditional memory to disk allows the 763B model to fit within 512 GB total unified memory.
  4. DSpark k=5 is effective on coding prompts with 80-86% draft rate; on prompts written in natural language in a plain text/prose form, this drops to 22%.
  5. 29.5 tok/s at C=1 is sufficient for development and prototyping; for production serving, B300-class hardware is ~10× faster.
  6. This configuration is a development platform, not a data center alternative. The ability to run a 763B model on office hardware demonstrates the boundaries of edge and on-premises deployment.

Appendix A: Docker Image Build Instructions

This appendix contains full instructions to build the vllm-dsv41:latest image from scratch. All commands are run on the head node (192.168.1.153).

A.1 Clone the Repo 

cd /home/nvidia
git clone https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark.git
cd DeepSeek-V4.1-Flash-vLLM-DGX-Spark

A.2 Checkout vLLM Branch 

bash build/fetch_vllm_branch.sh /home/nvidia/vllm
# commit e47aa780b
rm -f vllm && cp -a /home/nvidia/vllm ./vllm

A.3 Build C Extensions 

docker run -d --name v41build --gpus all --network host --entrypoint sleep \
  -v /tmp/v41build-src:/src vllm/vllm-openai:nightly-8a728663c1c3eeace834a95f5654fa653cc1998c 7200
docker exec v41build pip install cmake ninja -q
bash build/build_stable_ext.sh
docker cp v41build:/src/build/_C_stable_libtorch.abi3.so build/
docker cp v41build:/src/build/_moe_C_stable_libtorch.abi3.so build/

A.4 Overlay Chain 

# overlay1: base + vLLM tree + 2 .so files
docker build -t vllm-dsv41:overlay1 -f build/Dockerfile.overlay1 .

# overlay3: FlashInfer 0.7.0rc1
bash build/build_overlay3.sh

# overlay4: CUTLASS mxfp8_gemm_sm120
bash build/build_overlay4.sh

# overlay5: sparse_mla_sm120 + verify
cp build/prewarm5.py build/verify5.py /tmp/
bash build/build_overlay5.sh

A.5 Patch Layer 

FROM vllm-dsv41:overlay5
COPY DeepSeek-V4.1-Flash-vLLM-DGX-Spark/patch/engram.py /usr/local/lib/python3.12/dist-packages/vllm/models/deepseek_v4_1/common/engram.py
COPY DeepSeek-V4.1-Flash-vLLM-DGX-Spark/patch/model_state.py /usr/local/lib/python3.12/dist-packages/vllm/models/deepseek_v4_1/nvidia/model_state.py
COPY DeepSeek-V4.1-Flash-vLLM-DGX-Spark/patch/weight_utils.py /usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/weight_utils.py
COPY DeepSeek-V4.1-Flash-vLLM-DGX-Spark/patch/attention.py /usr/local/lib/python3.12/dist-packages/vllm/models/deepseek_v4_1/attention.py
COPY DeepSeek-V4.1-Flash-vLLM-DGX-Spark/patch/flashinfer_sparse.py /usr/local/lib/python3.12/dist-packages/vllm/models/deepseek_v4_1/nvidia/flashinfer_sparse.py
COPY DeepSeek-V4.1-Flash-vLLM-DGX-Spark/patch/sparse_swa.py /usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/sparse_swa.py
COPY DeepSeek-V4.1-Flash-vLLM-DGX-Spark/patch/sparse_attn_indexer.py /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/sparse_attn_indexer.py
ENTRYPOINT []

cd /home/nvidia && docker build -t vllm-dsv41:latest -f Dockerfile.patches .

A.6 Worker Distribution

sparkrun auto-distributes the image, but if manual distribution is needed:

docker save vllm-dsv41:latest | ssh nvidia@192.168.1.147 docker load
docker save vllm-dsv41:latest | ssh nvidia@192.168.1.166 docker load
docker save vllm-dsv41:latest | ssh nvidia@192.168.1.148 docker load

Appendix B: Engram Row Ranges

In Engram-on-disk mode, each rank (node) holds different row ranges on its local disk:

Rank Node Layer 1 Rows Layer 14 Rows
0 153 (head) [0, 96000564) [0, 96003054)
1 147 [96000564, 192001740) [96003054, 192007016)
2 166 [192001740, 288003654) [192007016, 288011564)
3 148 [288003654, 384006168) [288011564, 384016682)

A total of ~384M rows, evenly split across 4 ranks.

Appendix C: Verification Commands

API Health 

curl http://192.168.1.153:8000/v1/models

Simple Test 

curl http://192.168.1.153:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4.1-flash","max_tokens":50,
       "messages":[{"role":"user","content":"Count from 1 to 10."}]}'

Aetherix B.V. — aetherix.com · info@aetherix.com

Oude Apeldoornseweg 41, A1.08, 7333 NR Apeldoorn, Netherlands

This report was generated from measurements produced with the CordatusAI/llm-benchmark tool. Deployment recipe adapted from tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark boot10 config. Report date: September 2026