spark-vllm-docker

Project Overview

This repository contains Docker configurations and startup scripts optimized for running vLLM on NVIDIA DGX Spark, from single-node to multi-node clusters using Ray or vLLM's native PyTorch distributed mode. It supports InfiniBand/RDMA (NCCL), custom environment configuration, and high-performance model loading through fastsafetensors and InstantTensor. Cluster setups include direct connections between dual Sparks, QSFP/RoCE switch configurations, and 3-node mesh configurations.

Key Features

• Multi-node inference: Single-node and multi-node cluster deployment
• High-performance model loading: Optimized with fastsafetensors and InstantTensor
• InfiniBand/RDMA support: High-speed interconnect via NCCL
• Multiple cluster configurations: Dual Spark direct, QSFP/RoCE switch, 3-node mesh
• Flexible build options: Precompiled wheels, rebuilding vLLM or FlashInfer from source

Target Audience

• Developers using NVIDIA DGX Spark for LLM inference
• AI engineers needing multi-node distributed inference
• Researchers interested in high-performance vLLM deployments

Quick Start

1. Default image: eugr/spark-vllm:latest
2. Run build-and-copy.sh to build and start the container
3. For multi-node, configure passwordless SSH between Sparks
4. Refer to the Networking Guide

Core Advantages

• Optimized vLLM deployment for DGX Spark
• Multi-node distributed inference support
• Flexible build options for custom vLLM versions
• Community-driven open-source project (not affiliated with NVIDIA)

System Requirements

• Hardware: NVIDIA DGX Spark
• Network: InfiniBand/RDMA (optional), passwordless SSH (multi-node)