How to Deploy Qwen2.5-VL-3B with vLLM

# Create a working directory (choose your preferred location)

# Create virtual environment with Python 3.12
# Make sure you have Python 3.12 installed: sudo apt install python3.12 python3.12-venv
python3.12 -m venv venv

# Activate virtual environment
source venv/bin/activate

# Verify Python version (should be 3.12.x)
python --version

# Upgrade pip
pip install --upgrade pip

# Install vLLM (this takes ~5-10 minutes)
pip install vllm

# Make sure you're in your working directory and activate virtual environment
source venv/bin/activate

# Launch vLLM serve (full native context)
vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --max-num-batched-tokens 32768 \
  --gpu-memory-utilization 0.9 \
  --port 8000

curl http://localhost:8000/v1/models

{
  "object": "list",
  "data": [{
    "id": "Qwen/Qwen2.5-VL-3B-Instruct",
    "object": "model",
    "created": 1770105752,
    "owned_by": "vllm",
    "root": "Qwen/Qwen2.5-VL-3B-Instruct",
    "parent": null,
    "max_model_len": 32768
  }]
}

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-3B-Instruct",
    "prompt": "Hello, how are you?",
    "max_tokens": 50
  }'

# Encode video to base64
VIDEO_BASE64=$(base64 -w 0 /path/to/your/video.mp4)

# Time the request and measure latency
time curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"Qwen/Qwen2.5-VL-3B-Instruct\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {
          \"type\": \"video_url\",
          \"video_url\": {
            \"url\": \"data:video/mp4;base64,\$VIDEO_BASE64\"
          }
        },
        {
          \"type\": \"text\",
          \"text\": \"Describe the user actions in this video in a sequential list.\"
        }
      ]
    }],
    \"max_tokens\": 1024
  }"

1. A soccer player in a red jersey is standing on the field, clapping his hands together.
2. The player continues to clap his hands, looking around the field.
3. The player then turns his head to the side and looks at something off-screen.