How to Deploy InternVL3.5-8B with vLLM

# Create a working directory (choose your preferred location)

# Create virtual environment with Python 3.12
# Make sure you have Python 3.12 installed: sudo apt install python3.12 python3.12-venv
python3.12 -m venv venv

# Activate virtual environment
source venv/bin/activate

# Verify Python version (should be 3.12.x)
python --version

# Upgrade pip
pip install --upgrade pip

# Install vLLM (this takes ~5-10 minutes)
pip install vllm

# Make sure you're in your working directory and activate virtual environment
source venv/bin/activate

# Launch vLLM serve (full native context)
vllm serve OpenGVLab/InternVL3_5-8B \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --max-num-batched-tokens 32768 \
  --gpu-memory-utilization 0.9 \
  --port 8000

curl http://localhost:8000/v1/models

{
  "object": "list",
  "data": [{
    "id": "OpenGVLab/InternVL3_5-8B",
    "object": "model",
    "created": 1770105248,
    "owned_by": "vllm",
    "root": "OpenGVLab/InternVL3_5-8B",
    "parent": null,
    "max_model_len": 32768
  }]
}

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "OpenGVLab/InternVL3_5-8B",
    "prompt": "Hello, how are you?",
    "max_tokens": 50
  }'

{
  "choices": [{
    "text": " I'm good, I just wanted to do a little experiment..."
  }]
}

# Encode video to base64
VIDEO_BASE64=$(base64 -w 0 /path/to/your/video.mp4)

# Time the request and measure latency
time curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"OpenGVLab/InternVL3_5-8B\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {
          \"type\": \"video_url\",
          \"video_url\": {
            \"url\": \"data:video/mp4;base64,\$VIDEO_BASE64\"
          }
        },
        {
          \"type\": \"text\",
          \"text\": \"Describe the user actions in this video in a sequential list.\"
        }
      ]
    }],
    \"max_tokens\": 1024
  }"

Certainly! Here is a detailed description of the user actions in a sequential list based on the provided video frames:

1. **Initial Clapping**:
    - The individual starts clapping their hands together.

2. **Separating Hands**:
    - The person separates their hands slightly while continuing to clap.

3. **Double Clapping Motion**:
    - The clapping motion continues in a rhythmic double clap (clapping each hand separately).

4. **Movement to the Right**:
    - The individual begins to move to the right side of the frame while continuing to clap.
...