Deployment Troubleshooting Guide

Comprehensive troubleshooting guide for Hyperscape deployments across Cloudflare Pages, Railway, and Vast.ai.

Cloudflare Pages
Railway
Vast.ai
Database Issues
Streaming Issues
Security & Secrets

Cloudflare Pages

Build Fails with Module Resolution Errors

Symptoms:

Failed to resolve module specifier "vite-plugin-node-polyfills/shims/buffer"

Cause: Vite polyfills shims not resolving to dist files. Fix (commit e012ed2):

// vite.config.ts
resolve: {
  alias: {
    'vite-plugin-node-polyfills/shims/buffer': 'vite-plugin-node-polyfills/dist/shims/buffer',
    'vite-plugin-node-polyfills/shims/global': 'vite-plugin-node-polyfills/dist/shims/global',
    'vite-plugin-node-polyfills/shims/process': 'vite-plugin-node-polyfills/dist/shims/process',
  }
}

CSP Errors Loading Google Fonts

Symptoms:

Refused to load the stylesheet 'https://fonts.googleapis.com/...' because it violates CSP

Cause: Content Security Policy doesn’t allow Google Fonts. Fix (commit e012ed2):

// public/_headers
/*
  Content-Security-Policy: style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; font-src 'self' https://fonts.gstatic.com

CORS Errors Loading Assets from R2

Symptoms:

Access to fetch at 'https://assets.hyperscape.club/...' from origin 'https://hyperscape.gg' has been blocked by CORS policy

Cause: R2 bucket CORS not configured. Fix (commit 143914d):

# Run CORS configuration script
bash scripts/configure-r2-cors.sh

# Or manually with wrangler
wrangler r2 bucket cors set hyperscape-assets --cors-config cors-config.json

CORS config format:

{
  "allowed": {
    "origins": ["*"],
    "methods": ["GET", "HEAD"],
    "headers": ["*"]
  },
  "exposed": ["ETag"],
  "maxAge": 3600
}

Multi-line Commit Messages Break Deploy

Symptoms:

Error: Invalid commit message format

Cause: Workflow doesn’t handle multi-line commit messages. Fix (commit 3e4bb48):

# .github/workflows/deploy-pages.yml
- name: Get commit message
  id: commit
  run: |
    COMMIT_MSG=$(git log -1 --pretty=%B | head -1)
    echo "message=$COMMIT_MSG" >> $GITHUB_OUTPUT

Railway

Database Connection Fails

Symptoms:

Error: connect ECONNREFUSED

Cause: DATABASE_URL not set or incorrect. Fix:

Get DATABASE_URL from Railway dashboard (PostgreSQL plugin)
Add to Railway environment variables
Redeploy

Verify:

# In Railway logs
grep "DATABASE_URL" /app/logs/server.log
# Should show: configured (not NOT SET)

CSRF Errors from Cloudflare Pages

Symptoms:

403 Forbidden: CSRF token validation failed

Cause: Cross-origin requests from Cloudflare Pages to Railway server. Fix: CSRF validation now skipped for known clients (commit 8626299):

// Server already protected by Origin header + JWT
// Skip CSRF for cross-origin requests from trusted domains

Port Conflicts

Symptoms:

Error: listen EADDRINUSE: address already in use :::5555

Cause: Port 5555 already in use. Fix:

# Railway uses PORT environment variable
# Don't hardcode port 5555 in production
const port = process.env.PORT || 5555;

Vast.ai

Xorg Fails to Start

Symptoms:

[deploy] Xorg failed to start (check /var/log/Xorg.99.log)

Cause: NVIDIA driver issues, display already in use, or missing dependencies. Fix:

# Check NVIDIA driver
nvidia-smi

# Install Xorg components
apt-get install -y xserver-xorg-core

# Kill existing display servers
pkill -9 Xorg; pkill -9 Xvfb
sleep 2

# Check Xorg logs
cat /var/log/Xorg.99.log

Fallback: Deployment automatically falls back to Xvfb (software rendering).

WebGPU Not Available

Symptoms:

WebGPU is not supported in this browser

Cause: Vulkan not working, wrong Chrome channel, or Xvfb fallback. Fix:

# Check Vulkan
vulkaninfo --summary

# Check VK_ICD_FILENAMES
echo $VK_ICD_FILENAMES
# Should be: /usr/share/vulkan/icd.d/nvidia_icd.json

# Check Chrome version
google-chrome-unstable --version
# Should be: Google Chrome 1xx.x.xxxx.xx dev

# Check DUEL_CAPTURE_USE_XVFB
pm2 show hyperscape-duel | grep DUEL_CAPTURE_USE_XVFB
# Should be: false (for Xorg) or true (for Xvfb)

PulseAudio Not Working

Symptoms:

[RTMPBridge] PulseAudio not accessible, falling back to silent audio

Cause: PulseAudio not running, chrome_audio sink missing, or permission errors. Fix:

# Check PulseAudio status
pulseaudio --check && echo "Running" || echo "Not running"

# Restart PulseAudio
pulseaudio --kill
pulseaudio --start --exit-idle-time=-1 --daemonize=yes

# Create chrome_audio sink
pactl load-module module-null-sink sink_name=chrome_audio
pactl set-default-sink chrome_audio

# Verify
pactl list short sinks | grep chrome_audio

See Streaming Audio Capture for full guide.

Stream Not Appearing on Platforms

Symptoms:

Stream key configured but no stream on Twitch/Kick/X
FFmpeg running but no output

Cause: Wrong stream key, wrong RTMP URL, or network issues. Fix:

# Check RTMP status
cat /root/hyperscape/packages/server/public/live/rtmp-status.json

# Check FFmpeg processes
ps aux | grep ffmpeg

# Check stream keys (masked)
pm2 logs hyperscape-duel | grep "STREAM_KEY"
# Should show: ***configured***

# Check FFmpeg logs for RTMP errors
pm2 logs hyperscape-duel | grep -iE "rtmp|error|failed"

# Verify RTMP URLs
pm2 logs hyperscape-duel | grep -E "KICK_RTMP_URL|X_RTMP_URL|TWITCH"

Correct URLs (commit 5dbd239):

Twitch: rtmp://live.twitch.tv/app
Kick: rtmps://fa723fc1b171.global-contribute.live-video.net/app
X: rtmp://sg.pscp.tv:80/x

DATABASE_URL Not Persisting

Symptoms:

Error: connect ECONNREFUSED (database connection)

Cause: Git reset overwrites .env file. Fix (commit eec04b0): Secrets are now written to /tmp/hyperscape-secrets.env before git reset, then copied back after. Verify:

# Check /tmp secrets
cat /tmp/hyperscape-secrets.env | grep DATABASE_URL

# Check .env
cat /root/hyperscape/packages/server/.env | grep DATABASE_URL

# Check PM2 environment
pm2 show hyperscape-duel | grep DATABASE_URL

PM2 Process Crashes Immediately

Symptoms:

[PM2] Process exited with code 1

Cause: Missing dependencies, database connection failure, or configuration errors. Fix:

# Check PM2 logs
pm2 logs hyperscape-duel --lines 100

# Check error logs
pm2 logs hyperscape-duel --err --lines 50

# Common issues:
# 1. DATABASE_URL not set
# 2. JWT_SECRET missing (required in production)
# 3. Bun not installed
# 4. Dependencies not installed

# Verify environment
pm2 show hyperscape-duel | grep -E "DATABASE_URL|JWT_SECRET|NODE_ENV"

Maintenance Mode Fails

Symptoms:

Warning: Failed to exit maintenance mode

Cause: Server not healthy, ADMIN_CODE wrong, or server URL incorrect. Fix:

# Check server health
curl http://localhost:5555/health

# Check maintenance status
curl https://your-server.com/admin/maintenance/status \
  -H "x-admin-code: your-admin-code"

# Manually exit maintenance mode
curl -X POST https://your-server.com/admin/maintenance/exit \
  -H "Content-Type: application/json" \
  -H "x-admin-code: your-admin-code"

Bun Not Found

Symptoms:

bash: bun: command not found

Cause: Bun not installed or not in PATH. Fix (commit abfe0ce):

# Install unzip (required for bun)
apt-get update && apt-get install -y unzip

# Install bun
curl -fsSL https://bun.sh/install | bash

# Add to PATH
export PATH="/root/.bun/bin:$PATH"

# Verify
bun --version

The deploy script now checks for bun and installs automatically.

Database Issues

Schema Migration Fails

Symptoms:

Error: relation "players" does not exist

Cause: Database schema not pushed or migrations not run. Fix:

cd packages/server
bunx drizzle-kit push --force

# Or run migrations
bunx drizzle-kit migrate

Database Connection Timeout

Symptoms:

Error: Connection timeout

Cause: Database cold start or network issues. Fix (commit dda4396): Database warmup with 3 retry attempts:

# In deploy-vast.sh
for i in 1 2 3; do
  if bun -e "
    const { Pool } = require('pg');
    const pool = new Pool({ connectionString: process.env.DATABASE_URL, max: 5 });
    pool.query('SELECT 1').then(() => { console.log('DB warmup successful'); pool.end(); process.exit(0); }).catch(e => { console.error('DB warmup failed:', e.message); pool.end(); process.exit(1); });
  "; then
    echo "Database connection verified"
    break
  else
    echo "Database warmup attempt $i failed, retrying..."
    sleep 3
  fi
done

Stale Schema After Pull

Symptoms:

Missing columns
Type errors
Constraint violations

Cause: Local database schema doesn’t match code. Fix:

# Reset local database (WARNING: deletes all data)
docker stop hyperscape-postgres
docker rm hyperscape-postgres
docker volume rm hyperscape-postgres-data server_postgres-data

# Restart with fresh schema
bun run dev

Streaming Issues

No Audio in Stream

Symptoms:

Video works but no audio
Silent stream

Cause: PulseAudio not running, chrome_audio sink missing, or FFmpeg not capturing. Fix:

# Check PulseAudio
pulseaudio --check && echo "OK" || echo "FAILED"

# Check chrome_audio sink
pactl list short sinks | grep chrome_audio

# Restart PulseAudio
pulseaudio --kill
pulseaudio --start --exit-idle-time=-1 --daemonize=yes
pactl load-module module-null-sink sink_name=chrome_audio
pactl set-default-sink chrome_audio

# Verify FFmpeg is capturing
pm2 logs hyperscape-duel | grep -i pulse

See Streaming Audio Capture for full guide.

Audio/Video Desync

Symptoms:

Audio plays ahead or behind video
Gradual drift over time

Cause: Missing wall clock timestamps or async resampling. Fix (commit b9d2e41):

# Ensure these flags are in FFmpeg args:
-use_wallclock_as_timestamps 1
-af aresample=async=1000:first_pts=0

# Verify in logs
pm2 logs hyperscape-duel | grep -E "use_wallclock|aresample"

Stream Buffering on Viewers

Symptoms:

Viewers experience frequent buffering
Stream quality drops

Cause: Insufficient buffer size or zerolatency tune. Fix (commit 4c630f1):

# Use film tune instead of zerolatency
export STREAM_LOW_LATENCY=false

# Increase buffer size to 4x bitrate
# This is now the default (18000k for 4500k bitrate)

# Verify in logs
pm2 logs hyperscape-duel | grep -E "tune|bufsize"
# Should see: -tune film -bufsize 18000k

FFmpeg Crashes

Symptoms:

[RTMPBridge] FFmpeg exited with code=1

Cause: Invalid arguments, missing codecs, or RTMP connection failure. Fix:

# Check FFmpeg logs
pm2 logs hyperscape-duel | grep -A 20 "FFmpeg"

# Check FFmpeg version
ffmpeg -version

# Test FFmpeg manually
ffmpeg -f lavfi -i testsrc=size=1280x720:rate=30 \
  -f lavfi -i anullsrc=r=44100:cl=stereo \
  -t 10 \
  -c:v libx264 -preset ultrafast -tune film \
  -c:a aac -b:a 128k \
  -f flv test-output.flv

Security & Secrets

JWT_SECRET Missing Error

Symptoms:

Error: JWT_SECRET is required in production/staging environments

Cause: JWT_SECRET not set in production. Fix:

# Generate secret
openssl rand -base64 32

# Add to .env or Railway environment
JWT_SECRET=your-generated-secret

Note: This is now enforced in production/staging (commit b56b0fd).

Stream Keys Not Working

Symptoms:

Stream keys configured but streams don’t appear
FFmpeg shows “Connection refused”

Cause: Stale stream keys in environment override .env values. Fix (commit a71d4ba):

# The deploy script now explicitly unsets and re-exports
unset TWITCH_STREAM_KEY X_STREAM_KEY X_RTMP_URL KICK_STREAM_KEY KICK_RTMP_URL
unset YOUTUBE_STREAM_KEY
export YOUTUBE_STREAM_KEY=""
source /root/hyperscape/packages/server/.env

# Verify keys are configured
pm2 logs hyperscape-duel | grep "STREAM_KEY"
# Should show: ***configured*** (not NOT SET)

Solana Keypair Not Found

Symptoms:

Error: Keypair file not found: ~/.config/solana/id.json

Cause: SOLANA_DEPLOYER_PRIVATE_KEY not set or decode-key.ts not run. Fix (commit 8a677dc):

# Set environment variable
export SOLANA_DEPLOYER_PRIVATE_KEY="[1,2,3,...]"  # JSON array
# OR
export SOLANA_DEPLOYER_PRIVATE_KEY="base58string"

# Run decode script
bun run scripts/decode-key.ts

# Verify
ls -la ~/.config/solana/id.json
solana-keygen pubkey ~/.config/solana/id.json

Secrets Lost After Git Reset

Symptoms:

DATABASE_URL works initially but fails after deployment
Stream keys stop working after git pull

Cause: Git reset overwrites .env file. Fix (commit eec04b0): Secrets are now written to /tmp/hyperscape-secrets.env before git reset:

# In deploy-vast.yml
cat > /tmp/hyperscape-secrets.env << 'EOF'
DATABASE_URL=${{ secrets.DATABASE_URL }}
# ... other secrets
EOF

# In deploy-vast.sh (after git reset)
cp /tmp/hyperscape-secrets.env /root/hyperscape/packages/server/.env

Common Error Messages

”Failed to resolve module specifier”

Fix: Update Vite config with polyfill aliases (commit e012ed2).

”WebGPU is not supported”

Fix: Use Chrome 113+, Edge 113+, or Safari 18+. Check webgpureport.org.

”CSRF token validation failed”

Fix: CSRF now skipped for cross-origin (commit 8626299). Update server code.

”Connection refused” (database)

Fix: Set DATABASE_URL, run database warmup (commit dda4396).

”Keypair file not found”

Fix: Set SOLANA_DEPLOYER_PRIVATE_KEY, run decode-key.ts (commit 8a677dc).

”PulseAudio: Connection refused”

Fix: Start PulseAudio, create chrome_audio sink. See Streaming Audio Capture.

”FFmpeg: No such file or directory”

Fix: Install FFmpeg (apt-get install ffmpeg) or set FFMPEG_PATH.

”Port already in use”

Fix: Kill process on port (lsof -ti:5555 | xargs kill -9) or use different port.

Diagnostic Commands

Check All Services

# Server health
curl http://localhost:5555/health

# Database connection
cd packages/server && bun -e "const { Pool } = require('pg'); const pool = new Pool({ connectionString: process.env.DATABASE_URL }); pool.query('SELECT 1').then(() => console.log('OK')).catch(e => console.error(e));"

# PulseAudio
pulseaudio --check && pactl list short sinks | grep chrome_audio

# Vulkan
vulkaninfo --summary

# NVIDIA
nvidia-smi

# FFmpeg
ffmpeg -version

# Chrome
google-chrome-unstable --version

# PM2
pm2 status

Check Environment Variables

# Server
pm2 show hyperscape-duel | grep -E "DATABASE_URL|JWT_SECRET|STREAM_KEY|DISPLAY|VK_ICD"

# Or check .env directly
cat /root/hyperscape/packages/server/.env

Check Logs

# PM2 logs
pm2 logs hyperscape-duel --lines 200

# Error logs only
pm2 logs hyperscape-duel --err --lines 100

# Filter for specific issues
pm2 logs hyperscape-duel | grep -iE "error|failed|warning"

# Streaming logs
pm2 logs hyperscape-duel | grep -iE "rtmp|ffmpeg|stream|pulse"

# Xorg logs
cat /var/log/Xorg.99.log

Performance Issues

High CPU Usage

Symptoms:

CPU > 90%
Lag or frame drops

Cause: Too many agents, high encoding quality, or inefficient code. Fix:

# Reduce video bitrate
export STREAM_VIDEO_BITRATE_KBPS=3000  # Reduce from 4500

# Reduce resolution
export STREAM_CAPTURE_WIDTH=1024
export STREAM_CAPTURE_HEIGHT=576

# Reduce FPS
export STREAM_FPS=24  # Reduce from 30

# Disable audio
export STREAM_AUDIO_ENABLED=false

# Limit agents
export AUTO_START_AGENTS_MAX=5  # Reduce from 10

High Memory Usage

Symptoms:

Memory > 4GB
PM2 restarts process

Cause: Memory leaks, too many agents, or large buffers. Fix:

# Check memory usage
pm2 status
pm2 show hyperscape-duel | grep memory

# Reduce agents
export AUTO_START_AGENTS_MAX=5

# Enable memory management
export MALLOC_TRIM_THRESHOLD_="-1"
export MIMALLOC_ALLOW_DECOMMIT="0"

# Restart to clear memory
pm2 restart hyperscape-duel

Stream Lag or Stuttering

Symptoms:

Choppy video
Audio dropouts
Frame drops

Cause: Insufficient buffering, network issues, or CPU overload. Fix:

# Increase buffer size
export STREAM_LOW_LATENCY=false  # Enables 4x buffer (18000k)

# Increase thread queue size
# Edit rtmp-bridge.ts:
-thread_queue_size 2048  # Increase from 1024

# Check CPU usage
top
# If > 90%, reduce encoding quality or resolution

Network Issues

WebSocket Connection Fails

Symptoms:

WebSocket connection failed

Cause: Wrong URL, firewall, or server not running. Fix:

# Check server is running
curl http://localhost:5555/health

# Check WebSocket port
lsof -i :5555

# Test WebSocket
wscat -c ws://localhost:5555/ws

# Check firewall (Vast.ai)
# Ensure port 35079 is exposed

CORS Errors

Symptoms:

Access to fetch at '...' has been blocked by CORS policy

Cause: Missing CORS headers or R2 CORS not configured. Fix:

# For R2 assets
bash scripts/configure-r2-cors.sh

# For API endpoints
# Server already has CORS enabled for known origins

​Deployment Troubleshooting Guide

​Table of Contents

​Cloudflare Pages

​Build Fails with Module Resolution Errors

​CSP Errors Loading Google Fonts

​CORS Errors Loading Assets from R2

​Multi-line Commit Messages Break Deploy

​Railway

​Database Connection Fails

​CSRF Errors from Cloudflare Pages

​Port Conflicts

​Vast.ai

​Xorg Fails to Start

​WebGPU Not Available

​PulseAudio Not Working

​Stream Not Appearing on Platforms

​DATABASE_URL Not Persisting

​PM2 Process Crashes Immediately

​Maintenance Mode Fails

​Bun Not Found

​Database Issues

​Schema Migration Fails

​Database Connection Timeout

​Stale Schema After Pull

​Streaming Issues

​No Audio in Stream

​Audio/Video Desync

​Stream Buffering on Viewers

​FFmpeg Crashes

​Security & Secrets

​JWT_SECRET Missing Error

​Stream Keys Not Working

​Solana Keypair Not Found

​Secrets Lost After Git Reset

​Common Error Messages

​”Failed to resolve module specifier”

​”WebGPU is not supported”

​”CSRF token validation failed”

​”Connection refused” (database)

​”Keypair file not found”

​”PulseAudio: Connection refused”

​”FFmpeg: No such file or directory”

​”Port already in use”

​Diagnostic Commands

​Check All Services

​Check Environment Variables

​Check Logs

​Performance Issues

​High CPU Usage

​High Memory Usage

​Stream Lag or Stuttering

​Network Issues

​WebSocket Connection Fails

​CORS Errors

​Related Documentation

Deployment Troubleshooting Guide

Table of Contents

Cloudflare Pages

Build Fails with Module Resolution Errors

CSP Errors Loading Google Fonts

CORS Errors Loading Assets from R2

Multi-line Commit Messages Break Deploy

Railway

Database Connection Fails

CSRF Errors from Cloudflare Pages

Port Conflicts

Vast.ai

Xorg Fails to Start

WebGPU Not Available

PulseAudio Not Working

Stream Not Appearing on Platforms

DATABASE_URL Not Persisting

PM2 Process Crashes Immediately

Maintenance Mode Fails

Bun Not Found

Database Issues

Schema Migration Fails

Database Connection Timeout

Stale Schema After Pull

Streaming Issues

No Audio in Stream

Audio/Video Desync

Stream Buffering on Viewers

FFmpeg Crashes

Security & Secrets

JWT_SECRET Missing Error

Stream Keys Not Working

Solana Keypair Not Found

Secrets Lost After Git Reset

Common Error Messages

”Failed to resolve module specifier”

”WebGPU is not supported”

”CSRF token validation failed”

”Connection refused” (database)

”Keypair file not found”

”PulseAudio: Connection refused”

”FFmpeg: No such file or directory”

”Port already in use”

Diagnostic Commands

Check All Services

Check Environment Variables

Check Logs

Performance Issues

High CPU Usage

High Memory Usage

Stream Lag or Stuttering

Network Issues

WebSocket Connection Fails

CORS Errors

Related Documentation