CI/CD Troubleshooting Guide

Overview

This guide documents common CI/CD issues, their root causes, and solutions based on recent fixes to the Hyperscape build pipeline.

Database Migration Issues

Issue: Migration 0050 Duplicate Table Errors

Symptom:

ERROR: relation "agent_duel_stats" already exists (42P07)

Cause (commit e4b6489):

Migration 0050 duplicated CREATE TABLE statements from earlier migrations
Example: agent_duel_stats was created in migration 0039 and again in 0050
On fresh databases, running all migrations sequentially caused duplicate table errors

Solution: Added IF NOT EXISTS to all CREATE TABLE and CREATE INDEX statements in migration 0050:

-- Before
CREATE TABLE agent_duel_stats (...);

-- After
CREATE TABLE IF NOT EXISTS agent_duel_stats (...);

Prevention:

Always use IF NOT EXISTS for CREATE TABLE in migrations
Check migration history before adding new tables
Test migrations on fresh database before committing

Issue: FK Ordering in Sequential Migrations

Symptom:

ERROR: relation "arena_rounds" does not exist

Cause (commit eb8652a):

Migration 0050 references tables from older migrations (e.g., arena_rounds)
On fresh databases, FK constraints may fail if tables aren’t created in dependency order
Sequential migration execution doesn’t guarantee FK ordering

Solution: Use drizzle-kit push for declarative schema creation + SKIP_MIGRATIONS=true:

# CI integration tests
bunx drizzle-kit push
SKIP_MIGRATIONS=true bun run start

Why This Works:

drizzle-kit push creates schema declaratively (no ordering issues)
SKIP_MIGRATIONS=true tells server to skip built-in migration execution
Server starts with pre-created schema

Issue: drizzle-kit push + Server Migration Conflict

Symptom:

ERROR: relation "users" already exists
Server migration fails after drizzle-kit push

Cause (commit b5d2494):

Running drizzle-kit push creates tables without populating migration journal
Server’s built-in migration code tries to create tables again
Results in duplicate table errors

Solution: Do NOT run drizzle-kit push separately in CI. Let server handle migrations:

# ❌ WRONG
bunx drizzle-kit push
bun run start  # Server tries to migrate again

# ✅ CORRECT (Option 1: Server migrations)
bun run start  # Server runs migrations automatically

# ✅ CORRECT (Option 2: External schema + skip)
bunx drizzle-kit push
SKIP_MIGRATIONS=true bun run start

SKIP_MIGRATIONS Environment Variable

Purpose: Skip server migration when schema is created externally When to Use:

CI/testing environments using drizzle-kit push
External schema management tools
Integration tests that create schema before server startup

What It Skips (commit 6a5f4ee):

Built-in migration execution
hasRequiredPublicTables validation check
Migration recovery loop

Important: You MUST create the database schema externally before starting the server with SKIP_MIGRATIONS=true. Example:

# Integration test workflow
bunx drizzle-kit push
SKIP_MIGRATIONS=true bun run test:integration

Dependency Issues

Issue: ESLint ajv TypeError

Symptom:

TypeError: Class extends value undefined is not a constructor or null

Cause (commit b344d9e):

Root package.json forced ajv@8 via overrides
@eslint/eslintrc requires ajv@6 for Draft-04 schema support
Version conflict caused constructor chain to break

Solution: Remove ajv version overrides from root package.json:

// ❌ WRONG
{
  "overrides": {
    "ajv": "^8.18.0"  // Breaks @eslint/eslintrc
  }
}

// ✅ CORRECT
{
  "overrides": {
    // No ajv override - let packages use their required versions
  }
}

Prevention:

Don’t force major version upgrades via overrides
Check package peer dependencies before overriding
Test ESLint after adding overrides

Issue: Missing hls.js Dependency

Symptom:

ERROR: Cannot find module 'hls.js'
Build fails in CI for gold-betting-demo

Cause (commit cfdabf3):

StreamPlayer.tsx imports hls.js but it was not declared in package.json
Works locally due to workspace hoisting
Fails in CI where bun resolves dependencies strictly

Solution: Add missing dependency to package.json:

{
  "dependencies": {
    "hls.js": "^1.4.0"
  }
}

Prevention:

Run bun install --frozen-lockfile to catch missing deps
Test builds in clean environment (Docker)
Use bun run build before committing

Issue: Foundry/Anvil Not Available in CI

Symptom:

anvil: command not found
Integration tests fail

Cause (commit b344d9e):

Integration tests require anvil binary for local Ethereum node
Foundry toolchain not installed in CI environment

Solution: Add Foundry toolchain to CI workflow:

# .github/workflows/integration.yml
- name: Install Foundry
  uses: foundry-rs/foundry-toolchain@v1
  
- name: Run integration tests
  run: bun run test:integration

Local Development:

# Install Foundry
curl -L https://foundry.paradigm.xyz | bash
foundryup

# Verify installation
anvil --version

Package Exclusions

Excluded from CI Tests

Packages:

@hyperscape/contracts (commit 99dec96)
@hyperscape/gold-betting-demo (commit 93f9633)
@hyperscape/evm-contracts (commit 034f9c9)

Reasons:

contracts: MUD CLI + @trpc/server compatibility issue
gold-betting-demo: hls.js dependency resolution issue (fixed in cfdabf3, but still excluded)
evm-contracts: Foundry/anvil not available in CI

Turbo Filter:

# Exclude from test run
turbo run test --filter='!@hyperscape/contracts' --filter='!@hyperscape/evm-contracts'

Re-enabling: Tests will be re-enabled when dependency conflicts are resolved.

Chain Setup Issues

Issue: Chain Setup Fails in CI

Symptom:

setup-chain.mjs fails: anvil not found
MUD contracts tests fail

Cause (commit 034f9c9):

setup-chain.mjs tries to start anvil and deploy MUD contracts
Anvil binary not available in CI environment
MUD CLI has compatibility issues

Solution: Skip chain setup when CI=true:

// scripts/setup-chain.mjs
if (process.env.CI === 'true') {
  console.log('Skipping chain setup in CI environment');
  process.exit(0);
}

Local Development:

# Install Foundry first
curl -L https://foundry.paradigm.xyz | bash
foundryup

# Then run setup
bun run setup-chain

Asset Management

Issue: Assets Directory Already Exists

Symptom:

fatal: destination path 'assets' already exists and is not an empty directory

Cause (commit 6ce05cc):

CI workflow clones assets repo
Previous run left assets directory
Git clone fails on non-empty directory

Solution: Remove assets directory before cloning:

# .github/workflows/ci.yml
- name: Clone assets
  run: |
    rm -rf assets
    git clone https://github.com/HyperscapeAI/assets.git assets

Prevention:

Always clean up in CI workflows
Use rm -rf before git clone
Consider using git clone --depth 1 for faster clones

Security Audit

Issue: Build Fails on High Severity Vulnerabilities

Symptom:

npm audit found 2 high severity vulnerabilities
CI build fails

Cause (commit 19bebe2):

bigint-buffer has high severity vulnerability
No upstream patch available
CI audit threshold set to high

Solution: Lower audit threshold to critical:

# .github/workflows/security.yml
- name: Security audit
  run: npm audit --audit-level=critical

Rationale:

Allows builds to pass while waiting for upstream fixes
Critical vulnerabilities still block builds
High/moderate vulnerabilities logged but don’t fail CI

Remaining Vulnerabilities:

bigint-buffer (high) - no patched version available
elliptic (moderate) - no patched version available

Recent Security Fixes (commit a390b79)

Resolved:

✅ Playwright ^1.55.1 (fixes GHSA-7mvr-c777-76hp, high)
✅ Vite ^6.4.1 (fixes GHSA-g4jq-h2w9-997c, GHSA-jqfw-vq24-v9c3, GHSA-93m4-6634-74q7)
✅ ajv ^8.18.0 (fixes GHSA-2g4f-4pwh-qvx6)
✅ Root overrides for: @trpc/server, minimatch, cookie, undici, jsondiffpatch, tmp, diff, bn.js, ai

Total: 14 of 16 vulnerabilities resolved

Documentation Updates

Issue: Mintlify API Failures Block CI

Symptom:

Mintlify API call failed
CI workflow fails

Cause (commit 034f9c9):

Mintlify service outages
API rate limits
Network issues

Solution: Add continue-on-error to docs update step:

# .github/workflows/update-docs.yml
- name: Update documentation
  continue-on-error: true
  run: npm run docs:update

Rationale:

Documentation updates are not critical for build success
Allows CI to continue even if docs API is down
Docs can be updated manually if needed

Build Resilience

Issue: Circular Dependencies Break Clean Builds

Symptom:

tsc fails: Cannot find module '@hyperscape/shared'
procgen and plugin-hyperscape builds fail

Cause (commit 5666ece):

Circular dependencies between packages
@hyperscape/shared imports from @hyperscape/procgen
@hyperscape/procgen peer-depends on @hyperscape/shared
When turbo runs clean build, tsc fails because the other package’s dist/ doesn’t exist yet

Solution: Use tsc || echo pattern for resilient builds:

// packages/procgen/package.json
{
  "scripts": {
    "build": "tsc || echo 'Build completed with circular dep warnings'"
  }
}

Why This Works:

Build exits 0 even with circular dep errors
Packages produce partial output sufficient for downstream consumers
Turbo can continue build pipeline

Prevention:

Avoid circular dependencies when possible
Use peer dependencies carefully
Test clean builds: bun run clean && bun run build

TypeScript Errors

Issue: Type Errors Block CI

Symptom:

AgentManager.ts: Type 'EmbeddedHyperscapeService' is not assignable to type 'HyperscapeService'
ArenaService.ts: Argument of type 'unknown' is not assignable to parameter

Cause (commit 5e60439):

Type mismatches after refactoring
Missing type casts
Private methods called from tests

Solutions: Type Casts:

// AgentManager.ts
const service = embeddedService as HyperscapeService;

Parameters Utility:

// ArenaService.ts
type PositionParam = Parameters<typeof teleportToArena>[2];
const position = unknownParam as PositionParam;

Visibility Changes:

// ArenaRoundService.ts
// Change from private to public for test access
public getEligibleAgents(): string[] { ... }

Test Infrastructure

WebGPU Mocks for Three.js

Issue: Three.js WebGPU renderer requires browser globals Symptom:

ReferenceError: GPUShaderStage is not defined

Solution (commit 25ba63c): Create vitest.setup.ts with WebGPU mocks:

// packages/server/vitest.setup.ts
globalThis.GPUShaderStage = {
  VERTEX: 1,
  FRAGMENT: 2,
  COMPUTE: 4,
};

globalThis.GPUBufferUsage = {
  MAP_READ: 1,
  MAP_WRITE: 2,
  COPY_SRC: 4,
  COPY_DST: 8,
  INDEX: 16,
  VERTEX: 32,
  UNIFORM: 64,
  STORAGE: 128,
  INDIRECT: 256,
  QUERY_RESOLVE: 512,
};

// ... more WebGPU globals

Configure in vitest.config.ts:

export default defineConfig({
  test: {
    setupFiles: ['./vitest.setup.ts'],
  },
});

ArenaService Test Helpers

Issue: Cannot spy on private methods Solution (commit 25ba63c): Add protected passthrough methods:

// ArenaService.ts
protected getDb() {
  return this.world.getSystem("database");
}

protected getEligibleAgents() {
  return this.arenaRoundService.getEligibleAgents();
}

// Test file
const dbSpy = vi.spyOn(arenaService as any, 'getDb');

Database Mock Helper:

function setDbMock(world: World, mockDb: any) {
  vi.spyOn(world, 'getSystem').mockImplementation((name) => {
    if (name === 'database') return mockDb;
    return null;
  });
}

Streaming Infrastructure

Issue: WebGPU Crashes on RTX 5060 Ti

Symptom:

Chrome crashes during WebGPU initialization
Vulkan ICD errors in logs

Cause (commits 0257563, 30cacb0):

RTX 5060 Ti has broken Vulkan ICD on Vast.ai
WebGPU defaults to Vulkan backend
Vulkan initialization crashes Chrome

Solutions: 1. Use GL ANGLE Backend:

// Chrome launch args
'--use-angle=gl',
'--use-gl=angle',

2. Remove RTX 5060 Ti from GPU Search:

// vast-keeper GPU filter
const excludedGPUs = ['RTX 5060 Ti'];

3. Use System FFmpeg:

# Dockerfile
RUN apt-get install -y ffmpeg
# Don't use static FFmpeg build (causes SIGSEGV)

Issue: RTX 4090 WebGPU Performance

Symptom:

WebGPU works but performance is suboptimal
GL backend used instead of Vulkan

Solution (commit 80bb06e): Switch ANGLE to Vulkan backend for RTX 4090:

// Chrome launch args for RTX 4090
'--use-angle=vulkan',
'--use-vulkan',
'--enable-features=Vulkan',

GPU-Specific Configuration:

const gpuModel = detectGPU();
const angleBackend = gpuModel.includes('RTX 4090') ? 'vulkan' : 'gl';
args.push(`--use-angle=${angleBackend}`);

Issue: Static FFmpeg Build SIGSEGV

Symptom:

Segmentation fault (core dumped)
FFmpeg crashes during encoding

Cause (commits 55a07bd, 536763d):

Static FFmpeg builds have compatibility issues
SIGSEGV during H.264 encoding

Solution: Use system FFmpeg instead of static build:

# Dockerfile
RUN apt-get update && apt-get install -y ffmpeg

# Don't use ffmpeg-static npm package

Verification:

which ffmpeg  # Should be /usr/bin/ffmpeg
ffmpeg -version

Vast.ai Deployment

Issue: vastai CLI Not on PATH

Symptom:

vastai: command not found

Cause (commits 3ce7d64, 5c2a566):

vastai installed via pip but not on PATH
Python venv not activated

Solution: Use python venv for vastai install:

# Dockerfile
RUN python3 -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
RUN pip3 install vastai

Alternative:

# Use python module invocation
python3 -m vastai search offers

Issue: Python Version Too Old

Symptom:

vastai-sdk requires Python 3.10+
Current: Python 3.9

Cause (commit 621ae67):

Debian bullseye-slim has Python 3.9
vastai-sdk requires Python 3.10+

Solution: Upgrade to Debian bookworm-slim:

# Before
FROM debian:bullseye-slim

# After
FROM debian:bookworm-slim

Issue: PEP 668 Externally Managed Environment

Symptom:

error: externally-managed-environment
pip install fails on Debian 12

Cause (commit d9e9111):

Debian 12 enforces PEP 668
System Python is externally managed
pip install blocked by default

Solution: Use --break-system-packages flag:

pip3 install --break-system-packages vastai

Better Solution: Use python venv (see above)

Playwright Issues

Issue: Chromium Not Installed

Symptom:

Error: Executable doesn't exist at /path/to/chromium

Cause:

Playwright browsers not installed
CI environment missing browser binaries

Solution:

# Install Playwright browsers
bunx playwright install chromium

# Or install all browsers
bunx playwright install

# With dependencies (Linux)
bunx playwright install --with-deps chromium

CI Workflow:

- name: Install Playwright
  run: bunx playwright install --with-deps chromium

Docker Issues

Issue: DNS Resolution Fails in Container

Symptom:

getaddrinfo ENOTFOUND
npm install fails in Docker

Cause (commit fd17248):

Container DNS not configured
Default resolv.conf doesn’t work

Solution: Overwrite resolv.conf with Google DNS:

# Dockerfile
RUN echo "nameserver 8.8.8.8" > /etc/resolv.conf
RUN echo "nameserver 8.8.4.4" >> /etc/resolv.conf

Note: Use > for first line (overwrite), >> for subsequent lines (append)

Issue: Build Context Too Large

Symptom:

Sending build context to Docker daemon: 5.2GB
Build times out

Solution: Add comprehensive .dockerignore:

node_modules
.git
.github
dist
build
*.log
.env
.env.*
packages/*/node_modules
packages/*/dist
packages/*/build

CI Workflow Best Practices

Graceful Degradation

Principle: Non-critical steps should not fail the entire build Examples: Documentation Updates:

- name: Update docs
  continue-on-error: true
  run: npm run docs:update

Asset Sync:

- name: Sync assets
  continue-on-error: true
  run: bun run assets:sync

Conditional Execution

Skip Steps in CI:

// setup-chain.mjs
if (process.env.CI === 'true') {
  console.log('Skipping in CI');
  process.exit(0);
}

Skip Steps Locally:

# Only run in CI
if [ "$CI" = "true" ]; then
  npm run ci-only-task
fi

Caching Strategies

Bun Dependencies:

- name: Cache bun dependencies
  uses: actions/cache@v3
  with:
    path: ~/.bun/install/cache
    key: ${{ runner.os }}-bun-${{ hashFiles('**/bun.lock') }}

Playwright Browsers:

- name: Cache Playwright browsers
  uses: actions/cache@v3
  with:
    path: ~/.cache/ms-playwright
    key: ${{ runner.os }}-playwright-${{ hashFiles('**/package.json') }}

Debugging CI Failures

Enable Debug Logging

GitHub Actions:

- name: Run tests
  env:
    DEBUG: '*'
    VERBOSE: 'true'
  run: bun run test

Bun:

BUN_DEBUG=1 bun run build

Reproduce Locally

Use CI Environment:

# Set CI flag
export CI=true

# Use same Node/Bun version
bun --version  # Should match CI

# Clean install
rm -rf node_modules
bun install --frozen-lockfile

# Run CI commands
bun run build
bun run test

Docker Reproduction:

# Build CI image
docker build -t hyperscape-ci -f Dockerfile.ci .

# Run CI commands
docker run hyperscape-ci bun run test

Inspect Artifacts

Save Logs:

- name: Upload logs
  if: failure()
  uses: actions/upload-artifact@v3
  with:
    name: test-logs
    path: logs/

Save Screenshots:

- name: Upload screenshots
  if: failure()
  uses: actions/upload-artifact@v3
  with:
    name: test-screenshots
    path: packages/*/tests/**/__screenshots__/

Common Error Patterns

Pattern: “Cannot find module”

Causes:

Missing dependency in package.json
Incorrect import path
Build order issue (dependency not built yet)

Solutions:

Add to dependencies: bun add <package>
Fix import path
Check turbo.json dependsOn

Pattern: “ECONNREFUSED”

Causes:

Service not started
Wrong port
Service crashed

Solutions:

Check service startup logs
Verify port in .env
Check for port conflicts: lsof -ti:5555

Pattern: “Timeout”

Causes:

Service slow to start
Network latency
Deadlock

Solutions:

Increase timeout
Add retry logic
Check for circular waits

Monitoring & Alerts

CI Failure Notifications

Slack Integration:

- name: Notify on failure
  if: failure()
  uses: 8398a7/action-slack@v3
  with:
    status: ${{ job.status }}
    webhook_url: ${{ secrets.SLACK_WEBHOOK }}

Discord Integration:

- name: Notify on failure
  if: failure()
  uses: sarisia/actions-status-discord@v1
  with:
    webhook: ${{ secrets.DISCORD_WEBHOOK }}

Health Checks

Server Health Endpoint:

// packages/server/src/routes/health-routes.ts
app.get('/health', async (request, reply) => {
  return {
    status: 'ok',
    uptime: process.uptime(),
    memory: process.memoryUsage(),
    version: process.env.COMMIT_HASH,
  };
});

CI Health Check:

- name: Wait for server
  run: |
    timeout 60 bash -c 'until curl -f http://localhost:5555/health; do sleep 1; done'

Performance Optimization

Parallel Builds

Turbo Configuration:

{
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**"]
    }
  }
}

Benefits:

Builds packages in parallel when possible
Respects dependency order
Caches outputs for incremental builds

Incremental Testing

Run Only Changed Tests:

# Turbo automatically detects changes
turbo run test --filter='...[HEAD^]'

Skip Unchanged Packages:

turbo run test --filter='[HEAD^]'

Rollback Procedures

Revert Failed Deployment

Railway:

# Rollback to previous deployment
railway rollback

# Or deploy specific commit
railway up --commit abc123

Cloudflare Pages:

# Rollback via dashboard
# Or redeploy previous commit
git revert HEAD
git push

Database Rollback

Drizzle Migrations:

# Rollback last migration
bunx drizzle-kit drop

# Restore from backup
pg_restore -d hyperscape backup.sql

Important: Always backup before migrations in production

References

Commit e4b6489: Migration 0050 IF NOT EXISTS fix
Commit eb8652a: drizzle-kit push + SKIP_MIGRATIONS
Commit b344d9e: ESLint ajv fix + Foundry toolchain
Commit 25ba63c: WebGPU mocks + test helpers
Commit 034f9c9: Chain setup skip + docs continue-on-error
Commit 5666ece: Circular dependency resilience
Commit a390b79: Security audit fixes
CI Workflows: .github/workflows/

​CI/CD Troubleshooting Guide

​Overview

​Database Migration Issues

​Issue: Migration 0050 Duplicate Table Errors

​Issue: FK Ordering in Sequential Migrations

​Issue: drizzle-kit push + Server Migration Conflict

​SKIP_MIGRATIONS Environment Variable

​Dependency Issues

​Issue: ESLint ajv TypeError

​Issue: Missing hls.js Dependency

​Issue: Foundry/Anvil Not Available in CI

​Package Exclusions

​Excluded from CI Tests

​Chain Setup Issues

​Issue: Chain Setup Fails in CI

​Asset Management

​Issue: Assets Directory Already Exists

​Security Audit

​Issue: Build Fails on High Severity Vulnerabilities

​Recent Security Fixes (commit a390b79)

​Documentation Updates

​Issue: Mintlify API Failures Block CI

​Build Resilience

​Issue: Circular Dependencies Break Clean Builds

​TypeScript Errors

​Issue: Type Errors Block CI

​Test Infrastructure

​WebGPU Mocks for Three.js

​ArenaService Test Helpers

​Streaming Infrastructure

​Issue: WebGPU Crashes on RTX 5060 Ti

​Issue: RTX 4090 WebGPU Performance

​Issue: Static FFmpeg Build SIGSEGV

​Vast.ai Deployment

​Issue: vastai CLI Not on PATH

​Issue: Python Version Too Old

​Issue: PEP 668 Externally Managed Environment

​Playwright Issues

​Issue: Chromium Not Installed

​Docker Issues

​Issue: DNS Resolution Fails in Container

​Issue: Build Context Too Large

​CI Workflow Best Practices

​Graceful Degradation

​Conditional Execution

​Caching Strategies

​Debugging CI Failures

​Enable Debug Logging

​Reproduce Locally

​Inspect Artifacts

​Common Error Patterns

​Pattern: “Cannot find module”

​Pattern: “ECONNREFUSED”

​Pattern: “Timeout”

​Monitoring & Alerts

​CI Failure Notifications

​Health Checks

​Performance Optimization

​Parallel Builds

​Incremental Testing

​Rollback Procedures

​Revert Failed Deployment

​Database Rollback

​References

CI/CD Troubleshooting Guide

Overview

Database Migration Issues

Issue: Migration 0050 Duplicate Table Errors

Issue: FK Ordering in Sequential Migrations

Issue: drizzle-kit push + Server Migration Conflict

SKIP_MIGRATIONS Environment Variable

Dependency Issues

Issue: ESLint ajv TypeError

Issue: Missing hls.js Dependency

Issue: Foundry/Anvil Not Available in CI

Package Exclusions

Excluded from CI Tests

Chain Setup Issues

Issue: Chain Setup Fails in CI

Asset Management

Issue: Assets Directory Already Exists

Security Audit

Issue: Build Fails on High Severity Vulnerabilities

Recent Security Fixes (commit a390b79)

Documentation Updates

Issue: Mintlify API Failures Block CI

Build Resilience

Issue: Circular Dependencies Break Clean Builds

TypeScript Errors

Issue: Type Errors Block CI

Test Infrastructure

WebGPU Mocks for Three.js

ArenaService Test Helpers

Streaming Infrastructure

Issue: WebGPU Crashes on RTX 5060 Ti

Issue: RTX 4090 WebGPU Performance

Issue: Static FFmpeg Build SIGSEGV

Vast.ai Deployment

Issue: vastai CLI Not on PATH

Issue: Python Version Too Old

Issue: PEP 668 Externally Managed Environment

Playwright Issues

Issue: Chromium Not Installed

Docker Issues

Issue: DNS Resolution Fails in Container

Issue: Build Context Too Large

CI Workflow Best Practices

Graceful Degradation

Conditional Execution

Caching Strategies

Debugging CI Failures

Enable Debug Logging

Reproduce Locally

Inspect Artifacts

Common Error Patterns

Pattern: “Cannot find module”

Pattern: “ECONNREFUSED”

Pattern: “Timeout”

Monitoring & Alerts

CI Failure Notifications

Health Checks

Performance Optimization

Parallel Builds

Incremental Testing

Rollback Procedures

Revert Failed Deployment

Database Rollback

References