Operations Runbook

Reference guide for running the LightChallenge off-chain pipeline in development and production.

Note (LightChain v2 migration): The verification pipeline has moved from the v1 AIVM commit/reveal/PoI ceremony to the LightChain v2 worker network (chain ID 8200). The single end-to-end runner runChallengePayLightchainJob replaces the old aivmIndexer + runChallengePayAivmJob pair. The canonical reference for the new architecture is docs/lightchain-v2-integration.md. Sections below referencing aivm_jobs, AIVMInferenceV2, or ChallengePayAivmPoiVerifier are historical and being progressively rewritten.

Deployment targets

LightChallenge runs across three hosting surfaces:

Smart contracts deploy separately via hardhat deploy (DEPLOY.md).

Important Fly gotcha: fly deploy must be run with the repo root as the positional working directory, otherwise the build context becomes fly/ (~2 bytes) and COPY package.json fails. See fly/README.md for the exact command.

Redeploying contracts or migrating chain ID?

Read docs/chain-redeploy-runbook.md — the canonical “everything that needs to change when on-chain identity changes” reference. Includes the address sync helper:

npx tsx scripts/ops/syncChainAddresses.ts

which wipes + rewrites all chain-related env vars on Vercel + Fly atomically (using printf-not-echo to avoid \n corruption) and triggers redeploys.

Pipeline Architecture

Provider APIs (Strava, Fitbit, FACEIT, OpenDota, Riot) + Manual uploads (Apple Health, Garmin, Google Fit)
        │                                                         Desktop GSI App (Dota 2, CS2, LoL)
        │                                                                │
        │                                                                ▼ wsGateway (port 3100)
        │                                                         public.game_sessions + public.live_game_events
        │                                                                │
        │                                                                ▼ gsiEvidenceBridge (15s poll)
        ▼ progressSyncWorker (active-period, every 15min) + evidenceCollector (proof-window, final reconciliation)
public.evidence
        │
        ▼ evidenceEvaluator
public.verdicts
        │
        ▼ challengeDispatcher (gates on active + verdict)
public.aivm_jobs (queued)
        │
        ▼ challengeWorker
AIVMInferenceV2.requestInferenceV2() [on-chain]
        │
        │ [Lightchain native workers: commit → reveal → attest until quorum]
        │
        ▼ InferenceFinalized event
aivmIndexer → attemptFinalizationBridge()
        → ChallengePay.submitProofFor() → ChallengePayAivmPoiVerifier.verify() + ChallengePay.finalize() [on-chain]
        │
public.aivm_jobs (done)
public.challenges (status = Finalized)

Claims path (parallel):
ChallengePay *Claimed events → claimsIndexer → public.claims

Status sync (parallel):
ChallengePay status events → statusIndexer → public.challenges.status

Fitness Activity Isolation Model

Each fitness activity type is isolated end-to-end to prevent cross-contamination (e.g., walking workouts cannot count toward running challenges).

Evidence Collection (iOS / HealthKit)

Key design decisions:

• distanceWalkingRunning is not sent as evidence — it combines walking+running into one ambiguous value. Workout-level queries provide isolated per-type distance.
• flightsClimbed sends type hike (not walk) — stair elevation counts toward hiking.
• activeEnergyBurned sends type calories (not steps) — it’s a cross-activity aggregate.
• Steps are always cross-activity (pedometer counts all on-foot motion).
• Calories and exercise time are always cross-activity aggregates.

Evaluator Isolation (offchain)

Full Rule path: activities.filter(a => a.type === rule.challengeType) — only activities matching the rule’s challengeType are considered. A running challenge only sees type: "run" records.

Simplified Rules path: activityMatchesSimpleMetric() enforces type-specific filtering:

• walking_km → only walk activities
• hiking_km → only hike activities
• cycling_km → only cycle activities
• swimming_km → only swim activities
• rowing_km → only rowing activities
• yoga_min → only yoga activities
• hiit_min → only hiit activities
• strength_sessions → only strength activities
• Generic metrics (steps, distance_km, active_minutes, calories, exercise_time) accept all activity types.

canonicalType() Mapping (offchain/evaluators/fitnessEvaluator.ts)

Maps provider-specific type strings to canonical types:

• run, virtualrun, trail_run, running → run
• walk, walking → walk
• hike, hiking, trail, mountaineering → hike
• cycle, ride, virtualride, cycling, bike → cycle
• swim, swimming, openwater → swim
• strength, weighttraining, functional_training → strength
• yoga, pilates, flexibility → yoga
• hiit, crossfit, crosstraining, mixed_cardio, circuit_training → hiit
• rowing, rowing_machine, indoor_rowing → rowing
• calories, active_energy, calorie_burn → calories
• exercise_time → exercise_time

Gaming Pipeline (Desktop GSI Capture)

For private matches and live game tracking (Dota 2, CS2, LoL), the desktop GSI capture app streams game state data directly from the player’s machine — no external API keys needed.

Architecture

Player's PC                                          Server
┌──────────────────────────┐                  ┌───────────────────────┐
│ Dota 2 / CS2 (GSI)       │                  │                       │
│ → localhost:3000/gsi     ├──────────────────│→ Desktop App (Tauri)  │
│                          │                  │                       │
│ League of Legends        │                  │         │ WebSocket   │
│ → localhost:2999 (poll)  ├──────────────────│→        ▼             │
└──────────────────────────┘                  │  wsGateway.ts (:3100) │
                                              │         │             │
                                              │         ▼             │
                                              │  game_sessions        │
                                              │  live_game_events     │
                                              │         │             │
                                              │         ▼             │
                                              │  gsiEvidenceBridge.ts │
                                              │         │             │
                                              │         ▼             │
                                              │  public.evidence      │
                                              │  (existing pipeline)  │
                                              └───────────────────────┘

Two Evidence Paths for Gaming

Both paths produce evidence in public.evidence with the same record format, so the evaluator → AIVM pipeline works identically regardless of source.

Desktop App Setup

1. Install & run the Tauri desktop app:

   cd desktop && npm install && npm run dev

2. Game configuration:

   • Dota 2: Copy desktop/gsi-configs/gamestate_integration_lightchallenge.cfg to your Dota 2 cfg/ directory, restart Dota
   • CS2: Copy desktop/gsi-configs/gamestate_integration_lightchallenge_cs2.cfg to your CS2 cfg/ directory, restart CS2
   • LoL: No config needed — auto-detected via Live Client Data API at localhost:2999

3. Connect: Enter wallet address, set gateway URL (ws://uat.lightchallenge.app:3100), click Connect

Server-Side Workers

Two workers must be running for the GSI pipeline:

The bridge worker:

• Polls every 15s for completed sessions not yet bridged
• Finds active challenges matching the wallet + platform
• Converts session summary (K/D/A, win/loss, hero) into connector-format evidence records
• Accumulates records (multiple games per challenge are appended, deduplicated by match_id)
• Uses existing provider names (opendota, faceit, riot) so the gaming evaluator handles them identically

Ports

Gaming API Keys (for API connector path only)

Prerequisites

• Node.js 22 + npm install at repo root
• webapp/.env.local with all required variables set (see .env.example)
• LightChain testnet RPC accessible (https://rpc.testnet.lightchain.ai)
• PostgreSQL database (Neon or local) with DATABASE_URL set

1. Database Migration

Run once before starting any workers, and after adding new migration files:

npx tsx db/migrate.ts

Applied migrations are tracked in public.schema_migrations. Re-running is safe — already-applied files are skipped.

Current migrations: 001_evidence_verdicts through 027_expanded_fitness_models. See db/DATABASE.md for full schema documentation.

2. Identity Seed (one-time)

Migrate legacy offchain/.state/identity_bindings.json → public.identity_bindings:

npx tsx db/seed_identity.ts

Safe to re-run (upsert on conflict). Only needed if migrating from a pre-DB version of the system.

3. Evidence Collector

Finds challenges currently in their proof submission window (endTs <= now AND proofDeadlineTs > now), identifies participants who have not yet submitted evidence, fetches activity data for exactly the challenge period (startTs to endTs), and stores normalized records in public.evidence.

The collector uses challenge timeline data from public.challenges.timeline (JSONB) to determine the proof window and the precise date range for data fetching. There is no fixed lookback; each challenge’s own period defines what data to retrieve.

npx tsx offchain/workers/evidenceCollector.ts

Server-side collection providers (fetched automatically during proof window):

• strava — OAuth, token auto-refresh, fetches activities for challenge period
• fitbit — OAuth, token auto-refresh, fetches daily steps + activity logs for challenge period
• opendota — free API, fetches Dota 2 matches by Steam32 ID for challenge period
• riot — API key required, fetches LoL matches by PUUID for challenge period
• faceit — API key required, fetches CS2 matches by Steam64→FACEIT player ID for challenge period

Upload-only providers (skipped by evidence collector — evidence via file upload or auto-proof):

• apple — no server-side API; users upload Apple Health ZIP export (or iOS AutoProofService pushes data)
• garmin — no public API (enterprise-only); users upload TCX/GPX/JSON export (or iOS AutoProofService pushes data)
• googlefit — API deprecated by Google in 2025; users upload Google Takeout JSON

Writes to: public.evidence

Note: The collector skips insertion when the incoming evidence_hash matches the previous row for the same (challenge_id, subject, provider) — no duplicate rows.

Auto-Proof API Endpoint

POST /api/challenge/{id}/auto-proof provides on-demand, per-user evidence collection as a complement to the background evidence collector.

Behavior:

• Triggers immediate evidence collection for the authenticated user and the specified challenge
• Only works during the proof submission window (endTs <= now AND proofDeadlineTs > now); returns an error otherwise
• For server-side providers (Strava, Fitbit, OpenDota, Riot, FACEIT): pulls data server-side for the exact challenge period (startTs to endTs) and stores it in public.evidence
• For upload-only providers (Apple Health, Garmin): returns "upload-required" with the date range (startTs, endTs) so the client can collect and upload data for that period

Callers:

• Webapp: called when a user views a challenge that is in its proof window (ensures evidence is collected promptly without waiting for the next collector poll)
• iOS AutoProofService: called when the iOS app detects a challenge has entered its proof window (allows the app to either upload local health data or trigger server-side fetching)

Authentication: Requires standard x-lc-address / x-lc-signature / x-lc-timestamp headers (see section 16).

3b. Progress Sync Worker

Runs during active challenges (not just the proof window) to keep real-time progress updated from ALL connected API-based providers. Periodically fetches activity data and upserts evidence, so /api/challenge/{id}/my-progress reflects the latest data.

When a challenge enters the proof window, the worker performs a final reconciliation fetch to ensure evidence is complete before the evaluator generates a verdict.

npx tsx offchain/workers/progressSyncWorker.ts

API-based providers synced automatically:

• strava — OAuth, token auto-refresh, fetches activities for challenge period
• fitbit — OAuth, token auto-refresh, fetches daily steps + activity logs
• opendota — free public API, fetches Dota 2 matches by Steam32 ID
• riot — API key required, fetches LoL matches by PUUID
• faceit — API key required, fetches CS2 matches by Steam64→FACEIT player ID

Upload-only providers (NOT synced by this worker — no server-side API):

• apple — data pushed from iOS AutoProofService (HealthKit)
• garmin — users upload TCX/GPX/JSON export files
• googlefit — API deprecated by Google in 2025; users upload Google Takeout JSON

How it works:

1. Finds active challenges where startsAt <= now < proofDeadline
2. For each, finds participants with linked API-provider accounts or gaming identity bindings
3. Fetches activity data for the challenge period (start → now for active, start → end for proof window)
4. Upserts evidence row per (challenge, subject, provider) — replaces stale data
5. Progress is automatically visible via GET /api/challenge/{id}/my-progress

Production: Included in ecosystem.config.cjs as progress-sync. Auto-starts with pm2 start ecosystem.config.cjs.

Requires: Migration 025 (evidence_challenge_subject_provider_uq unique index).

Writes to: public.evidence (upsert)

4. Evidence Evaluator

Polls public.evidence for rows that have no corresponding verdict, runs the appropriate evaluator per provider, and upserts the result to public.verdicts.

npx tsx offchain/workers/evidenceEvaluator.ts

Evaluators:

• fitnessEvaluator — for apple, garmin, strava, fitbit, googlefit
• gamingEvaluator — for opendota, riot, steam, faceit

Writes to: public.verdicts

Safety: Unknown providers produce a pass: false verdict to drain the queue rather than block it.

Must run before: challengeDispatcher (dispatcher gates on verdict existence).

5. Challenge Dispatcher

Scans public.challenges for active challenges that have a matching passing verdict and queues them into public.aivm_jobs.

The dispatcher supports two evaluation modes:

Threshold mode (default): dispatches as soon as the challenge subject has a passing verdict. This is the standard pass/fail flow.

Competitive mode: for challenges with rule.mode === "competitive". The dispatcher waits until the proof deadline passes (all evidence is in), then:

1. Fetches all verdicts for the challenge with score IS NOT NULL
2. Ranks participants by score descending
3. Breaks ties by earliest evidence submission (created_at ascending)
4. Marks the top-N participants as winners (pass=true), the rest as losers (pass=false)
5. Enqueues a single AIVM job (normal flow from here)
6. The AIVM indexer calls submitProofFor for each passing participant during finalization

No new environment variables are required for competitive mode. Detection is automatic based on the challenge’s proof.params.rule.mode field.

npx tsx offchain/dispatchers/challengeDispatcher.ts

Reads from: public.challenges (status = active), public.verdicts (pass = true for threshold; score IS NOT NULL for competitive)

Writes to: public.aivm_jobs (status = queued), public.verdicts (pass and metadata updated during competitive ranking)

Idempotent: already-queued challenges are not re-queued (UNIQUE constraint on challenge_id).

6. Challenge Worker

Dequeues jobs from public.aivm_jobs and submits them to the Lightchain AIVM network via requestInferenceV2. Sets job status to submitted. Does NOT attempt commit/reveal/attest — those are performed autonomously by Lightchain workers and validators.

npx tsx offchain/workers/challengeWorker.ts

Required env vars: LCAI_WORKER_PK, AIVM_INFERENCE_V2_ADDRESS, AIVM_TASK_REGISTRY_ADDRESS

Reads from: public.aivm_jobs (status = queued)

Writes to: public.aivm_jobs (status → submitted or failed), public.challenges (binding recorded)

7. AIVM Indexer

Watches events from the Lightchain AIVMInferenceV2 contract and updates the DB as the Lightchain network processes submitted tasks. When InferenceFinalized is observed, triggers the finalization bridge to call submitProofFor and ChallengePay.finalize().

npx tsx offchain/indexers/aivmIndexer.ts

Required env vars: AIVM_INFERENCE_V2_ADDRESS, NEXT_PUBLIC_RPC_URL

Event → DB action:

Checkpoint: last processed block stored in public.indexer_state under key last_aivm_block.

Finalization bridge: idempotent — will not retry if proof.finalizationAttempted is already set. If finalize() reverts (e.g. BeforeDeadline), the error is logged and processing continues.

8. Claims Indexer

Watches ChallengePay claim events and Treasury ClaimedETH events, persisting each claim into public.claims. This is the secondary/hardening source of truth for claimed state. The UI also writes to public.claims immediately after a successful transaction (primary path).

npx tsx offchain/indexers/claimsIndexer.ts

Events indexed: WinnerClaimed, LoserClaimed, RefundClaimed, ClaimedETH

Checkpoint: last processed block stored in public.indexer_state under key last_claims_block.

Writes to: public.claims (upsert on conflict — idempotent)

9. Status Indexer

Watches ChallengePay status-changing events and keeps public.challenges.status aligned with on-chain state. This closes the gap where only aivmIndexer wrote Finalized — canceled challenges now sync automatically.

npx tsx offchain/indexers/statusIndexer.ts

Events indexed: Finalized, Canceled

Checkpoint: last processed block stored in public.indexer_state under key last_status_block.

Writes to: public.challenges.status (idempotent — only updates when status differs)

10. Recommended Startup Order

Option A: PM2 (recommended for persistent deployment)

# Step 1 — DB migration (run once before starting workers)
npx tsx db/migrate.ts
 
# Step 2 — Start all workers with PM2
npm install -g pm2      # one-time install
pm2 start ecosystem.config.cjs
 
# Useful PM2 commands
pm2 status              # check all worker health
pm2 logs                # tail all worker logs
pm2 logs evidence-collector  # tail specific worker
pm2 restart all         # restart everything
pm2 stop all && pm2 delete all  # tear down
 
# Auto-start on reboot
pm2 startup             # generate startup script
pm2 save                # save current process list

Option B: Manual (separate terminals)

# Step 1 — DB migration (run once before starting workers)
npx tsx db/migrate.ts
 
# Step 2 — Workers and indexers (run persistently)
npx tsx offchain/workers/evidenceCollector.ts       # provider APIs → public.evidence
npx tsx offchain/workers/evidenceEvaluator.ts       # public.evidence → public.verdicts
npx tsx offchain/dispatchers/challengeDispatcher.ts # verdicts → public.aivm_jobs queue
npx tsx offchain/workers/challengeWorker.ts         # aivm_jobs → requestInferenceV2 on-chain
npx tsx offchain/indexers/aivmIndexer.ts            # Lightchain AIVM events → finalize
npx tsx offchain/indexers/claimsIndexer.ts          # ChallengePay claim events → public.claims
npx tsx offchain/indexers/statusIndexer.ts         # ChallengePay status events → challenges.status
 
# Gaming: Desktop GSI capture pipeline
npx tsx offchain/workers/wsGateway.ts               # WebSocket gateway (port 3100) for desktop clients
npx tsx offchain/workers/gsiEvidenceBridge.ts       # game_sessions → public.evidence bridge
 
# Webapp (can run at any time independently)
cd webapp && npm run dev

Dependency ordering: evidenceEvaluator must be running before challengeDispatcher, which must be running before challengeWorker. The indexers are independent. For gaming: wsGateway must be running for desktop clients, gsiEvidenceBridge must be running to convert completed game sessions into evidence for the evaluator pipeline.

11. Deployed Contract Addresses (Testnet)

All addresses also stored in webapp/public/deployments/lightchain.json.

Archived contracts (deployed on-chain historically, not part of active product)

Previous contract addresses (superseded)

Roles and admin (current)

Each contract has a unique admin/owner wallet. No single key compromise affects multiple contracts.

MultiSigWallet (Protocol fee custody)

The protocol fee recipient is a 2-of-3 multi-sig wallet at 0x907bdbDd77B06488208C4B0492AF0aaeA4eFFf54.

Workflow: Any signer calls submitTransaction() → 2 of 3 signers call confirmTransaction() → any signer calls executeTransaction(). Signer management (add/remove/replace) is done via confirmed multi-sig transactions to self.

Post-deploy checklist (after any contract redeploy)

Run npx tsx scripts/ops/postDeploySetup.ts to automate all post-deploy steps:

1. Accept ChallengePay admin: ChallengePay.acceptAdmin() from ADMIN_PRIVATE_KEY wallet
2. Accept ChallengeAchievement admin: Set at construction (no accept needed)
3. Accept EventChallengeRouter ownership: acceptOwnership() from ROUTER_OWNER_PK wallet
4. Accept TrustedForwarder ownership: acceptOwnership() from FORWARDER_OWNER_PK wallet
5. Set ChallengePay dispatcher: setDispatcher(worker, true) from CP admin
6. Set ChallengeTaskRegistry dispatcher: setDispatcher(worker, true) from TaskRegistry owner
7. Grant Treasury OPERATOR_ROLE: grantRole(OPERATOR_ROLE, ChallengePay) from TREASURY_ADMIN_PK wallet
8. Update env: Set all NEXT_PUBLIC_*_ADDR in webapp/.env.local to new addresses
9. Rebuild webapp: cd webapp && npm run build

Verification

Run npx tsx scripts/ops/verifyDeployment.ts to check all 23 deployment invariants (bytecode, ownership, roles, dispatchers, wallet uniqueness, balances).

Dispatcher setup (one-time after deploy)

When ChallengeTaskRegistry is redeployed, register the worker wallet as a dispatcher so it can call recordBinding(). This is automated in scripts/deployPoiVerifierV2.ts if LCAI_WORKER_PK is set during deploy.

Testnet AIVM workers

The Lightchain testnet has active native workers that process inference requests for any model. Native workers commit + reveal automatically; native validators attest until quorum, emitting InferenceFinalized. Our aivmIndexer then drives ChallengePay.finalize().

Verified: requests for apple_health.steps@1 and other LightChallenge model IDs are picked up and finalized by native workers — no local simulation needed.

To drive the pipeline locally (e.g. no active workers for a given model):

PRIVATE_KEY=0x... CHALLENGE_TASK_REGISTRY_ADDRESS=0x... \
  CHALLENGEPAY_AIVM_POI_VERIFIER_ADDRESS=0x... DATABASE_URL=... \
  LIGHTCHAIN_RPC=https://rpc.testnet.lightchain.ai \
  npx tsx scripts/_e2e_simulate_aivm.ts

This script is testnet-only. Do not use in production.

12. Ops Scripts Reference

The scripts below are located in scripts/ops/. Each has a clear lifecycle classification.

seedStatusIndexer.ts — one-time, first deploy

Run once before starting statusIndexer on a fresh or newly deployed environment. Sets last_status_block checkpoint to current_block - LOOKBACK so the indexer does not scan from genesis. Safe to re-run (uses ON CONFLICT DO UPDATE).

npx tsx scripts/ops/seedStatusIndexer.ts
# Optional: override lookback (default 50000 blocks ≈ ~12hrs on this chain)
STATUS_INDEXER_SEED_LOOKBACK=100000 npx tsx scripts/ops/seedStatusIndexer.ts

After seeding, start the indexer and it will backfill only recent blocks.

backfillChainOutcome.ts — on-demand, idempotent

Queries challenges WHERE status='Finalized' AND chain_outcome IS NULL and reads the outcome field directly from ChallengePay.getChallenge() on-chain for each one. Run after a new deployment on an existing DB, or if the statusIndexer missed events.

npx tsx scripts/ops/backfillChainOutcome.ts

Safe to run any number of times — only updates rows where chain_outcome IS NULL.

cancelTerminalJobs.ts — on-demand, idempotent

Cancels any queued/failed/processing aivm_jobs rows whose challenges have reached a terminal state (Finalized/Canceled). The challengeDispatcher handles this automatically each poll cycle going forward — this script is only needed for pre-existing stale rows on a fresh deployment.

npx tsx scripts/ops/cancelTerminalJobs.ts

Safe to run any number of times — only updates rows that match the stale condition.

Run AIVM job for a specific challenge manually

DATABASE_URL=... LCAI_WORKER_PK=0x... AIVM_INFERENCE_V2_ADDRESS=0x... \
  npx tsx offchain/runners/runChallengePayAivmJob.ts <challengeId>

Sign an AIVM proof manually (fitness / file-based)

LIGHTCHAIN_RPC=https://rpc.testnet.lightchain.ai \
PRIVATE_KEY=0x... \
AIVM_VERIFIER=0x... \
CHALLENGE_ID=42 \
SUBJECT=0x... \
EXPECTED_CALLER=0x... \
RULE_JSON=data/examples/rule_10k_3x_week.json \
ACTIVITIES_JSON=data/examples/activities_run.json \
  npx hardhat run scripts/ops/signAivmProof.ts --network lightchain

backfillRegistry.ts — on-demand, idempotent

Finds challenges in public.challenges where registry_status != 'success' and attempts MetadataRegistry.ownerSet() for each. The write-once contract policy means AlreadySet reverts are treated as success (the URI is already on-chain).

DATABASE_URL=... BASE_URL=https://app.lightchallenge.ai \
  npx hardhat run scripts/ops/backfillRegistry.ts --network lightchain

Safe to run any number of times. The signer must be the MetadataRegistry owner.

13. MetadataRegistry Architecture

Source model

Write policy

• ownerSet() — write-once. Reverts with AlreadySet if URI already exists.
• ownerForceSet() — explicit overwrite for corrections. Emits distinct MetadataForceSet event with old+new URI.
• ownerClear() — removes URI. Emits MetadataCleared.
• All writes are owner-only. The owner is the system/admin wallet (METADATA_REGISTRY_KEY).
• Creators do not write to MetadataRegistry directly.

Active flow

1. Frontend creates challenge on-chain (ChallengePay.createChallenge)
2. Frontend calls POST /api/challenges → DB upsert (public.challenges)
3. API route attempts MetadataRegistry.ownerSet(challengePay, id, uri) using METADATA_REGISTRY_KEY
4. Result stored in public.challenges.registry_status / registry_tx_hash / registry_error
5. If write fails → registry_status = 'failed' → backfillRegistry.ts retries later

Failure policy

Soft failure with retry. Challenge creation never fails because of a registry write failure. The DB is the primary metadata store for the product. The on-chain registry is for external discovery. Failed writes are logged in registry_status and retried via backfillRegistry.ts.

DB tracking columns

Monitoring

• Query: SELECT id, registry_status, registry_error FROM public.challenges WHERE registry_status IN ('pending', 'failed');
• Any MetadataForceSet event on-chain indicates an admin correction — investigate.
• MetadataCleared events indicate an admin removal.

14. DB Quick-Checks

-- Pending evidence (no verdict yet)
SELECT e.id, e.challenge_id, e.subject, e.provider, e.created_at
FROM   public.evidence e
LEFT   JOIN public.verdicts v
         ON v.challenge_id = e.challenge_id
        AND lower(v.subject) = lower(e.subject)
WHERE  v.id IS NULL
ORDER  BY e.created_at;
 
-- Latest verdicts
SELECT challenge_id, subject, pass, evaluator, updated_at
FROM   public.verdicts
ORDER  BY updated_at DESC
LIMIT  20;
 
-- All AIVM job statuses
SELECT challenge_id, status, attempts, task_id, last_error, updated_at
FROM   public.aivm_jobs
ORDER  BY updated_at DESC
LIMIT  30;
 
-- Jobs stuck waiting on Lightchain network
SELECT challenge_id, status, task_id, updated_at
FROM   public.aivm_jobs
WHERE  status IN ('submitted', 'committed', 'revealed')
ORDER  BY updated_at ASC;
 
-- Recent claims
SELECT challenge_id, subject, claim_type, amount_wei, source, claimed_at
FROM   public.claims
ORDER  BY claimed_at DESC
LIMIT  20;
 
-- Indexer checkpoints
SELECT key, value FROM public.indexer_state ORDER BY key;
 
-- Achievement mints
SELECT token_id, challenge_id, recipient, achievement_type, minted_at
FROM   public.achievement_mints
ORDER  BY minted_at DESC
LIMIT  20;
 
-- Reputation leaderboard
SELECT subject, points, level, completions, victories
FROM   public.reputation
ORDER  BY points DESC
LIMIT  20;

15. Troubleshooting

Evidence evaluator not creating verdicts

1. Check public.evidence has rows: SELECT count(*) FROM public.evidence;
2. Check evaluator logs for unknown provider errors — these produce pass: false verdicts immediately
3. Verify DATABASE_URL is set and the evaluator process can reach the DB

Jobs stuck in queued (never submitted)

1. Check challengeWorker is running
2. Verify LCAI_WORKER_PK wallet has sufficient LCAI for gas
3. Check AIVM_INFERENCE_V2_ADDRESS and AIVM_TASK_REGISTRY_ADDRESS are set correctly
4. Check AIVM_REQUEST_FEE_WEI — if set too low, requestInferenceV2 may revert

Jobs stuck in submitted/committed/revealed (Lightchain not finalizing)

1. Confirm the aivmIndexer is running and checkpointing (query indexer_state)
2. Verify AIVM_INFERENCE_V2_ADDRESS in .env.local matches the live address
3. Check if the AIVM request deadline has expired (~1hr on testnet) — expired requests cannot be finalized. A new request must be submitted.
4. The testnet has active native workers — check whether the task_id appears in AIVM contract events

InferenceFinalized observed but ChallengePay.finalize() reverted with BeforeDeadline

This is normal when the challenge finalize window has not opened yet. The indexer logs the revert and continues. Finalization will succeed when the challenge period ends.

Claims indexer not persisting claims

1. Verify CHALLENGEPAY_ADDRESS and NEXT_PUBLIC_TREASURY_ADDR are set
2. Query indexer_state to check the last_claims_block checkpoint value
3. If checkpoint is far behind current block, the indexer may need time to catch up

AIVM request deadlines on testnet

AIVM requests expire approximately 1 hour after creation on the Lightchain testnet. If an old request has an expired deadline, the finalization bridge will revert. Create a fresh request by re-running the challenge worker job.

16. API Authentication Headers

Webapp API routes that perform writes or return user-specific data support two authentication methods.

Method 1: Wallet Signature (primary — web app)

The frontend sends these automatically via middleware; external callers (scripts, monitoring) must set them manually.

The server verifies that x-lc-signature is a valid EIP-191 signature of the lightchallenge:{x-lc-timestamp} message by x-lc-address. Stale timestamps (older than 5 minutes) are rejected.

Method 2: Transaction Receipt Verification (fallback — mobile clients)

Mobile wallets (via WalletConnect) can perform eth_sendTransaction but not personal_sign reliably on LightChain’s custom chain (ID 8200). For routes that involve an on-chain transaction, the API accepts a fallback:

1. Client sends txHash and subject (wallet address) in the request body (no signature headers needed)
2. Server fetches the on-chain transaction receipt via RPC
3. Server verifies receipt.status == 0x1 (success) and receipt.from == subject
4. Optionally verifies receipt.to matches the expected contract (e.g. ChallengePay)

Routes supporting tx-receipt auth:

• POST /api/challenges — challenge metadata save after createChallenge() tx
• PATCH /api/challenges — challenge metadata update
• POST /api/challenge/[id]/participant — join record after joinChallengeNative() tx

Security notes:

• Only succeeds if a real on-chain tx was sent from the claimed wallet
• The tx must have succeeded (status 0x1)
• When expectedTo is checked, the tx must target the correct contract
• Implementation: verifyByTxReceipt() in webapp/lib/auth.ts

Admin Endpoints

Admin endpoints (e.g. /api/admin/*) additionally require the ADMIN_KEY env var to be set on the server, and the calling address must match the configured admin wallet.

17. Dispatcher Role for submitProofFor

The aivmIndexer finalization bridge calls ChallengePay.submitProofFor() to submit AIVM proofs on behalf of challenge participants. This function is restricted to addresses with the dispatcher role on ChallengePay.

The dispatcher is set via ChallengePay.setDispatcher(address) by the contract admin. The LCAI_FINALIZE_PK wallet must be registered as the dispatcher before the indexer can finalize challenges.

If submitProofFor reverts with an authorization error, check:

1. The LCAI_FINALIZE_PK address is the current dispatcher: query ChallengePay.dispatcher()
2. The admin has called setDispatcher() after the most recent ChallengePay deployment
3. The finalization wallet has sufficient LCAI for gas

See DEPLOY.md Step 4 for setup instructions.

18. Deprecated Columns

public.challenges.aivm_request_started and aivm_request_started_at have no writers or readers in the codebase. They are superseded by public.aivm_jobs.status. Safe to ignore; will be removed in a future migration.

19. Legacy Compatibility

Active product architecture: Lightchain AIVM + PoI (Proof of Inference) via ChallengePayAivmPoiVerifier. No alternate signer-based or manual verifier is part of the active product.

Active verification path: evidence → verdict → AIVM request → Lightchain commit/reveal/attest → InferenceFinalized → ChallengePay.submitProofFor() → ChallengePayAivmPoiVerifier.verify() → ChallengePay.finalize().

Contract classification:

• Core product: ChallengePay, Treasury, ChallengePayAivmPoiVerifier, ChallengeTaskRegistry
• Core product: ChallengeAchievement (soulbound ERC-721 + ERC-5192; read-only dependency on ChallengePay)
• Core product: MetadataRegistry (active on-chain metadata pointer layer; write-once by default, system-managed)
• Admin-only optional: EventChallengeRouter (multi-outcome event routing; admin scripts only, not on user-facing product path)
• Dormant infrastructure: TrustedForwarder (EIP-2771 gasless relay; deployed but not activated, relay disabled by default via RELAY_ENABLED)
• Archived: AivmProofVerifier (Path A EIP-712 trusted-signer; moved to .attic/contracts_archive/, admin scripts archived, not part of active product)

The following legacy artifacts exist in the codebase for backward compatibility but are not part of the active product:

Policy: Do not use legacy ZK/Plonk concepts for new models, admin UX, product flows, or documentation unless explicitly reactivated by a product decision.

20. Health Checks & Monitoring

Process health

All 7 worker/indexer processes log their service name on every poll cycle. A healthy system produces periodic log output. If a process goes silent, it has crashed or hung.

Recommended: Run all workers under a process manager (PM2, systemd, Docker) that auto-restarts on crash. All workers handle SIGINT/SIGTERM gracefully and drain DB connections before exit.

Indexer lag detection

-- Check indexer checkpoint vs chain head
-- If (chain_head - checkpoint) > 100 blocks, the indexer is lagging
SELECT key, value::bigint AS last_block FROM public.indexer_state ORDER BY key;

Compare against chain head: cast block-number --rpc-url $LCAI_RPC

Expected lag: ≤ 12 blocks (CONFIRMATION_BLOCKS) + MAX_BLOCK_RANGE (2000) in worst case.

Job pipeline health

-- Jobs stuck for >1 hour (needs investigation)
SELECT challenge_id, status, attempts, updated_at, last_error
FROM   public.aivm_jobs
WHERE  status NOT IN ('done', 'canceled', 'dead')
  AND  updated_at < now() - interval '1 hour'
ORDER  BY updated_at ASC;
 
-- Dead jobs (exhausted all retries)
SELECT challenge_id, last_error, attempts, updated_at
FROM   public.aivm_jobs WHERE status = 'dead'
ORDER  BY updated_at DESC LIMIT 20;

Wallet balance monitoring

Worker and finalizer wallets must maintain LCAI balance for gas + AIVM request fees.

# Check worker wallet balance
cast balance $LCAI_WORKER_ADDRESS --rpc-url $LCAI_RPC
 
# Check finalizer wallet balance (if separate)
cast balance $LCAI_FINALIZE_ADDRESS --rpc-url $LCAI_RPC

Alert threshold: < 0.1 LCAI (100 finalize transactions at ~0.001 LCAI each).

Evidence pipeline throughput

-- Evidence waiting for evaluation (should stay near 0)
SELECT count(*) AS pending_evidence
FROM   public.evidence e
LEFT   JOIN public.verdicts v
         ON v.challenge_id = e.challenge_id
        AND lower(v.subject) = lower(e.subject)
WHERE  v.id IS NULL;

21. Failure Recovery Procedures

Worker crash and restart

All workers are safe to restart at any time. On restart:

• evidenceCollector: resumes polling; no checkpoint needed (re-scans proof-window challenges and fills missing evidence)
• evidenceEvaluator: picks up unevaluated evidence from DB (idempotent)
• challengeDispatcher: re-scans eligible challenges; ON CONFLICT prevents duplicates
• challengeWorker: claims jobs via FOR UPDATE SKIP LOCKED; in-flight jobs remain in processing and will be retried after timeout
• aivmIndexer: resumes from last_aivm_block checkpoint in DB
• statusIndexer: resumes from last_status_block checkpoint
• claimsIndexer: resumes from last_claims_block checkpoint

RPC failure

If the Lightchain RPC is unreachable:

• Indexers log errors and continue polling (next cycle retries)
• Worker’s AIVM request submission fails → job marked failed, retried on next cycle
• Finalization bridge failures are logged without setting finalizationAttempted, allowing automatic retry

Database failure

• All workers crash on fatal DB errors (correct behavior — process manager restarts them)
• pg.Pool handles transient connection drops automatically — no manual reconnection needed
• On DB recovery, all workers resume normal operation immediately

Stuck jobs recovery

# Reset stuck processing jobs back to queued (safe — FOR UPDATE SKIP LOCKED prevents conflicts)
UPDATE public.aivm_jobs
SET    status = 'queued', updated_at = now()
WHERE  status = 'processing'
  AND  updated_at < now() - interval '30 minutes';
 
# Cancel jobs for challenges that are already finalized
npx tsx scripts/ops/cancelTerminalJobs.ts

Reorg recovery

If a reorg deeper than CONFIRMATION_BLOCKS (12) is suspected:

1. Check indexer_state checkpoint values
2. Run scripts/ops/reconcileDemo.ts to reconcile DB state with on-chain state
3. Run scripts/ops/backfillChainOutcome.ts to fix any stale chain_outcome values

22. Production Deployment Checklist

Pre-deployment

• All env vars from .env.example sections 1-5 are set in webapp/.env.local
• DATABASE_URL points to production PostgreSQL with SSL
• LCAI_WORKER_PK and LCAI_FINALIZE_PK wallets are funded with LCAI
• ADMIN_KEY is set (random secret string, ≥32 chars)
• OAUTH_ENCRYPTION_KEY is set (openssl rand -hex 32)
• NEXT_PUBLIC_RPC_URL and LCAI_RPC point to a reliable RPC endpoint
• Contract addresses in webapp/public/deployments/lightchain.json match live deployment

Database

• npx tsx db/migrate.ts completes without errors
• npx tsx scripts/ops/seedStatusIndexer.ts (first deploy only)
• Verify public.schema_migrations shows all migrations applied

Contracts

• ChallengePay.admin() returns the expected admin address
• Treasury.hasRole(OPERATOR_ROLE, <ChallengePay>) returns true
• ChallengePay.dispatchers(<finalize_wallet>) returns true
• ChallengePayAivmPoiVerifier is set as verifier on relevant challenges

Workers

• All 7 workers start without errors
• evidenceCollector logs provider accounts on first poll
• aivmIndexer logs finalization bridge status (ENABLED/DISABLED)
• statusIndexer and claimsIndexer log their ChallengePay addresses

Webapp

• cd webapp && npm run build succeeds
• /explore page loads and shows challenges
• /challenge/<id> page loads for a known challenge
• Admin panel at /admin authenticates correctly with ADMIN_KEY

Post-deployment

• Monitor indexer lag (section 20) for first 10 minutes
• Verify wallet balances are sufficient
• Run a test challenge through the full pipeline (create → evidence → verdict → AIVM → finalize → claim)

23. Competitive Challenges

Competitive challenges rank participants by score rather than applying a pass/fail threshold. The on-chain contract (ChallengePay) still uses binary outcome (winner/loser), so competitive ranking is resolved off-chain before finalization.

Flow

Participants submit evidence during proof window
        │
        ▼ evidenceEvaluator
public.verdicts (score + metadata populated)
        │
        │ [proof deadline passes]
        │
        ▼ challengeDispatcher (competitive mode)
Ranks all verdicts by score DESC, tie-breaks by earliest created_at
Top-N marked pass=true, rest pass=false
        │
        ▼ AIVM job queued → challengeWorker → requestInferenceV2
        │
        │ [Lightchain network: commit → reveal → attest]
        │
        ▼ InferenceFinalized → aivmIndexer
submitProofFor() for each winner → ChallengePay.finalize()

Migration

• 018_verdicts_score_competitive.sql — adds score (numeric) and metadata (jsonb) columns to public.verdicts. The score column stores the evaluated metric value used for competitive ranking (e.g. total steps, total kills). The metadata column stores structured evaluation details (e.g. match IDs, per-day breakdowns).
• 019_seed_demo_challenges.sql — seeds demo competitive and threshold challenges for testing.

Apply with the standard migration runner:

npx tsx db/migrate.ts

Detection

A challenge is competitive if its rule config contains mode: "competitive". The dispatcher checks the following paths in order:

1. proof.params.rule.mode
2. proof.params.mode
3. params.rule.mode
4. params.mode

The topN value (number of winners) is read from the same paths. Defaults to 1 if not specified.

Tie-breaking

When multiple participants have the same score, the tie is broken by created_at on the verdict row — the participant who submitted evidence earliest wins. This is deterministic and auditable.

DB queries for competitive challenges

-- All competitive challenges
SELECT id, subject, status,
       proof->'params'->'rule'->>'mode' AS mode,
       proof->'params'->'rule'->>'topN' AS topN
FROM   public.challenges
WHERE  proof->'params'->'rule'->>'mode' = 'competitive';
 
-- Ranking for a specific competitive challenge
SELECT subject, score, pass, metadata, created_at
FROM   public.verdicts
WHERE  challenge_id = <ID>
ORDER  BY score DESC NULLS LAST, created_at ASC;

24. PM2 Worker Management

All off-chain workers are managed via PM2 using ecosystem.config.cjs at the project root.

Workers

Starting Workers

# Start all workers
pm2 start ecosystem.config.cjs
 
# Start a specific worker
pm2 start ecosystem.config.cjs --only evidence-evaluator
 
# Restart all (picks up code changes)
pm2 restart all --update-env
 
# Check status
pm2 status
 
# Tail logs (all workers)
pm2 logs
 
# Tail logs (specific worker)
pm2 logs evidence-evaluator --lines 50
 
# Save current process list (for auto-restart)
pm2 save

Auto-Start on Boot (macOS)

PM2 uses launchd on macOS to auto-start workers on system boot:

# Generate and install launchd startup script (requires sudo)
sudo env PATH=$PATH:/opt/homebrew/bin \
  $(which pm2) startup launchd -u $(whoami) --hp $HOME
 
# Save the current process list so PM2 knows what to start
pm2 save

After running these commands, PM2 will automatically restore saved workers on reboot.

To remove auto-start:

pm2 unstartup launchd

Troubleshooting

# Check for crashed workers
pm2 status  # look for "errored" or high restart count (↺)
 
# View recent errors for a specific worker
pm2 logs evidence-evaluator --err --lines 100
 
# Flush all logs (if disk fills up)
pm2 flush
 
# Kill all PM2 processes and daemon
pm2 kill
 
# Restart from scratch
pm2 start ecosystem.config.cjs && pm2 save

Environment

All workers load environment variables from webapp/.env.local via dotenv. Key variables:

• DATABASE_URL — PostgreSQL connection string (required)
• NEXT_PUBLIC_RPC_URL — LightChain RPC endpoint (required for indexers)
• NEXT_PUBLIC_CHALLENGEPAY_ADDR — ChallengePay contract address (required for indexers)
• KEEPER_PRIVKEY — Private key for on-chain transactions (required for challengeWorker, aivmIndexer)

When contract addresses change (e.g. after redeployment), update webapp/.env.local and restart all workers:

pm2 restart all --update-env && pm2 save