Incident #13 — Per-Participant Queueing (2026-04-30)

A walking challenge where the joiner’s evidence was correct (160% of target) and yet the on-chain payout was zero. Root cause was an off-chain queueing assumption; the contract was correct throughout. Documenting here so the bug class doesn’t return.

What happened

Challenge 13 — “Walking Challenge TestnetV2”

• Created by 0xa969…B886 (the subject in challenges table)
• Joined on chain by 0xa969… (creator) and 0xd25b…E83D (joiner)
• 0xd25b… submitted HealthKit evidence: 1166 steps in the 3-hour window
• Walk-equivalent distance: 1166 × 0.762 m / 1000 = 0.888 km vs 0.5 km goal — passed
• On chain at finalize: both addresses were treated as losers, both received LoserClaimed(amount=0)

The iOS app showed “Payout sent to wallet” with a green checkmark; the chain showed they had lost their stake.

Five cascading bugs

Bug C — the structural one

offchain/dispatchers/challengeDispatcher.ts keyed candidate verdicts on lower(v.subject) = lower(challenges.subject):

-- pre-fix (broken)
where exists (
  select 1 from public.verdicts v
  where v.challenge_id = c.id
    and lower(v.subject) = lower(c.subject)   -- only the creator
    and v.pass = true
)

challenges.subject is the creator. Per the creator-doesn’t-auto-join rule, creators don’t automatically join. Any challenge where joiners (not the creator) have the only passing verdicts produced zero dispatcher candidates → no aivm_jobs row → no proof submitted → no winner mark → all-loser autoDistribute.

The contract ChallengePay.submitProofFor(id, participant, proof) already supported per-participant verification. Only the off-chain layer needed fixing.

Bug D — sweep note misleading

sweepExpiredChallenges hard-coded "deadline-sweep: no evidence submitted" as the finalization note — even when evidence (and a passing verdict) existed. Made forensic triage harder.

Bug E — UI lying about payout

webapp/lib/challenges/lifecycle.ts resolved the autoDistributed branch from verdict_pass (DB) instead of claimedTotalWei + chainOutcome (chain). When a late evaluator iteration set verdict_pass = true (~14h after the proof deadline), the UI flipped to “Reward Sent” even though the on-chain LoserClaimed(amount=0) had already happened.

Bug A — looked like a stride-conversion gap, wasn’t

The original 16:50 evaluator emitted “No activities in period for challengeType”. Looked like a steps→km conversion failure. Investigation showed the current code already does the conversion correctly (offchain/inference/metrics.ts:walkingKm falls back to stepsToKm(steps_count) for type steps). The original failure was a pre-revision evaluator that was iterated on during the same day.

Bug B — late evaluator re-run

The 06:36 next-day verdict (pass=true) came after the user finished iterating on the evaluator code. Not a production race; a dev-loop artifact.

The fix (commits eb5caf42, 5c688fe9, ff80f444)

Schema

db/migrations/050_aivm_jobs_per_participant.sql:

ALTER TABLE public.aivm_jobs ADD COLUMN subject text;
UPDATE public.aivm_jobs j SET subject = lower(c.subject) FROM public.challenges c
  WHERE j.challenge_id = c.id AND j.subject IS NULL;
ALTER TABLE public.aivm_jobs ALTER COLUMN subject SET NOT NULL;
ALTER TABLE public.aivm_jobs DROP CONSTRAINT aivm_jobs_challenge_id_key;
ALTER TABLE public.aivm_jobs ADD CONSTRAINT aivm_jobs_challenge_id_subject_key UNIQUE (challenge_id, subject);

Code

• challengeDispatcher.getThresholdCandidates returns one row per (challenge, passing-verdict participant) pair
• ensureQueuedJob(challengeId, subject) keys on the new composite UNIQUE
• runChallengePayLightchainJob(challengeId, subject) overrides challenge.subject with the participant before judging; markRejected / markDone key on (challenge_id, subject)
• challengeWorker claims by (challenge_id, subject); active-jobs set keyed on the pair so the same challenge runs concurrently for different participants

UI

webapp/lib/challenges/lifecycle.ts autoDistributed branch — order of authority is now:

1. cancelled (status = canceled/rejected) → “Refund Sent” / “No Payout”
2. chainOutcome === 2 && claimedWei === 0 → “No Payout” (incident-#13 shape)
3. claimedWei > 0 → “Reward Sent”
4. Fallthrough → “No Payout”

verdict_pass is intentionally ignored in this branch — it’s a DB-only signal that lags the chain.

Sweep diagnostic

sweepExpiredChallenges now queries evidence/verdict/job counts at finalize time and writes a precise finalizationNote plus a finalizationDiag jsonb so future sweeps tell you exactly why no proof was submitted.

Tests

test/lifecycle.test.ts — 6 cases pinning the autoDistributed paths, with one specifically named “incident #13” that asserts the verdict_pass=true + chainOutcome=Fail + claimedWei=0 shape returns “No Payout”, not “Reward Sent”.

Recovery — admin path was already there

The funds weren’t lost. Per contracts/ChallengePay.sol:917-920, when winnersPool == 0 at finalize, the entire distributable loser pool is granted to the protocol multi-sig as a Treasury bucket allowance. autoDistributeWorker then pushes the allowance to the multi-sig wallet.

For challenge 13, the 0.09 LCAI ended up in 0xd9e56435…1469 (the 2-of-3 protocol multi-sig). scripts/ops/reimburseChallenge13.ts drives the standard multi-sig flow (submit → confirm × 2 → execute) using the three signer keys from wallets.json:

Operational lesson

The protocol multi-sig is the admin recovery path. No forceWithdraw / rescue / escape function is needed in ChallengePay. The contract’s _snapshotAndBook already routes loser-pool dust to the multi-sig when there are no winners. For any future incident where users should have won but didn’t:

1. Identify the affected addresses + amounts
2. Check the multi-sig balance (it will hold the funds if the no-winners branch fired)
3. Submit two multi-sig transactions (one per recipient) using the script as a template
4. Get 2-of-3 confirmations + execute

For incidents where there was a partial winners pool (some passed, some didn’t, but all should have passed): the multi-sig won’t hold the funds — they’re in winner allowances. Either pay top-up from a separate funded wallet, or, for larger events, deploy a Reimbursement helper that the multi-sig calls once.

Process lesson

Three things are worth instituting going forward:

1. Late verdicts shouldn’t relabel finalized challenges. Either make verdicts immutable post-deadline, or add a verdicts.created_at <= proofDeadlineTs guard on the dispatcher and UI.
2. evaluator field in verdicts should be code-versioned. When apple:phase8 overwrites an apple:phase7 verdict, that’s information worth keeping (e.g., a verdict_history table) so the UI can show “we re-evaluated and it now passes” rather than silently flipping the label.
3. The aivm_jobs UNIQUE constraint is now (challenge_id, subject). Any new code that talks to this table must remember it. The historical UNIQUE(challenge_id) assumption was the proximate cause of incident #13.

See also

• AI Verification — the full verification pipeline
• Architecture — system map
• Operations runbook — pm2, deadline sweeps, env handling