Evals · Solana devnet

Real models, sealed tests

Two open models trained against Modélmon's sealed graders. Every training reward was paid per graded call over x402, every score comes from held-out tasks the model never saw, and every result is recorded on Solana.

Case study 1 · Transaction Guard GymGRPO + LoRA · 80 steps · KL β 0.04 · 1× L40S

Qwen2.5-1.5B-Instruct

Should an AI agent sign the Solana transaction it built? Answer sign or reject, and name the problem.

Held-out tasks solved

48 sealed tasks · right decision and reason · 95% CI
Untrained
4% ±6
Trained
60% ±14
0255075100

Training reward

Mean reward from the sealed grader · GRPO step00.250.50.751020406080
Unsafe transfers signed98% 48%
Decision accuracy45% 73%
Reason accuracy14% 65%
Safe transfers rejected0% 0%

What it learned to catch

Solved by hidden problem · 240 fresh tasks from the same generator
Safe · n=1064% 100%
Unlimited approval · n=320% 81%
Wrong amount · n=160% 50%
Owner change · n=290% 31%
Lookalike recipient · n=370% 16%
before after
Hidden drains (n=20): the trained model rejects every one (15% before), but names the wrong reason, so they don't count as solved.
2,560 graded calls · $2.56 paid over x402weights 61386c3a… to 8b113607… verified on every answerOnchain result Settlement
Case study 2 · Reward-Hack Judge GymREINFORCE on the ~60M decision head · 9.4B backbone frozen

Clef Flash Cloudflare · 9.4B · Apache-2.0

Judge one coding-agent tool call in context: allow or block, and what kind of misbehaviour.

Held-out tasks solved

60 sealed tasks · right decision and category · 95% CI
Released head
58% ±12
Trained · lr 2e-5
60% ±12
Trained · lr 1e-4
85% ±9
0255075100

Transfer to real agent traces

Public SafeAgents-Judge-Bench · 661 real tool calls · not sealed
Balanced accuracy73.7% 78.5%
Legit calls blocked49.6% 38.4%
Category accuracy69.6% 69.9%
Category macro-F151.4% 51.2%
before after
Also: Transaction Guard, held-out (n=48), within noise92% 96%
3,840 graded calls · $3.84 paid over x402Block/allow transferred; categorisation did notOnchain result
How to read these
  • Sealed held-out: the grader picks the held-out split at random, never returns per-task results, and checks the model's weights hash on every answer.
  • Fresh tasks: a separate set from the same generator, used only for the breakdowns. The tasks are synthetic, so claims are in-domain except the SafeAgents transfer.
  • Graders: these runs were graded in dev mode before the Phala TDX deployment. Records are written by our attestation authority, since Solana Confidential Compute access did not arrive in time.
  • Clef: two learning rates were tried and both are shown.