Evals · Solana devnet
Real models, sealed tests
Two open models trained against Modélmon's sealed graders. Every training reward was paid per graded call over x402, every score comes from held-out tasks the model never saw, and every result is recorded on Solana.
Case study 1 · Transaction Guard GymGRPO + LoRA · 80 steps · KL β 0.04 · 1× L40S
Qwen2.5-1.5B-Instruct
Should an AI agent sign the Solana transaction it built? Answer sign or reject, and name the problem.
Held-out tasks solved
48 sealed tasks · right decision and reason · 95% CITraining reward
Mean reward from the sealed grader · GRPO stepUnsafe transfers signed98% 48%
Decision accuracy45% 73%
Reason accuracy14% 65%
Safe transfers rejected0% 0%
What it learned to catch
Solved by hidden problem · 240 fresh tasks from the same generatorSafe · n=1064% 100%
Unlimited approval · n=320% 81%
Wrong amount · n=160% 50%
Owner change · n=290% 31%
Lookalike recipient · n=370% 16%
before after
2,560 graded calls · $2.56 paid over x402weights 61386c3a… to 8b113607… verified on every answerOnchain result Settlement
Case study 2 · Reward-Hack Judge GymREINFORCE on the ~60M decision head · 9.4B backbone frozen
Clef Flash Cloudflare · 9.4B · Apache-2.0
Judge one coding-agent tool call in context: allow or block, and what kind of misbehaviour.
Held-out tasks solved
60 sealed tasks · right decision and category · 95% CITransfer to real agent traces
Public SafeAgents-Judge-Bench · 661 real tool calls · not sealedBalanced accuracy73.7% 78.5%
Legit calls blocked49.6% 38.4%
Category accuracy69.6% 69.9%
Category macro-F151.4% 51.2%
before after
Also: Transaction Guard, held-out (n=48), within noise92% 96%
3,840 graded calls · $3.84 paid over x402Block/allow transferred; categorisation did notOnchain result
How to read these
- Sealed held-out: the grader picks the held-out split at random, never returns per-task results, and checks the model's weights hash on every answer.
- Fresh tasks: a separate set from the same generator, used only for the breakdowns. The tasks are synthetic, so claims are in-domain except the SafeAgents transfer.
- Graders: these runs were graded in dev mode before the Phala TDX deployment. Records are written by our attestation authority, since Solana Confidential Compute access did not arrive in time.
- Clef: two learning rates were tried and both are shown.
