Anthropic's Claude Sonnet 5 is priced at $2 per million input tokens and $10 per million output tokens through the end of summer — closer to previous Sonnet tiers than to Opus. But competitive pricing only matters if quality holds for your specific use case. In a live benchmark built with Claude Code, product builder Claire found Sonnet 5 finishing near the bottom of her personal preference ranking.

Building a Repeatable Benchmark

Most model evaluations are one-off vibe checks — useful in the moment, impossible to compare across time. Claire's How I AI Bench fixes that with frozen inputs, a fixed rubric, and identical tasks run every time a new model drops.

Key design decisions behind the benchmark:

  • 64 total generations across five models: Sonnet 5, Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro
  • Tasks spanning PRDs, prototypes, agentic work, and agent personality
  • A hand-scored 1-to-5 gut rating per output, exported as JSON via a custom HTML scoring page built in ~45 minutes with Claude Code
  • Claude Code pulling from stored session history on her desktop to generate benchmark tasks tailored to her actual work

"The human signal turned out to be the most useful part of the whole benchmark."

Where LLM Judges Fall Short

Claire used both GPT-5.5 and Opus 4.8 as automated judges — and neither was spiky enough. They clustered scores toward the middle of the scale and missed things immediately obvious on a visual pass: broken prototypes, ignored wireframe constraints.

The automated judges ranked Gemini 3 Pro highest and Sonnet 4.6 lowest. Claire's ranking was nearly the opposite. When she applied a 70/30 Claire-to-LLM weighted index, Sonnet 4.6 jumped to first place. That divergence signals the rubric needs to encode more human taste before automated scores can be trusted.

Task-by-Task Model Recommendations

Based on the Claire-weighted index, here's where each model earned its place:

  • GPT-5.5 → PRDs
  • Sonnet 4.6 → Prototypes and chitchat
  • Opus 4.8 / Sonnet 5 → Codebase navigation and dense UI work
  • Sonnet 4.6 → Daily agent work (preferred for personality, not benchmark scores)

Claire pays for API credits to run her OpenClaw agent specifically on Sonnet 4.6 because of how it communicates. No other model matched it on the voice eval — prompts like "ugh, deploys are red again" — and personality turned out to matter as much as performance.

What Breaks in v1

The agentic bug-hunting task was too easy: every model aced it, making it useless as a differentiator. Claire plans to retire it, tighten the rubric, and keep running the benchmark blind with each new release.

Managing Agents From Your Phone

Alessio Fanelli, founder of Kernel Labs and co-host of the Latent Space podcast, walked through a different angle on agentic work: managing autonomous coding agents asynchronously from a mobile device using OpenAI Symphony, Linear, and a cloud VPS.

From Agent Prompter to Agent Manager

Early agentic workflows stalled at the second or third intervention — local runtimes and clunky interfaces killed momentum. Fanelli's setup solves this by moving execution to the cloud and routing communication through multiple channels (Linear, shell, mobile).

The key insight: Symphony isn't complex orchestration infrastructure. It's an opinionated Markdown spec that tells the model how to behave. The models are now reliable enough to lock to it faithfully.

Token Costs as a Feedback Loop

Fanelli tracks token cost per task — a primitive most agent setups omit entirely. Tasks ranged from 15 million to 221 million tokens in his ledger. The 221-million-token job (making an app deployable on Vercel) made complete sense in retrospect. Without that data, there's no way to improve specs or tooling over time.

Other operational lessons from his setup:

  • Purge skills files every few months — models add instructions rather than replace them, and bloated files eventually contradict themselves
  • Give agents better senses — Kernel Labs built Glimpse, a Playwright extension, because agents hitting UI ambiguity was the real bottleneck, not orchestration
  • Heterogeneous data businesses (trading cards, vintage clothing, irregular inventory) are the biggest unlocked opportunity — LLMs are the first technology malleable enough to handle inconsistent, visual, contextual data without heavy preprocessing

The Pokémon Card Demo

Fanelli also demoed a distinctly different use case: using Codex with browser access to hunt for underpriced Pokémon cards on eBay — a live example of applying agent infrastructure to the kind of messy, visual, domain-specific data that previously couldn't be automated at scale.