At VB Transform 2026, Ryan Phillips, head of enterprise product at Runway ML, delivered one of the more candid engineering postmortems of the conference — detailing how Runway builds, breaks, and ships its real-time generative video models, and why the lessons are transferable even to teams that never touch model training.

"I think even if you are not all building models yourselves, it's helpful to learn how we do it because I think almost all of the lessons are applicable to what you all are doing day-to-day."

Runway is an applied AI research company focused on general world models. Its flagship demo at the talk was Runway Characters, a real-time video model that enables zero-latency, back-and-forth interaction with AI-generated avatars — a capability that would have required hundreds of hours of manual frame stitching just five years ago.

Building Evals Like a Product, Not a Chore

Phillips made a pointed argument: evaluation isn't an engineering task — it's an organizational one. Building a high-quality eval set requires cross-functional alignment across product, design, research, and sales to agree on what "quality" actually means.

Runway runs internal workshops where team members review generated outputs together, drilling down on specific failure modes. To stress-test the model beyond standard human faces, they used "Tooth" — a non-human character with no nose and unusual teeth — as an edge case in their eval set.

Grading is granular. A video where a character's face remained intact but background elements like a net started morphing was logged as a major failure. The tooling behind all of this? A plain Excel spreadsheet, updated daily with "minor" and "major" failure categories.

"We set a bar before we get started on what percentage we need to pass, and when we hit that, we ship the model. So it's not magical."

For teams dealing with non-deterministic quality drift at scale, Phillips noted that LLMs are increasingly viable as automated visual judges — particularly for detecting the kinds of morphing artifacts common in generative video eval sets.

The Bug That Became a Feature

This was the talk's most instructive sequence. Runway's distillation and adversarial post-training (APT) pipeline — which cuts generation time by 80–90% and recovers visual sharpness lost during distillation — introduced an unexpected side effect: characters would sway or drift off-center during real-time generation.

The team spent weeks trying to patch the model itself. No luck. Then they noticed something: if the user's input image was already perfectly centered, the output stayed stable.

Instead of continuing to fight the underlying model, Runway shipped a frontend feature called "Optimize for Image Quality" — which automatically re-centers the user's avatar before generation begins.

"Turn model limitations into product features so that you can actually expand how the model works. It might feel like a limitation internally, but your customers will not see it that way if you're kind of building this in as a product feature."

Users never saw a bug. They saw a helpful auto-correction tool.

Infrastructure Debugging Goes All the Way to the Hardware

Shortly after launch, Runway noticed 8% of API calls were dropping to 16 frames per second, causing visible stutter. Tracing the cause required full-stack observability — the team used an AI agent powered by Claude alongside Datadog and Sentry to isolate the issue.

The culprit: a single data center in the us-east-1 region. The fix wasn't a config change or a software patch.

"They actually went and physically replaced those GPUs in the data center to fix it."

For enterprise teams shipping real-time AI applications, the implication is uncomfortable but important: hardware anomalies will silently degrade model performance, and you need the observability stack to catch them before users do.

"Failure Hell" and the Future Creative Paradigm

Phillips was candid about the non-linear nature of AI development. Teams routinely get stuck for weeks on a single problem with no visible progress — what he called "failure hell." His advice: don't shortcut it. Consistent iteration eventually triggers sudden, exponential improvement.

Looking further ahead, he argued that the creative role in enterprise teams is shifting. Rather than producing individual assets — one ad, one illustration — designers and marketers will increasingly be responsible for defining worlds, aesthetics, and parameters that agents and real-time models then instantiate dynamically.

That's a significant mental model shift for any team deploying generative AI in production today.