Kog Claims GPU Inference Speeds That Rival Custom Silicon

Paris-based startup Kog is challenging the assumption that agentic AI requires exotic custom silicon, claiming its inference engine hits 3,000+ tokens per second on standard datacenter GPUs — territory previously reserved for purpose-built chips from the likes of Cerebras and Groq. With $5M raised and a novel architecture/runtime co-design philosophy, Kog is betting the real bottleneck is software, not hardware.

The Prevailing Wisdom Kog Wants to Dismantle

The AI infrastructure industry has coalesced around a convenient story: if you want low-latency inference fast enough to power real agentic workflows, you need specialized silicon. That means exotic chips from the likes of Cerebras, Groq, and SambaNova — companies that have spent years and billions building purpose-built inference hardware.

That thesis got a loud endorsement when Cerebras went public in a blockbuster IPO that valued it at $56 billion, confirming that fast inference is now its own infrastructure category.

Kog thinks the industry is solving the wrong problem.

What Kog Actually Built

Kog is a Paris-based AI infrastructure startup building a real-time inference engine for AI agents, combining low-level GPU engineering with LLM architecture research. It was founded in 2023 by Gaël Delalleau, an École Polytechnique engineer whose career spans cybersecurity research and high-performance GPU work.

The team of 11 includes 10 engineers and researchers, five of whom hold PhDs.

Kog has opened a public tech preview of the Kog Inference Engine (KIE), achieving 3,000 output tokens per second per request on 8× AMD MI300X GPUs and 2,100 tokens/s on 8× NVIDIA H200s — in FP16, with no speculative decoding.

For context, that compares to approximately 100 tokens per second for ChatGPT.

At 3,000 tokens per second, each agent iteration cycle drops from 5 minutes to 10 seconds — meaning a stack running KIE can complete 30 cycles in the time a standard stack completes one.

The Co-Design Thesis

The core technical insight isn't just aggressive kernel tuning. Kog's position is that the runtime can no longer be treated as a separate serving layer — the model architecture itself has to expose the right structure for the engine to exploit. For the fastest single-request inference, architecture and runtime must be designed together. Laneformer is their first model trained from scratch to explore that co-design point.

Kog released the weights and model code of Laneformer 2B on Hugging Face Hub — a 2.3-billion-parameter instruction-tuned coding model designed specifically for high-speed decoding.

The team trained the final model from scratch on approximately 4 trillion pre-training tokens, continued on roughly 2 trillion code and reasoning-heavy tokens, and instruction-tuned on about 210 million tokens.

On the systems side, Kog replaced standard communication layers (RCCL) to unlock linear scaling for tensor parallelism across high-end GPUs — a key chokepoint in multi-GPU inference setups that most frameworks leave untouched.

Why This Matters for Agentic AI

Kog optimizes single-request decode speed specifically because, as AI agents become more autonomous, this is the metric that matters most. "The frontier is shifting from intelligence alone to intelligence multiplied by iteration speed."

The argument is that current LLM inference speed is bottlenecked by software, not the underlying GPU hardware's memory bandwidth ceiling — and that for agents running sequential workflows, per-request decode speed matters far more than aggregate server throughput.

Kog also expects its thousands-of-tokens-per-second results to scale well beyond the current 2B-parameter preview model, because single-request decoding depends on active-parameter count per token, not total parameters — and current frontier MoE models only activate a fraction of their parameters per token.

Funding and Recognition

Kog has raised $5M from Varsity VC and BPI France's Deep Tech Program, and was awarded the French Tech 2030 label in October 2025 — a French government recognition granted to select national deep-tech companies contributing to strategic sectors.

The Competitive Picture

The pitch is potentially enticing as AI enthusiasm gives way to anxiety over soaring operating costs: companies may not need to migrate to a new hardware ecosystem to get dedicated-silicon speeds — they might just need someone to use the GPUs they already own more cleverly.

Kog positions KIE as a drop-in replacement for vLLM, requiring no code refactoring and remaining fully compatible with existing container ecosystems — a pragmatic design decision that dramatically lowers the barrier to adoption for teams already running standard GPU clusters.

For founders building agentic products on top of commodity cloud GPUs, the implication is direct: if Kog's benchmarks hold at scale and across larger models, the specialized-silicon moat may be narrower than the market currently assumes.