Tabular LLMs Can Now Out-Predict Tuned Gradient-Boosted Trees — Here's How They Work

A new class of foundation models treats tabular data the way LLMs treat text — predicting missing columns zero-shot, no training required. On the TabArena benchmark, the best of them now outperform fully tuned XGBoost. Here's what's driving the shift and where the old guard still holds.

For decades, gradient-boosted trees — XGBoost, LightGBM, CatBoost — have been the default answer for structured, tabular prediction tasks. They're fast to train, interpretable enough for most business contexts, and hard to beat with limited data. That dominance is now being challenged by a new class of model: tabular foundation models, or tabular LLMs.

What Tabular LLMs Actually Do

The core idea is deceptively simple. A tabular foundation model treats a spreadsheet row the way a language model treats a sentence — each column is a token, and the model's job is to predict a missing one. Like a text LLM, these models are pretrained on large corpora of tabular datasets, then applied zero-shot to new tables at inference time. No dataset-specific fine-tuning. No hyperparameter search. No cross-validation loops.

This is architecturally significant. Traditional tree-based models need to be retrained from scratch on every new dataset. Tabular LLMs, once pretrained, generalize — they learn what it means to predict a column in general, across many schemas, domains, and data types.

TabArena: The Benchmark That Changed the Conversation

TabArena is the evaluation framework that has brought this comparison into sharp relief. On its leaderboard, the strongest open tabular foundation models now sit above fully tuned gradient-boosted trees — a threshold that seemed out of reach just two years ago.

The benchmark evaluates models across a diverse collection of real-world tabular tasks, measuring accuracy without any dataset-specific tuning for the foundation models. The fact that zero-shot performance can exceed tuned XGBoost on a broad benchmark is a meaningful signal, not a cherry-picked demo.

How the Architecture Works

Most tabular LLMs share a few common design choices:

In-context learning: The model receives a handful of labeled rows from the target dataset as context at inference time — similar to few-shot prompting in text LLMs.
Serialization: Rows are converted into sequences the transformer can process, either as raw numerical tokens, text representations, or hybrid encodings.
Pretraining on diverse tables: Models are trained on thousands of heterogeneous datasets, learning generalizable statistical patterns rather than dataset-specific ones.
No gradient updates at inference: Prediction is entirely forward-pass, making deployment fast and infrastructure-light.

The leading open model in this space — which the original article independently reproduced — demonstrates that the architectural choices around tokenization and attention over column context are where most of the performance gains come from.

Where XGBoost Still Wins

Tabular LLMs are not a universal replacement. The original analysis maps out several conditions where gradient-boosted trees still hold the edge:

Large datasets: With hundreds of thousands of labeled rows, tuned GBTs often recover their advantage. Foundation models shine most in low-data regimes.
High-cardinality categoricals and feature engineering: Domain-specific preprocessing can unlock performance that general pretraining can't replicate.
Latency-sensitive production systems: A tuned LightGBM model is still orders of magnitude faster at inference than a transformer-based predictor.
Regulatory and interpretability requirements: Tree-based models remain easier to audit and explain to non-technical stakeholders.

What This Means for Practitioners

For startup founders and ML practitioners, the immediate implication is practical: tabular LLMs are now worth including in your baseline comparisons, especially when labeled data is scarce or when you need quick iteration without hyperparameter tuning overhead.

The longer-term implication is structural. The pattern of "pretrain once, deploy anywhere" that transformed NLP and computer vision is now arriving in the most unsexy, most business-critical corner of machine learning — the spreadsheet. As these models scale and their pretraining corpora grow, the gap with tuned tree methods is likely to narrow further.

The TabArena leaderboard is worth bookmarking. It's updated regularly and is becoming the de facto standard for honest comparison in this space.