Every enterprise AI pitch eventually arrives at the same slide: your company's knowledge, instantly queryable by anyone, powered by an LLM. The demo usually looks convincing. The production system rarely does. A detailed breakdown from Towards Data Science unpacks why that gap exists — and what engineering teams need to build to close it.

The 5% Problem

The central argument is blunt: the demo is roughly 5% of the actual work. Getting an LLM to answer a question using a PDF you've uploaded takes an afternoon. Getting it to reliably answer questions across a company's full, messy, contradictory, permissions-gated knowledge base is an engineering project measured in months.

This is sometimes called building a context layer — a structured system that sits between raw company data and a language model, responsible for fetching, filtering, ranking, and formatting the right information at inference time.

What a Context Layer Actually Contains

A production context layer involves considerably more than a vector database and an embedding model. The key components include:

  • Ingestion pipelines — handling diverse formats (Slack threads, Notion pages, Confluence docs, spreadsheets, PDFs) with format-specific parsing logic, not just dumping text into chunks
  • Chunking strategy — decisions about chunk size and overlap that directly affect retrieval quality; there's no universal default
  • Metadata tagging — attaching author, date, department, and access-level information so retrieval can be filtered correctly
  • Hybrid retrieval — combining dense vector search with sparse keyword search (e.g., BM25) to handle both semantic queries and precise term lookups
  • Re-ranking — a second-pass model that re-orders retrieved candidates before they're passed to the LLM
  • Permissions enforcement — ensuring a query from a sales rep doesn't surface documents only HR should see

Each of these is a non-trivial engineering surface. Each can fail silently.

The 'Company Brain' Framing

The piece uses the term company brain to describe the larger goal: not just a search tool, but an evolving system that reflects what the organization actually knows at any given moment. That requires treating the context layer as a living data product — one that needs monitoring, feedback loops, and regular reindexing as source documents change.

This is a meaningful distinction from static RAG demos. A company's knowledge doesn't sit still. Pricing changes. Org charts shift. Product documentation goes stale. A context layer that isn't continuously updated will confidently answer questions with outdated information — which, in many enterprise contexts, is worse than no answer at all.

Failure Modes to Anticipate

Several failure modes are worth anticipating early:

  1. Retrieval failures — the right document exists but isn't retrieved, often because the query phrasing doesn't match the document's language
  2. Context window stuffing — too many retrieved chunks overwhelm the model, diluting the relevant signal
  3. Hallucination on gaps — when retrieval returns nothing useful, models often generate plausible-sounding but false answers rather than admitting ignorance
  4. Cold start problems — new knowledge bases with sparse content produce poor results until sufficient material is indexed

Implications for Technical Founders and AI Teams

For teams building internal AI tooling or enterprise-facing AI products, the takeaway is largely about expectation calibration and resourcing. A working demo is not a working product. The investment required to go from a compelling prototype to something reliable enough for employees to actually trust — and use — is substantially larger than most initial project scopes account for.

The organizational dimension matters too. Getting clean, well-structured data into the ingestion pipeline requires cooperation from teams that own the source systems. That's often the hardest part, and it has nothing to do with the model.

For companies buying rather than building: the right questions to ask vendors aren't about the LLM — they're about the ingestion architecture, retrieval evaluation methodology, and permission model. Those are the components that determine whether the system works at scale.