Playbooks
What actually changed about AI context windows, and why it matters practically
Context windows grew from roughly 4,000–8,000 tokens a few years ago to 1 million or more in current 2026 flagship models — but the number on the spec sheet ("physical" context window) and the amount of that context a model can actually use accurately ("effective" context window) are different things, and developers report real degradation well before the advertised limit. The practical lessons that emerged: context compaction hurts session coherence, quadratic attention cost makes "just paste everything" impractical at scale, and treating a plan file or external memory as a substitute for raw context retention often outperforms relying on the window alone.
Last updated Aug 21 · 18 min read
How big context windows actually got
The trajectory here is a genuinely dramatic one. A technical review of Meta's Llama 3 noted its context was "bumped up to 8,192 tokens from 4,096" — and at the time, that was still described as "quite small w.r.t. modern standards (e.g. GPT-4 is 128K)." That's the scale of change worth holding onto: what counted as a meaningful upgrade a few years ago (doubling from 4K to 8K) is now dwarfed by a single flagship model's context by more than two orders of magnitude.
By 2026, context windows at the frontier have grown again by roughly another order of magnitude. Every current Anthropic Opus, Sonnet, and Fable-line model carries a 1-million-token context window, with only the Haiku 4.5 tier remaining at the older 200K ceiling (Fastio; MorphLLM). On the OpenAI side, the base GPT-5 API supports a 400,000-token context window with 128,000 max output tokens, while newer GPT-5.5 and GPT-5.4 variants push to 1 million-plus tokens via API (Rohan Paul / X; developer documentation on GPT-5.4/5.5). Google's Gemini 3 Pro similarly defaults to a one-million-token window for web, app, and developer use via Vertex AI and AI Studio, with Gemini 3 Flash offering a smaller 200,000-token window optimized for lower-latency chatbot and streaming use cases, and Gemini 1.5 Pro remaining available in some workflows with an upgradeable window reaching two million tokens (AI-Toolbox).
In practical terms, a million tokens is roughly 750,000 words of English text — enough to hold a 1,500-page book, an entire mid-sized codebase, or a year of chat history in a single prompt, at least in theory.
Physical vs. effective context window
The single most important distinction practitioners have converged on is between a model's physical context window (the advertised, marketed token maximum) and its effective context window (the smaller amount it can actually process accurately before quality visibly degrades). Real-world reports on Hacker News describe this gap directly: one developer found a tested model "works okay out to roughly 20-40k tokens" but "degrades significantly" beyond that threshold — despite a much larger advertised window — and found GPT-4o performing better than the alternative they tested under the 128K mark (HN: Gemini 1M context window needle haystack).
This gap is caused, per practitioner discussion, by proprietary filtering: the model or serving system discards tokens judged less relevant rather than treating every token in the window with equal attention weight — meaning the marketed number describes capacity, not necessarily usable, accurate capacity (HN: context window LLM).
Real-world commentary on very large windows is genuinely split, which is itself informative. One HN commenter calls Gemini 2.0's 1M-context PDF ingestion "hopeless" beyond roughly 20-40k tokens with real degradation, while another describes Gemini 2.5's long-context handling as a "breakthrough moment" compared to older models that "get lost, hallucinate, and are pretty much worthless" past 200k tokens (HN: Gemini 1M context window needle haystack). Both accounts are real; they likely reflect genuine version-to-version improvement alongside task-dependent variance — some document types and query patterns degrade sooner than others.
A separate, real mechanism compounds this: LLMs tend to show poorer performance the larger the input size grows, in part because training data doesn't saturate evenly across massive context lengths, and quadratic attention-cost scaling makes "just stuff everything into context" economically impractical even when it's technically possible (HN: needle in haystack test LLM).
Why context compaction hurts, not helps
"Context compaction" — the process by which an AI coding tool or chat session prunes or summarizes older content once a session fills its window — is a necessary mechanism, but real-world reports describe it as actively costly to output quality. HN commenters report that once context fills and aggressive token pruning/compaction kicks in, it "severely diminishes model coherence and memory retention" during the rest of the session (HN: context window LLM).
This matters concretely for agentic coding work, where sessions routinely run long. One developer reports real coding sessions running "about 350k tokens long," arguing that a model capped at 200K tokens "really isn't a sophisticated enough model" for that kind of sustained workload (HN: context window LLM). This is a genuinely real, reported constraint — not every workflow needs anywhere close to that much context, but sustained multi-hour agentic sessions can approach it faster than developers expect.
A related, distinct pattern is the "memory bottleneck": even when individual pieces of context fit comfortably within the window, an agent can lose track of earlier architectural decisions and cross-file relationships over a long session — the problem isn't fitting one file, it's retaining the relationships between files and decisions across a multi-hour task.
Practical examples
Real, documented example — "Flatty" and codebase-flattening tools. A real Show HN project converts an entire codebase into a single flat text file specifically to work around upload and context limitations when feeding large projects to an LLM — direct evidence that context-window limits shape real tooling decisions, not just theoretical discussion (HN: context window LLM).
Real, documented example — GitRead. Another real Show HN project, "GitRead," uses expanded context windows specifically to auto-generate README files directly from a repository's full content — an example of a new class of tool that expanded context made newly practical, rather than merely faster (HN: context window LLM).
Real, documented example — condensing a 55,000-word guide. One practitioner found a 55,000-word guide "too big for a context window" to get coherent answers from directly, so they condensed it into a separate skill/reference file containing key frameworks rather than pasting the whole document — a real, concrete example of working around effective-context limits by pre-summarizing rather than relying on raw window size (HN: context window marketing vs actual).
Real, documented example — plan files as external memory. A concrete practitioner tip circulating on Hacker News: tell the LLM to scan the project and create a markdown plan file to solve the task first, treating the plan file as external memory rather than relying on the model to hold everything in its own context across a long session (HN: context window LLM).
Data and evidence
– Llama 3's context was increased from 4,096 to 8,192 tokens, described at the time as still small relative to GPT-4's 128K — illustrating how fast the baseline moved even within a few years.
– Current 2026 flagship windows: Claude Opus/Sonnet/Fable-line models at 1M tokens (Haiku 4.5 at 200K) (Fastio); GPT-5 API at 400K, with GPT-5.4/5.5 reaching 1M+ (Rohan Paul / X); Gemini 3 Pro at 1M by default, Gemini 3 Flash at 200K, Gemini 1.5 Pro upgradeable to 2M in some workflows (AI-Toolbox).
– Real reported effective-context degradation point: one practitioner reported usable accuracy only out to roughly 20-40k tokens on a tested model despite a much larger advertised window (HN: Gemini 1M context window needle haystack).
– Real reported sustained-session length: agentic coding sessions reaching approximately 350k tokens in practice, per one developer's account (HN: context window LLM).
– ChatGPT's tiered context in practice (distinct from the underlying model's API context): free users 8K tokens, Plus users 32K tokens, Pro users 128K tokens — illustrating that the consumer product experience is often far smaller than the underlying model's technical maximum.
– Evidence not sufficiently verified: there is no single controlled, cross-vendor benchmark in the sources reviewed here that definitively ranks Claude, GPT, and Gemini's effective context windows against each other at a specific token count — the evidence available is real but consists of scattered, sometimes conflicting, practitioner reports rather than one authoritative study.
Comparisons
Physical context window vs. effective context window. The physical window is what's marketed and what a model will technically accept without erroring out; the effective window is the smaller amount it can actually reason over accurately, since proprietary filtering and attention-mechanism limits mean not every token receives equal weight as input length grows. Evaluating a model purely on its advertised maximum, without testing effective performance on your actual task type, is a documented, real mistake practitioners report making.
128K vs. 200K vs. 1M context windows. Moving from 128K to 200K meaningfully extended what fit in a single session for many real workloads (a mid-sized codebase, a long document); moving from 200K to 1M extends raw capacity further but, per multiple real reports, doesn't guarantee proportionally better accuracy — degradation before the advertised ceiling is a recurring, documented pattern rather than an edge case.
RAG vs. stuffing everything into context. Real HN commentary is genuinely split: one engineer reports direct full-context inclusion outperforming retrieval-augmented generation (RAG) for architectural understanding tasks, while another built a GraphRAG system specifically because raw context limits were still a practical bottleneck for their use case (HN: context window marketing vs actual). Neither account contradicts the other once you separate task types — the honest conclusion is that the right approach is workload-dependent, not universally "RAG is obsolete" or "RAG is always necessary."
Real-world use cases
– Codebase-flattening tools (like the real "Flatty" project) that convert an entire repository into one text file specifically to work around context and upload limits.
– Automated documentation generation (like "GitRead") that became newly practical once context windows grew large enough to hold a full repository's content in a single pass.
– Plan-file-as-memory workflows in agentic coding, where a markdown plan file created early in a session serves as durable external memory that survives context compaction better than relying on the model's own retained context.
– GraphRAG systems built specifically to route around context-window limits for large, structured knowledge bases where full-context inclusion still isn't practical.
Common mistakes
– Choosing a model purely based on advertised context-window size, without testing effective accuracy on your actual document type and task at realistic input lengths.
– Assuming a bigger window eliminates the need for RAG or chunking strategies entirely — real practitioner accounts are genuinely split on this, and it depends heavily on task type.
– Letting a long session run until context compaction kicks in automatically, rather than proactively managing what stays in context (e.g., via a plan file) before quality degrades.
– Ignoring the cost dimension of large contexts. Quadratic attention-cost scaling means every request at near-maximum context size costs meaningfully more than a shorter one — a real, practical consideration beyond raw capability.
– Confusing a consumer product's context limit with the underlying model's API context. ChatGPT's free/Plus/Pro tiers cap context well below what the underlying GPT-5-family model supports via API — a common source of confusion about "how big is the context window really."
– Treating "needle in a haystack" test results as proof of genuine comprehension, when commenters note the test reveals only surface-level retrieval capability, not deep reasoning over the full context (HN: needle in haystack test LLM).
Best practices
– Test effective context, not just advertised context, on your actual document type and task before committing to a model or window size for a production workflow.
– Use external memory (plan files, structured notes) for long sessions rather than relying purely on the model's own context retention, especially for agentic coding work.
– Match your retrieval strategy to your task type — full-context inclusion for tasks needing holistic understanding of a moderately sized document, RAG/GraphRAG for large, structured knowledge bases where full inclusion isn't practical or cost-effective.
– Budget for the cost curve, not just the token ceiling. Near-maximum-context requests carry real, disproportionate cost due to quadratic attention scaling — factor this into production cost estimates, not just capability planning.
– Distinguish "needle in a haystack" retrieval tests from genuine reasoning tests when evaluating a model — passing the former doesn't guarantee the latter.
– Re-test effective context periodically as models update — real reports show meaningful version-to-version differences (e.g., Gemini 2.0 vs. 2.5) in how gracefully long-context accuracy degrades.
– Check which context limit actually applies to your use case — a consumer chat product's tier limit, an API's advertised maximum, and the effective usable window are three different numbers that are easy to conflate.
Key takeaways
– Context windows have grown roughly two orders of magnitude in a few years — from single-digit-thousands to 1 million-plus tokens at the frontier across Claude, GPT, and Gemini.
– The advertised "physical" context window and the model's actual "effective" context window are genuinely different numbers — real reports show accuracy degrading well before the advertised ceiling.
– Context compaction, while necessary once a session fills its window, is reported to actively hurt coherence and memory retention — proactive external memory (plan files) is a documented workaround.
– Quadratic attention-cost scaling means very large contexts are both less reliably accurate and more expensive — "just paste everything" isn't a free strategy even when technically possible.
– RAG and full-context inclusion aren't strictly competing approaches — real practitioner accounts support both depending on task type, so the right choice is workload-specific, not universal.
Relevant tools.scult.in resources
For prompt patterns that specifically account for context-window limits and long-session management when working with Claude or ChatGPT, see the Claude and ChatGPT prompt libraries on tools.scult.in.
If you're building a production RAG pipeline, an agentic coding workflow, or any system where getting context management right (not just picking the biggest available window) is the actual engineering problem, that's precisely the kind of architecture decision worth discussing directly — it may be worth a conversation with SCULT.IN about AI agents & automation for help designing a context and retrieval strategy suited to your actual workload rather than defaulting to whichever model has the largest marketed window.
Frequently asked questions
What is a context window in AI?
The maximum amount of text (measured in tokens) a model can process in a single request, including instructions, conversation history, and any documents or code provided.
What is a token?
A unit of text (roughly a word or part of a word) that language models use to measure input and output length, rather than counting raw characters or words directly.
How big is Claude's context window in 2026?
Current Opus, Sonnet, and Fable-line models carry a 1-million-token context window; Haiku 4.5 remains at 200K tokens (Fastio).
How big is GPT-5's context window?
The base GPT-5 API supports 400,000 tokens with 128,000 max output tokens; newer GPT-5.4 and GPT-5.5 variants reach 1 million-plus tokens via API (Rohan Paul / X).
How big is Gemini's context window?
Gemini 3 Pro defaults to 1 million tokens; Gemini 3 Flash offers 200,000 tokens optimized for speed; Gemini 1.5 Pro remains available in some workflows with an upgradeable window up to 2 million tokens (AI-Toolbox).
Why doesn't a bigger context window always help?
Because the model's effective context window — the amount it can actually process accurately — is often smaller than its advertised physical maximum, with real reports of degradation well before the ceiling (HN: Gemini 1M context window needle haystack).
How much context can an AI model actually use well?
It varies by model and task, but real-world reports describe usable accuracy dropping off well before the advertised limit — one account cites roughly 20-40k tokens as the practical ceiling for a specific tested model, even with a much larger advertised window.
What does "context compaction" mean?
The process of pruning or summarizing older content once a session fills its context window, allowing the session to continue — reported to come at a real cost to coherence and memory retention.
Why does context compaction hurt output quality?
Because aggressive token pruning discards information the model may still need, reportedly diminishing coherence and memory retention for the rest of the session (HN: context window LLM).
What is a "needle in a haystack" test for LLMs?
A benchmark that hides a specific piece of information ("the needle") within a long context ("the haystack") to test whether the model can retrieve it — a test of surface-level retrieval, not necessarily deep comprehension (HN: needle in haystack test LLM).
What's the difference between a physical and effective context window?
The physical window is the advertised, marketed token maximum; the effective window is the smaller amount the model can actually use accurately before quality degrades, due to proprietary filtering and attention limits.
Does a large context window reduce the need for RAG?
It's genuinely debated — some engineers report full-context inclusion outperforming RAG for certain tasks, while others still rely on RAG/GraphRAG specifically because raw context limits remain a bottleneck for large or structured knowledge bases (HN: context window marketing vs actual).
What is RAG?
Retrieval-Augmented Generation — a technique where relevant chunks of external documents are retrieved and inserted into a model's context at query time, rather than including the entire knowledge base in every request.
What is GraphRAG?
A variant of RAG that incorporates a knowledge graph's structure (relationships between entities, not just text similarity) into retrieval, used by some practitioners specifically to address raw context-window limitations for large, structured datasets.
Why do sustained AI coding sessions need so much context?
Because real sessions accumulate conversation history, file contents, and tool outputs quickly — one developer reports real sessions reaching roughly 350,000 tokens, well beyond what shorter interactions require (HN: context window LLM).
Is 200,000 tokens enough for agentic coding work?
Not always — one practitioner argues a 200K cap "really isn't a sophisticated enough model" for sessions that reach roughly 350K tokens in real use, though this varies significantly by project size and session length (HN: context window LLM).
Why is context window size marketed so prominently now (100K, 200K, 1M, 2M)?
Because it's become a competitive, headline spec vendors market directly, even though the effective-vs-physical distinction means the raw number alone doesn't fully describe real-world capability.
What is the "memory bottleneck" in long AI sessions?
A pattern where an agent loses track of earlier decisions and cross-file relationships over a long session even when individual pieces of content fit comfortably within context — a retention problem distinct from raw capacity.
Does model performance actually degrade as input size grows, even within the stated limit?
Yes, per real practitioner reports — training data doesn't saturate evenly across massive context lengths, and quadratic attention-cost scaling makes very large contexts both less accurate and more expensive (HN: Gemini 1M context window needle haystack).
Why do "needle in a haystack" tests exist if the context window technically holds the data?
Because holding data in context and actually retrieving/reasoning over it are different capabilities — the test reveals whether a model can find specific information, not whether it deeply comprehends the surrounding material (HN: needle in haystack test LLM).
How do I work around a small context window for a large document?
Condense the document into a shorter reference file covering key frameworks or facts rather than pasting the whole thing, or use RAG to retrieve only the relevant sections per query — both are real, documented workaround patterns.
How do I manage long AI coding sessions without losing coherence?
Create an external plan file early in the session (treating it as durable memory) rather than relying on the model's own context retention across a multi-hour task, a specific practitioner-recommended technique.
How do I test a model's effective context window for my use case?
Run your actual task (not a generic benchmark) at increasing input lengths and check where accuracy meaningfully drops, rather than trusting the advertised maximum as a proxy for real performance.
How do I choose between full-context inclusion and RAG for my project?
Consider your task type: full-context inclusion tends to help with holistic understanding of moderately sized material, while RAG or GraphRAG becomes more valuable as your knowledge base grows larger or more structured than fits reasonably (and affordably) in one request.
How do I reduce the cost of large-context requests?
Trim unnecessary content before sending it, use retrieval to include only relevant sections rather than entire documents, and be aware that near-maximum-context requests carry disproportionate cost due to quadratic attention scaling.
How do I avoid context compaction hurting my session?
Proactively summarize or externalize important decisions into a separate file before the window fills, rather than letting automatic compaction handle it reactively once the session is already near capacity.
How do I know if my AI coding agent is being hurt by context compaction?
Watch for the agent seeming to "forget" earlier decisions, contradicting itself, or losing track of established conventions mid-session — symptoms consistent with the documented coherence and memory-retention cost of compaction.
How do I pick between Claude, GPT, and Gemini based on context window alone?
Don't rely on the raw number alone — check the effective context window for your specific task type, since real reports show meaningful degradation-point differences that aren't captured by the advertised maximum.
How do I structure a long document so a model handles it more reliably?
Break it into clearly labeled sections, and consider using RAG-style retrieval rather than pasting the entire raw document, especially past the effective-context range for your model.
How do I decide if my project's context needs actually require a 1M+ token model?
Estimate your actual typical session/document size first — many real workflows fit comfortably within 200K tokens, and the marketed 1M+ ceiling matters most for genuinely large codebases or extended agentic sessions.
Is the marketed context window a reliable predictor of real-world model performance?
Not on its own — real practitioner reports consistently describe effective performance degrading before the advertised ceiling, making the marketed number a capacity figure rather than a performance guarantee.
Does context window size correlate with reasoning quality?
Not directly — a larger window increases how much material a model can technically process, but reasoning quality over that material depends on separate factors (training, architecture, task type) that the window size alone doesn't determine.
Is there a theoretical limit to how large context windows can grow?
Evidence not sufficiently verified from the sources reviewed here — while windows have grown dramatically, quadratic attention-cost scaling is a real practical constraint on how far this can be pushed economically, though specific future ceilings weren't addressed in available sources.
Why do some engineers still prefer smaller, cheaper models with RAG over huge-context flagship models?
Because for many retrieval-style tasks, a well-tuned RAG pipeline with a smaller model can be more cost-effective and, per some real accounts, just as or more accurate than stuffing a huge document into a large-context flagship model.
Does the language of the input document affect context-window efficiency?
Yes — a real tokenizer comparison found meaningfully different encoding efficiency across languages, meaning equivalent non-English content can consume a different number of tokens (and cost) than English content of the same length, depending on the model's tokenizer.
128K vs. 200K vs. 1M context window — which should I choose?
It depends on your actual workload's typical size and your budget tolerance for larger-context request costs — 128K-200K comfortably covers most single-document or moderate-codebase tasks, while 1M is aimed at very large codebases or extended agentic sessions.
Claude's context window vs. GPT's — which holds up better in sustained real use?
Real accounts are mixed and largely anecdotal rather than benchmarked head-to-head; one practitioner reports GPT-5's "reliable 400K window" outperforming Claude's 200K in sustained sessions, but this is a single account, not a controlled comparison (evidence not sufficiently verified for a general ranking).
Physical context window vs. effective context window — which number should I actually plan around?
Plan around the effective context window for your specific task, established through your own testing — the physical/advertised number is useful for cost and hard-limit planning, not for predicting real accuracy.
RAG vs. full-context inclusion — which is the "modern" best practice in 2026?
Neither has fully displaced the other — real practitioner accounts support both approaches depending on task type, so "best practice" in 2026 is choosing based on your specific workload rather than defaulting to either approach universally.
Consumer chat product context limits vs. API context limits — why are they different?
Consumer products often apply tier-based limits (e.g., ChatGPT free/Plus/Pro tiers) below the underlying model's full API context, for cost and performance reasons distinct from the model's technical maximum.
My AI assistant seems to "forget" earlier instructions in a long session — what's happening?
This is consistent with context compaction or the documented memory-bottleneck pattern — once the window fills, pruning or summarization can discard information the model still needed, or it may lose track of earlier decisions even without literal compaction.
My model's output quality dropped noticeably as my document got longer, but I'm still under the advertised limit — why?
This matches the well-documented physical-vs-effective context gap — the advertised maximum describes capacity, not guaranteed accuracy at that length; your actual effective context window for this task may be smaller than the advertised ceiling.
My RAG pipeline feels redundant now that my model has a 1M-token window — should I remove it?
Not necessarily — test full-context inclusion against your existing RAG pipeline directly on your actual task before removing it, since real accounts are genuinely split on whether large windows make RAG unnecessary for a given use case.
My AI coding session is using way more tokens than I expected — why?
Long agentic sessions accumulate conversation history, tool outputs, and file contents quickly; real reports show sessions reaching roughly 350k tokens, so unexpectedly high usage in extended sessions is a documented, common pattern rather than unusual behavior.
My model passed a needle-in-a-haystack test but still gives shallow answers about my long document — why the mismatch?
Because the test measures surface-level retrieval, not deep reasoning — commenters explicitly note it reveals "only surface-level capability," so passing it doesn't guarantee genuine comprehension of complex material (HN: needle in haystack test LLM).
Should I pay for a model with the largest available context window, or optimize my pipeline instead?
For most workloads, optimizing your retrieval/chunking pipeline is likely more cost-effective than defaulting to the largest available window — reserve very large context windows for tasks genuinely requiring them (large codebases, extended agentic sessions) after testing effective performance.
Is Claude's 200K context (Haiku tier) pricing meaningfully different from its 1M-token tiers?
Pricing and exact tier structures change frequently and weren't independently verified as a specific figure in the sources reviewed here — check current provider documentation directly for up-to-date pricing.
Is Gemini's 1M-context pricing competitive for long-context, high-volume use cases?
Evidence not sufficiently verified as a specific comparison in the sources reviewed here — pricing structures vary and change; check current vendor documentation for exact figures before budgeting a production workload.
Should a startup building a document-analysis product default to the largest context window available?
Not automatically — test effective performance and cost against your actual document types first; a smaller, well-tuned RAG-based pipeline may outperform a brute-force large-context approach on both cost and accuracy for many real workloads.
What's the single most useful mental model to take away about context windows?
Separate the advertised (physical) number from the number that actually matters (effective, task-specific accuracy) — and treat context management (compaction, external memory, retrieval strategy) as an active engineering decision, not something the window size alone solves for you.
Sources
- https://hn.algolia.com/api/v1/search?query=context%20window%20LLM
- https://hn.algolia.com/api/v1/search?query=context%20window%20marketing%20vs%20actual
- https://hn.algolia.com/api/v1/search?query=Gemini%201M%20context%20window%20needle%20haystack
- https://hn.algolia.com/api/v1/search?query=needle%20in%20haystack%20test%20LLM
- https://fast.io/resources/claude-context-window-guide/
- https://www.morphllm.com/claude-context-window
- https://x.com/rohanpaul_ai/status/1953549303638557183
- https://www.ai-toolbox.co/gemini-models/gemini-context-window-token-limits-2026
Need this built into your business?
The free tools and prompts on this site handle the small, solved problems. If what you need is bigger — AI agents & automation, built and maintained for you — that's Scult's day job.
