A token is not always a word
Tokenization maps text into the units a model consumes. Different tokenizers can split the same text differently, including into subword pieces. Punctuation, code, identifiers, and different languages can produce very different token counts. A word-count estimate is useful for rough planning, but it is not an accurate billing or capacity measurement.
Use the tokenizer or usage information appropriate to the selected model when measuring a workload. Do not compare two models by character count alone. The relevant question is how much information each request consumes and whether the output completes the intended task.
Technical foundation: Hugging Face: The Tokenizers library
Reserve room for the answer and the workflow
A context budget must account for instructions, user content, retrieved evidence, conversation history, and tool results. Depending on the service, output and internal reasoning may have additional limits or accounting rules. Check the actual model contract rather than assuming every published context limit has identical semantics.
For an illustrative document assistant, assign separate budgets to the question, evidence, and response. If a retrieved file is too large, select relevant sections with their headings and identifiers. Do not silently remove the exception clause that changes the answer simply to hit a token target.
Conversation memory is a product artifact
An endless transcript contains repeated attempts, abandoned ideas, and temporary errors. Passing all of it forward can make obsolete decisions look current. Instead, maintain a compact task record: accepted requirements, unresolved questions, source references, and the latest approved artifact.
Summaries are lossy. Preserve exact values for important identifiers, quantities, and approvals outside prose summaries. A shopping assistant should read the current cart from application state, not reconstruct it from an earlier conversation. If a user removes an item, a stale summary must not put it back.
Measure before trimming
Log token totals alongside task outcome, with privacy-conscious metadata rather than unnecessary raw content. Compare successful and failed requests. A smaller prompt that causes repeated corrections can consume more resources overall than a slightly larger prompt that finishes correctly.
Remove duplicated instructions first. Then shorten irrelevant tool output, narrow retrieval, and reuse stable context where supported. Keep a regression set for the edits. A cost optimization is successful only if the required quality and safety remain intact. Avoid claiming a fixed saving percentage until the workload has actually been measured.
A context-budget review
- Count tokens using the selected model's tooling.
- Reserve space for the intended output and tool interactions.
- Store authoritative state outside conversation summaries.
- Retain provenance when compressing retrieved evidence.
- Track total usage per successful task, including retries.
