The AI Engineering Stack: Tokens, Retrieval, Agents, Evaluation, and Production

AI engineering is more than sending text to a language-model API. A useful application combines a model with data, retrieval, tools, permissions, evaluation, monitoring, and a clear boundary between generated suggestions and actions the software is allowed to take.

Tokens and Context Are Engineering Constraints

Models process tokenised input and produce tokenised output. Token counts affect context capacity, latency, and often cost. More context is not automatically better: irrelevant documents, duplicated instructions, and stale history can make a request harder to reason about. Select only the information needed for the task and preserve the source and timestamp of important facts.

Budget for input, output, tool responses, retries, and any repeated context across turns. Measure actual usage against representative requests rather than relying on a single short prompt.

Choose Between Prompting, Retrieval, and Fine-Tuning

Use prompting when the task can be described with clear instructions and examples. Use retrieval-augmented generation when the answer depends on a changing or private knowledge base that should remain outside model weights. Consider fine-tuning when you need a repeatable behavioural or formatting adaptation and have suitable training data, evaluation capacity, and a model that supports the approach.

These techniques are not mutually exclusive. A system may use a fine-tuned model with retrieval and constrained output. The decision should follow measured accuracy, freshness, privacy, maintenance cost, and failure analysis—not a rule that one method is always correct.

Retrieval Must Be Evaluated Separately

A retrieval pipeline commonly parses documents, splits them into chunks, creates embeddings or other indexes, retrieves candidates, optionally reranks them, and passes selected evidence to the model. Each step can fail. Poor parsing can lose tables; chunks can separate a claim from its context; retrieval can return similar but irrelevant passages; and generation can cite a source that does not support the answer.

Test retrieval recall and relevance using labelled questions. Test answer faithfulness separately. Preserve document identifiers, versions, permissions, and page or section references so the application can show where an answer came from.

An Agent Is a Model With Controlled Access to Tools

An agentic workflow lets a model select tools and sequence steps toward a goal. That can be useful for bounded tasks such as searching an approved knowledge base, preparing a draft, or collecting status from authorised services. It also introduces risk: the model may choose an unsuitable tool, misinterpret untrusted content, repeat an action, or attempt an operation beyond the user's intent.

Keep authority in application code. Enforce user permissions at every tool call, validate arguments, use idempotency for writes, set timeouts and budgets, and require confirmation before external or irreversible actions. Limit loops and define what happens when a tool fails.

Evaluate Before and After Deployment

Build a representative evaluation set and score correctness, source grounding, output validity, refusal behaviour, tool selection, latency, and cost. Include difficult and adversarial examples. Review regressions individually and keep human evaluation for tasks where correctness cannot be reduced to a simple exact match.

Monitor real production traffic with privacy-conscious telemetry. Track model and prompt versions, retrieval changes, tool errors, user corrections, and escalation. A change in a model provider, embedding model, document parser, or prompt can alter the behaviour of the whole system.

Security Is Part of the Architecture

Protect sensitive information, separate trusted instructions from untrusted content, and assume retrieved material can contain prompt injection. Validate model outputs before using them in SQL, shell commands, HTML, or business workflows. Keep tools least-privileged and isolate execution environments. Do not let a model decide its own access rights.

Pick Infrastructure for the Workload

For local development, a lightweight runtime may be sufficient. For shared inference, a serving engine with batching and memory management may be more appropriate. For hosted models, evaluate provider terms, data handling, regional availability, service limits, latency, and total cost. Benchmark with the actual workload and concurrency.

Production AI is a system of evidence, constraints, and feedback loops. The model matters, but reliability comes from the surrounding engineering.