An LLM demo can be convincing after only a few hours. It accepts a prompt, produces fluent language, and makes the underlying model look like the product. Production changes the question. The system now has to assemble the right context, preserve state, call tools safely, recover from partial failures, and produce useful behavior repeatedly—not only during a carefully chosen example.
At Gray Falkon, I built and maintained the Python backend for a customer-facing AI assistant that combined document retrieval, product context, conversation management, and tool-assisted reasoning. I also worked on LLM-powered messaging services for classification, translation, and response generation inside broader operational workflows. The applications differed, but the engineering lesson was consistent: the model call was only one component of the system.
This article describes the reusable engineering patterns at a general level and intentionally omits proprietary implementation details, customer data, and internal metrics.
The model generates language. The surrounding system determines whether that language is grounded, timely, recoverable, and useful.
Start with the operational job, not the model
The first design decision is not which model to use. It is defining the job the system must perform and the contract around that job. A production request usually contains more than a user message. It may also require account state, product records, retrieved documents, prior conversation turns, workflow status, authorization information, and rules governing which actions are allowed.
I treat the LLM as a probabilistic component inside a deterministic application boundary. The application owns input validation, data access, state transitions, permissions, persistence, and error handling. The model can interpret language, rank alternatives, or propose an action, but ordinary software should decide whether that action is valid and whether it may execute.
This boundary matters because product success is not the same as model fluency. A response can sound excellent while using stale context, selecting the wrong record, omitting a required action, or leaving the workflow in an inconsistent state.
Context assembly is a data-engineering problem
Retrieval-augmented generation is often summarized as “search, then place the results in the prompt.” That description hides most of the engineering work. The system must determine which sources are authoritative, which records belong to the current user or task, how fresh the information must be, how conflicting evidence should be handled, and what can fit within the available context budget.
A useful context pipeline produces structured evidence rather than an undifferentiated collection of text. For each retrieved item, the application should know its source, scope, version, timestamp when relevant, and relationship to the request. It should also be possible to distinguish “no relevant information exists” from “retrieval failed” and from “the system was not permitted to access that information.” Those states should not collapse into the same empty prompt.
The practical result is that retrieval deserves the same care as any other production data pipeline: validation, filtering, observability, regression tests, and explicit failure modes.
Conversation history is not application state
A transcript can help a model understand the immediate conversation, but it is a poor substitute for explicit state. Important facts should not exist only as sentences several turns back. The application should persist the identifiers, decisions, workflow stage, and durable outputs that later operations depend on.
This separation also makes the system easier to reason about. Conversation history answers, “What has been said?” Application state answers, “What is currently true, what has completed, and what may happen next?” The two can inform each other, but they have different consistency and retention requirements.
Database-backed state also supports recovery. If a worker stops, a provider times out, or a request is delivered more than once, the system can resume from a known transition rather than reconstructing reality from generated text.
Tool use needs a strict execution boundary
Tool calling makes an LLM application much more capable, but it also introduces risk. A generated function name or argument set is a proposal, not an instruction that should automatically be trusted. Before execution, deterministic code should validate the schema, confirm authorization, check relevant business rules, and reject incomplete or ambiguous arguments.
- Validate inputs: Types, required fields, permitted values, and referenced records must be checked outside the model.
- Make operations idempotent where possible: Retrying the same logical request should not create duplicate work or conflicting state.
- Bound execution: Timeouts, retry limits, and clear cancellation behavior prevent one dependency from holding the workflow indefinitely.
- Return structured results: The model should receive a compact success or failure object rather than guessing what happened from unstructured logs.
- Record consequential actions: Auditable transitions make debugging and review possible without treating the conversation itself as the system of record.
A useful rule is that the model may help decide what the user intends, while the application remains responsible for what the system actually does.
Asynchronous workflows change the architecture
Many AI workflows depend on slow or failure-prone services: document processing, browser-mediated operations, external APIs, database work, or downstream reporting. Keeping all of that inside a synchronous web request creates brittle latency and makes recovery difficult.
Event-driven Python services provide a cleaner boundary. The API can validate and persist a request, publish work for an appropriate consumer, and return a stable identifier. Workers can then perform bounded units of work and record their results. Orchestration can coordinate schedules and dependencies, while the database remains the durable source of truth.
This design does not eliminate failure; it makes failure representable. A workflow can be pending, running, complete, retryable, or terminally failed. That is far more useful than a long request that eventually disappears behind a generic timeout.
Assume partial failure
Production reliability comes from treating failure as an ordinary system state. The model provider may time out. Retrieved context may be missing. A queue may deliver an event more than once. A tool may succeed while its response is lost. A downstream service may reject a request after earlier steps have completed.
The response to each case should be designed before deployment. That may mean bounded retries with backoff, idempotency keys, transactional state changes, reconciliation jobs, explicit manual-review states, or a safe fallback that performs no action. The appropriate choice depends on the consequence of being wrong. Retrying a read is different from repeating an external action.
It is also important to preserve the difference between a model-level failure and a system-level failure. A structurally valid response can still be semantically poor, while a strong model response can be unusable because a dependency or state transition failed.
Evaluation has to cover the complete system
A single model benchmark cannot represent production quality. I prefer layered evaluation because each layer answers a different question:
- Contract tests verify schemas, permissions, routing, persistence, and tool behavior.
- Retrieval evaluations test whether the correct evidence is found and irrelevant evidence is excluded.
- Behavioral test cases check expected responses across ordinary, ambiguous, and adversarial inputs.
- Operational metrics measure latency, error rates, retry frequency, queue health, and dependency failures.
- Human review examines qualities that remain difficult to reduce to one automatic score, including usefulness, tone, and whether the response is appropriately cautious.
Regression sets are especially valuable. Prompts, retrieval rules, tool schemas, application code, and model versions can all change behavior. A release should be tested against representative cases, including previous failures, rather than evaluated only through fresh demonstrations.
Observability must follow the request across boundaries
Traditional service monitoring remains necessary, but AI applications also need traces that explain how a result was assembled. Useful diagnostic context can include the request identifier, selected data sources, retrieval outcome, model and configuration version, tool calls, state transitions, timing, and a structured failure reason.
That does not mean indiscriminately logging prompts or sensitive content. Observability should be designed with data minimization and access control in mind. The goal is to reconstruct system behavior without creating a second, poorly governed copy of the underlying data.
The most valuable dashboards connect infrastructure symptoms to product behavior. A healthy process count is not enough if retrieval is returning empty context, structured outputs are failing validation, or a downstream tool is quietly rejecting requests.
What changed between the demo and the product
The demo centered on generation. The product required a much larger set of guarantees:
- The correct user, record, and workflow context had to be selected.
- Durable state had to live outside the conversation.
- Generated tool requests had to pass deterministic validation.
- Slow work had to move across asynchronous boundaries safely.
- Retries had to avoid duplicating consequential actions.
- Failures had to be visible, classifiable, and recoverable.
- Changes had to be tested across the full pipeline, not only at the model layer.
This is the part of AI engineering I find most interesting: connecting probabilistic model behavior to deterministic operational expectations. A useful production AI system is not merely a model wrapped in an API. It is a carefully bounded software system that knows where uncertainty belongs, where validation is mandatory, and how to remain understandable when something goes wrong.
Next in this series: Designing event-driven Python systems for AI workflows.
Leave a Reply