Designing Event Driven Python Systems for Reliable AI Workflows

An API endpoint can make an AI workflow look simpler than it is. A request arrives, Python calls a model or another service, and a response comes back. That design works until the job also depends on document processing, image analysis, external APIs, database updates, or a downstream operational action. Then one request may outlive a web connection, cross several service boundaries, and partially succeed before anything reports an error.

The difficult question is no longer, “Can the model produce the right output?” It is, “Can the system determine what happened, resume safely, and avoid doing the same consequential work twice?”

I have built asynchronous Python services using queued workers and PostgreSQL state tracking, and I have maintained Apache Airflow workflows that coordinate data retrieval, classification, database updates, retries, and monitoring. Across those systems, the most reusable reliability patterns were not specific to the model. They were ordinary distributed systems practices applied carefully around probabilistic components.

This article describes those patterns at a general level. It intentionally omits proprietary architecture, customer information, internal endpoints, and performance claims.

An event driven design is useful when it makes incomplete work visible, recoverable, and safe to repeat.

Give every unit of work an identity

The first requirement is a durable job identifier. A browser connection, worker process, or queue delivery is temporary; the logical job is not. The identifier should be created when the application accepts the work and then travel through every event, log entry, state transition, and output record associated with that work.

A practical workflow also needs explicit states. The exact names vary. A request enters received after initial validation and assignment of a durable identifier. It becomes queued after the application records an event for asynchronous processing. It becomes running after a worker claims a particular attempt. It may enter waiting while progress depends on an external system or a scheduled continuation. It becomes completed after the expected output and final state are persisted. It enters retryable_failed when an attempt fails under conditions that may change. It enters manual_review when continuing automatically would be unsafe or ambiguous. It enters terminal_failed when another automatic attempt is not expected to succeed.

These states are not decorative labels. They define which transitions are allowed and which component is permitted to make each transition. If any worker can overwrite any state, the database becomes a log of competing opinions rather than a source of truth.

Persist intent before depending on delivery

A reliable API should not acknowledge work that exists only in memory. A common sequence begins by validating the request. The application then creates the durable job record and records the event that needs to be published. It publishes or dispatches the work only after that state exists. Finally, it returns the stable job identifier.

The database write and message publication are separate operations, which creates a problem with two related writes. The application can persist the job and fail before publishing, or publish the event and fail before recording the corresponding state. For work that cannot be silently lost, a transactional outbox is one useful pattern. The job and an outbox record are committed together, and a separate publisher delivers unsent events. This does not eliminate failure, but it turns an invisible gap into state the system can inspect and reconcile.

The queue should transport intent, not become the only record that the intent existed. Queues are designed to move work. The database should answer what the system currently believes about that work.

Treat the event as a versioned contract

An event should be more deliberate than an arbitrary Python dictionary. At minimum, consumers need enough information to identify the logical job, identify the individual event, interpret the schema, correlate activity across services, and locate the necessary data.

from datetime import datetime
from uuid import UUID

from pydantic import BaseModel, Field


class WorkEvent(BaseModel):
    event_id: UUID
    job_id: UUID
    event_type: str
    schema_version: int = 1
    attempt: int = Field(ge=0)
    occurred_at: datetime
    correlation_id: str
    payload_ref: str

The event can carry a reference to durable input instead of copying a large or sensitive payload through every transport. That keeps contracts smaller, reduces duplicated data, and makes access control easier to reason about. The tradeoff is that the referenced data must remain available and versioned long enough for retries and reprocessing.

Schema versions matter because producers and consumers rarely change at exactly the same time. A worker should reject or route an unsupported version explicitly rather than misinterpreting it as the current structure.

Design for duplicate delivery

Many message systems guarantee that a message will be delivered one or more times. That is useful, but it means a consumer may receive the same logical work more than once. Duplicate delivery is not an unusual edge case. It is part of the contract.

Idempotency means that repeating the same logical operation does not create a second, conflicting outcome. It usually requires more than checking whether an event identifier appeared before. The application must define what “the same logical operation” means at the business boundary.

Several controls work together. A unique idempotency key can identify the logical action. A receipt table can enforce a uniqueness constraint on processed events. Conditional state transitions can succeed only from an expected prior state. Database transactions can persist the result and completion state together. When an outside action supports it, the application can use an external provider key or look up an existing result before creating another one.

In a reusable OCR service, for example, I used idempotent database updates so that retrying processing did not create contradictory records. The same principle becomes more important when a workflow can send a message, create a case, or initiate another externally visible action.

An important distinction is that deduplication and idempotency are not identical. A deduplication cache may forget an old event. An idempotency rule at the business level should still protect the operation if that event returns later.

Retry only failures that may become successful

“Retry on exception” is not a recovery strategy. Some failures are transient: a connection reset, rate limit, temporary provider outage, or database lock may succeed later. Other failures are deterministic: a malformed event, unsupported schema, missing authorization, or invalid record reference will usually fail in the same way on every attempt.

The worker should classify failures and record a structured reason. Retryable failures can use bounded attempts, exponential backoff, and jitter. Deterministic failures should move to a terminal or review state quickly enough to avoid wasting capacity and hiding a problem with data quality behind repeated noise.

Model outcomes need their own classification. A valid response may still be inconclusive, fail a domain constraint, or require review. That is different from a network error and should not automatically enter the same retry loop. Retrying the identical model request may only produce a different answer without resolving the underlying ambiguity.

When a worker exhausts its retry policy, the event should not disappear. A dead letter route or durable failure state should preserve the job identifier, event type, attempt count, last structured error, and enough context for safe replay or manual review.

Make state transitions conditional

Two workers can observe the same job before either finishes updating it. A simple protection is to make state transitions conditional on the state the worker expected to find.

UPDATE jobs
SET status = 'running',
    attempt = attempt + 1,
    started_at = CURRENT_TIMESTAMP
WHERE id = :job_id
  AND status IN ('queued', 'retryable_failed');

If the update affects no rows, the worker does not own that transition. It should stop, inspect the current state when appropriate, and avoid performing the external action.

Tasks that run for a long time may also need a lease or heartbeat. Without one, running can mean either “healthy work is in progress” or “the worker disappeared yesterday.” A lease makes ownership temporary, while reconciliation can identify abandoned attempts and return eligible jobs to a recoverable state.

Use Airflow and event driven workers for different jobs

Event driven services and workflow orchestrators solve related but different problems.

Queued workers are a strong fit when work arrives irregularly through an API or external event, when individual jobs should begin quickly, and when services need to scale independently. The message boundary separates request intake from processing and lets each consumer handle a bounded unit of work.

Airflow is a strong fit for scheduled workflows or workflows with many dependencies. It can retrieve data, prepare inputs, run classification, persist results, generate reporting artifacts, and monitor whether the expected sequence completed. Its DAG makes dependencies, retries, schedules, and historical runs visible.

The two approaches can work together. An event driven service can persist operational work as it arrives, while Airflow handles periodic aggregation, reconciliation, backfills, or batch processing with several stages. The key is to avoid giving both systems ambiguous ownership of the same transition. Each boundary should have a clear source of truth and a clear recovery path.

Observability should follow the job

A request identifier that disappears at the queue boundary is not enough. The correlation identifier should connect the API request, published event, worker attempt, database transitions, dependency calls, and final outcome.

Useful operational signals include queue age, not only queue length. They include time spent in each workflow state, attempt counts, retry reasons, and jobs that remain running beyond their lease. They also include dead letter volume, manual review volume, schema version mismatches, model outputs rejected by deterministic validation, and dependency failures separated from application failures.

These measurements should be reported as actual observations, not assumed guarantees. I have worked primarily on reliable execution and operational visibility; I would not attach a p95 or p99 claim to a system unless that latency had actually been measured under a defined workload.

Logging also requires restraint. A trace should make behavior reconstructable without duplicating prompts, customer records, credentials, or sensitive payloads into a loosely governed logging system. Stable identifiers, state transitions, result classes, dependency timings, and carefully selected metadata are often more useful than a full content dump.

Test the failures the architecture claims to survive

Tests of successful execution prove that the components can cooperate once. Reliability tests should force the boundaries to misbehave.

I would include several failure cases. The same event may be delivered twice. A worker may stop after an external action but before acknowledging the message. A database update may succeed while event publication is interrupted. An old event may arrive after the job has already advanced. A consumer may receive an unsupported schema version. An external request may time out even though the provider completed it. A retry may begin while another worker still holds a valid lease. Reconciliation may also encounter a job whose state conflicts with its external outcome.

These scenarios reveal whether the system has real recovery behavior or only a retry button. They also clarify where an automated retry is safe, where a compensating action is possible, and where the correct response is to stop and request human review.

Reliability is the product of explicit boundaries

Event driven architecture does not make failure disappear. It makes failure observable and gives the application places to respond deliberately.

The API owns validation and durable acceptance. The event contract owns the handoff. The worker owns one bounded attempt. The database owns durable workflow state. The orchestrator owns declared dependencies. Reconciliation owns the gap between what should have happened and what the system can prove happened.

That separation is especially valuable in AI systems. The model can remain probabilistic while the surrounding application enforces deterministic rules about identity, authorization, state, retries, and consequential actions. The result is not a workflow that never fails. It is a workflow that can explain its state, recover safely, and keep incomplete work from becoming invisible.

Next in this series: visual similarity and OCR at scale.