Building a Production AI Assistant for Employee Knowledge and Operations

A case study in designing a shared assistant that could answer employee questions, investigate technical problems, and take bounded action without collapsing identity, memory, and execution into one trust boundary.

An employee should not have to know which repository, handbook page, dashboard, or subject-matter expert contains the answer to a routine question. The experience I wanted was simple: ask in the team’s existing chat interface, receive a useful answer with supporting context, and continue working.

The engineering behind that experience was less simple. The assistant had to serve technical and non-technical employees, preserve individual context without mixing users, consult shared knowledge, inspect operational evidence, and occasionally run tools. It also needed to explain when the available evidence was insufficient.

I owned the production architecture around the assistant. My work centered on the service boundaries, profile and tool permissions, memory model, isolated execution path, infrastructure as code, deployment verification, observability, and recovery strategy. This case study describes that design at a level that is useful to engineers while leaving out company-specific data, internal identifiers, and security-sensitive configuration.

The case study at a glance

• Problem: Employees needed one conversational entry point to information spread across documentation, repositories, and operational systems.
• My ownership: I designed and implemented the production architecture around the model, including identity isolation, memory boundaries, tool access, sandboxed execution, deployment controls, monitoring, and recovery.
• Architecture: A chat gateway routes authenticated requests into a stateful agent controller. The controller can retrieve personal context and shared knowledge, invoke bounded tools, or send technical work to an isolated runner.
• Difficult design choice: I kept one employee-facing interface while separating support and engineering work into different profiles and trust boundaries.
• Retrieval choice: I routed questions to different retrieval methods based on source type and authority instead of placing every source in one vector index.
• Model and provider choice: I used a strong reasoning model for orchestration, smaller models for bounded supporting work, and a separate fast provider for memory operations, all behind a model gateway.
• Prompt strategy: I separated stable policy, profile instructions, retrieved evidence, tool schemas, and the current request so that authority did not depend on one large prompt.
• Evaluation: I tested retrieval, tool selection, privacy boundaries, abstention, long-context recovery, deployment state, and failure handling, then converted failures into regression cases.
• Next-step planning: Evaluation findings become bounded engineering work with an owner, hypothesis, evidence requirement, success criterion, and regression check.
• Failure strategy: The system distinguishes model uncertainty, tool denial, execution failure, deployment drift, memory failure, and infrastructure failure so that each can be handled differently.
• Result: The implementation established a production foundation for a shared assistant without requiring broad access for every request. I am deliberately not attaching performance or adoption numbers that I cannot substantiate publicly.

The problem was information access, not chat

The visible request might be, “What is the current process for this issue?” A good answer could require several steps. The assistant may need to identify the employee and conversation, search a shared handbook, inspect a read-only repository, compare a current configuration with its owning code, and return a conclusion that separates observation from inference.

A single prompt could not provide those guarantees. The system needed to answer four questions before it acted:

1. Whose context is relevant to this request?
2. Which source is authoritative for the information?
3. Which tools is this profile allowed to use?
4. What evidence is required before the assistant can make a claim?

Those questions became the organizing principles for the architecture.

How the architecture evolved

I did not begin with the final production diagram. I treated the architecture as a sequence of constraints discovered through increasingly realistic use cases.

The first stage established the employee experience: an authenticated message enters through the existing team interface and reaches a model. The second stage made the answer useful by adding authoritative retrieval and bounded tools. The third stage addressed trust by separating profiles, per-user memory, shared knowledge, and command execution. The fourth stage made the system operable through evaluation, tracing, cost attribution, immutable deployment, and recovery.

The architecture and what talks to what

I built the orchestration layer on Hermes and connected it to Slack Socket Mode. The persistent outbound connection meant the assistant did not need a public inbound webhook. The gateway receives the authenticated user, channel, thread, and message, then builds a session key that preserves the correct conversation boundary.

The gateway passes that trusted identity to a profile router. The router selects the support or engineering profile and loads the tool allowlist for that profile. This is where a seemingly small product choice becomes a security boundary. Two employees can use the same interface while receiving different capabilities because their work requires different access.

The selected profile runs inside the stateful agent controller on a Google Cloud controller VM. Docker Compose runs the agent profiles and their supporting services as separate containers. The controller combines the system policy, recent conversation, retrieved context, and tool schemas before calling the model. Provider and model routes are declared in the Hermes configuration, while the memory subsystem uses its own restricted AI gateway. This keeps model selection outside the employee interface and separates the main agent’s authority from background memory processing.

When the model requests information, the controller invokes a tool through a typed adapter. The tool layer includes shared-document search, read-only repository access, private SearXNG search, Crawl4AI extraction, Camofox browser automation, and narrow operational connectors. Each adapter validates its input and returns structured output to the controller. The model never receives a general network socket or unrestricted filesystem merely because it can call a tool.

Technical commands follow a separate path. The controller sends a bounded task over a private, forced SSH transport to an isolated execution runner on separate compute. The runner uses short-lived containers with restricted credentials, network access, filesystem mounts, capabilities, and resources. Command-risk scanning and a smart approval layer distinguish clear false positives from uncertain or genuinely risky operations. The command output returns to the controller, where it becomes evidence for the response. Session state, long-term memory, and controller credentials stay outside the runner.

Three different stores make up the knowledge and memory plane. Honcho stores durable personal context in PostgreSQL and pgvector, with each authenticated Slack user mapped to a distinct memory identity. An Obsidian vault, synchronized through Obsidian Headless, holds shared procedures and runbooks in files that people can inspect and edit. Vendored Lossless Context Management stores raw messages, hierarchical summaries, and source lineage in a profile-isolated SQLite database, so an older part of a session can be recovered after prompt compaction. Keeping these stores separate makes ownership and correction clearer.

Around the runtime is an operations plane. Bitbucket Pipelines tests profiles, plugins, container contracts, shell scripts, Docker Compose configuration, and Terraform. Terraform defines the Google Cloud resources. Container images are resolved to immutable Artifact Registry digests before deployment. After infrastructure is applied, a runtime acknowledgement must match the intended configuration generation, boot identity, and running image digest. Monitoring receives redacted health and incident events. Application-consistent state is encrypted before it is copied to object storage, while derived indexes can be rebuilt.

The important relationship is that the model proposes and interprets, while the surrounding services enforce identity, access, execution, persistence, and deployment state.

The implementation stack and why each piece exists

• Hermes provides the stateful agent runtime, profiles, tool orchestration, delegation, approvals, and Slack gateway.
• Slack Socket Mode provides the employee-facing interface and authenticated runtime identity without requiring public inbound Slack traffic.
• Honcho, PostgreSQL, and pgvector provide per-user durable memory and semantic retrieval. Relational storage remains authoritative.
• Obsidian Headless synchronizes the human-reviewable shared knowledge vault. The vault, rather than an opaque vector database, is the canonical source for team procedures and runbooks.
• Lossless Context Management handles long conversations by retaining raw messages, hierarchical summaries, and their lineage after prompt compaction.
• SearXNG, Crawl4AI, and Camofox provide private search, extraction, and browser capabilities through controlled services.
• Docker Compose and an isolated runner VM separate the stateful controller from disposable command execution.
• Tirith and profile-specific tool allowlists add runtime checks around command risk and capability exposure.
• Terraform, Bitbucket Pipelines, and Artifact Registry make infrastructure, validation, and image identity reproducible.
• Encrypted application backups and runtime attestation support recovery and prove that the expected image and configuration reached the running system.

AI engineering and production engineering have different jobs

One of the most useful ways to reason about the system is to separate AI behavior from the software controls that make that behavior safe and operable.

AI engineering owns model routing, retrieval quality, prompt and context assembly, tool selection, delegated reasoning, abstention behavior, and nondeterministic evaluation. Production engineering owns identity, authorization, network boundaries, state durability, deployment, monitoring, backups, and recovery.

The two sides meet at explicit contracts. Retrieved content must include provenance. Tool calls must use typed schemas. Model routes must be declared and observable. Execution requests must carry a trusted identity and remain inside their assigned capability boundary. Evaluation failures must map back to an owning layer.

A request through the system

Consider an employee asking why an automated process did not complete.

First, the chat gateway binds the message to the authenticated employee and thread. The profile router selects a support profile whose default access is read-only. The controller retrieves relevant personal context and searches the shared knowledge base for the documented workflow.

If documentation answers the question, the assistant can respond with the source. If the employee is asking for a causal diagnosis, the controller uses diagnostic tools to inspect the available evidence. It traces the submitted input, effective configuration, owning code path, and chronological result. A queued record does not count as an execution attempt, and a configuration value does not prove that a running component consumed it.

If repository inspection is required, the support profile can search an approved read-only mirror. If a bounded command is necessary and authorized, the controller dispatches it to the isolated runner. The runner returns output without gaining access to the controller’s memory or secrets.

The controller then constructs the response from the evidence returned by those tools. If a decisive stage cannot be observed, the answer identifies the gap and the next useful check. If the request requires a privileged change, the system produces a handoff instead of silently widening its own authority.

Why I did not make every domain another agent

A common response to a large tool surface is to create a coordinator and a separate agent for every domain. That pattern can reduce tool confusion, but it also introduces additional prompts, model calls, shared-state questions, failure paths, and authorization surfaces.

I used profiles for trust boundaries and delegation for bounded work. The primary agent retains the task-wide plan, evidence, user interaction, and final response. A delegated model receives a small task only when its result can be checked independently. Tools remain grouped behind typed adapters and profile allowlists, so the primary agent does not need unrestricted access simply because the platform supports many capabilities.

This choice keeps coordination proportional to the problem. A new agent is justified when it needs a distinct objective, context, capability boundary, or independent lifecycle. A new tool category alone is not enough.

Where MCP fits

Model Context Protocol is one integration boundary in the system. It is useful when a capability should expose discoverable, typed operations through a service that owns authentication, validation, error handling, and API-specific behavior.

I do not treat MCP as a replacement for the rest of the architecture. Some capabilities are better implemented as local typed adapters. Command execution belongs in the isolated runner. Retrieval from the synchronized vault is a file and authority problem. Identity and authorization remain runtime controls. MCP helps standardize selected integrations, but it does not decide which user may call them or whether their results are trustworthy.

1. Retrieval choice: route by source and authority

I did not treat retrieval as a single vector-search problem. The system contains several knowledge types, and each one has a different owner, freshness model, and privacy boundary.

For shared policies and runbooks, I chose a synchronized Obsidian vault as the canonical source. Obsidian Headless keeps the runtime copy synchronized, while file and text search retrieve the relevant section directly. This preserves document structure and makes corrections visible to people. It also avoids presenting an embedding match as if it were an official policy.

For personal continuity, I used Honcho semantic retrieval backed by PostgreSQL and native pgvector indexes. The user identity comes from the authenticated Slack runtime, not from a name supplied in the prompt. Retrieval is filtered to that user before the result reaches the model.

For long conversations, I used Lossless Context Management as a separate context engine. It stores raw messages, builds hierarchical summaries, and retains source lineage. When older context becomes relevant, the agent can search the compacted history and expand the supporting messages. This is different from personal memory because it answers, “What happened earlier in this session?” rather than, “What should persist about this employee?”

For source code and technical documentation, lexical repository search is often more precise than semantic retrieval. Function names, configuration keys, error strings, and identifiers are exact evidence. The support profile searches an approved read-only mirror, while the engineering path can use its isolated workspace.

Web search and extraction are fallback sources for external information. Private SearXNG and Crawl4AI services return search and extracted content through controlled interfaces, while Camofox supports tasks that require browser interaction. Retrieved web content is treated as untrusted evidence, never as a new instruction.

The retrieval sequence is therefore authority-first. The router identifies the question type, searches the most authoritative eligible source, retrieves only the needed material, and records the source used. Semantic similarity helps find context, but it does not determine which source is allowed to define the answer.

2. Model selection rationale: spend reasoning where it matters

I separated model roles because the most capable model was not necessary for every operation.

The primary employee-facing Hermes agent uses a strong OpenAI reasoning model. I prioritized reliable tool calling, long-context behavior, code understanding, and the ability to distinguish evidence from inference. This model owns the task-wide plan, decides when tools are needed, evaluates their results, and writes the final response.

Delegated subtasks use a smaller model when the work is narrow and independently verifiable. Examples include extracting a constrained fact, comparing two bounded artifacts, or classifying a command for an approval workflow. The parent model retains the overall context and checks the returned result.

Memory derivation uses fast Google Gemini routes through a restricted gateway, while routine Hermes auxiliary tasks use smaller OpenAI models. These jobs run frequently, have structured outputs, and can be validated before their results enter future context. Embeddings use a dedicated OpenAI embedding model with a fixed dimension that matches the pgvector index.

This tiered approach reserves expensive reasoning for decisions that affect the employee while using smaller models for repeatable supporting work. It also makes cost and latency tunable without weakening every part of the system at once.

3. Brand and provider decision: managed, multi-provider, gateway-routed

I chose managed model providers rather than self-hosting the language models. The workload was conversational and uneven, and the system needed strong tool use and reasoning more than predictable continuous GPU utilization. Self-hosting would have added model serving, capacity planning, upgrades, and quality evaluation to an already complex production system.

The primary reasoning, delegation, and routine auxiliary routes use OpenAI models because tool calling, code-oriented reasoning, and Hermes integration were the strongest fit for the main workload. The Honcho memory subsystem uses Google Gemini models for frequent derivation and summarization work because those operations benefit from fast, economical inference. A managed OpenAI embedding model supplies vectors for semantic memory retrieval.

I did not scatter provider decisions through application code. Hermes configuration declares the main, delegated, and auxiliary routes. Honcho uses a separate, restricted AI gateway for memory derivation, summarization, and embeddings, with no tool access or fallback path. Credentials and route selection remain outside the employee conversation.

This design contains provider-specific behavior in one layer, supports fail-closed routing, and makes a later model evaluation possible without rewriting the chat, tool, or memory services. I still pin production routes rather than allowing an automatic provider fallback, because an unreviewed fallback can change latency, tool behavior, data handling, and output quality.

Cost is part of this routing decision. I would attribute usage by profile, model role, tool path, and request class rather than looking only at one provider invoice. That makes it possible to ask whether expensive reasoning improved a measurable failure mode, whether repeated retrieval created unnecessary context, or whether a delegated task cost more than handling it in the parent. A cheaper model is an improvement only when it preserves the required behavior.

4. Prompt design justification: separate authority from context

I used a layered prompt rather than one long set of instructions.

The first layer is the stable system policy, versioned in declarative profile files. It defines data handling, the difference between observation and inference, dangerous-action boundaries, and the requirement to acknowledge missing evidence.

The second layer is the profile contract. The support profile emphasizes plain-language answers, approved sources, read-only investigation, and escalation. The engineering profile includes a broader technical vocabulary and access to an isolated execution path. Each profile sees only the tools it is allowed to invoke.

The third layer is retrieved context. Personal memory, shared documentation, repository excerpts, and tool results are clearly marked as evidence. They do not gain authority to modify the system prompt or expand tool access. This matters because documentation, web pages, issue text, and source comments can contain instructions that were written for a person or can be actively malicious.

The fourth layer is the current request and typed tool schemas. Tool names, parameters, and return types tell the model what operations actually exist. Deterministic runtime checks then enforce canonical paths, authenticated identity, profile allowlists, confirmation state, and runner isolation outside the prompt.

For diagnostic work, I added an evidence contract to the prompt and tool output. A causal claim must identify the observed predicate, the code or workflow stage that consumes it, a counterexample or control when available, and whether the evidence is direct or inferred. When those fields cannot be supported, the model must keep the explanation as a hypothesis.

The prompt therefore guides reasoning and communication, while code and infrastructure enforce authority. That division lets the prompt remain useful without pretending it is a security boundary.

5. Evaluation and improvement: turn failures into regressions

I evaluated the assistant as a pipeline, not only as a model response.

The retrieval evaluation checks whether the system selected the correct source class, preserved user isolation, returned the relevant passage, and avoided using a less authoritative source when an approved source existed.

The reasoning evaluation uses representative questions with expected evidence and acceptable abstentions. It checks whether the answer distinguishes known facts from hypotheses, cites the inspected source, and asks for the next discriminating check when evidence is missing.

The tool evaluation checks correct tool selection, schema-valid arguments, read-only enforcement, path containment, timeout behavior, and safe handling of malformed tool output. Separate cases verify that a support request cannot reach engineering-only execution or another employee’s memory.

The context evaluation tests long sessions, repeated compaction, retrieval from summary lineage, and large tool results. The goal is to verify that the system can recover the supporting source after context has been compressed rather than trusting a summary with no drill-down path.

The deployment evaluation checks image immutability, profile rendering, policy configuration, infrastructure plans, service health, and runtime acknowledgement. A smoke test confirms that the deployed model route and tool surface match the approved configuration.

Layer	Example failure	Evaluation evidence
Retrieval	A less authoritative source overrides an approved procedure	Authority-ranked fixtures and expected passages
Identity	One employee retrieves another employee’s memory	Two-user isolation cases with distinct canaries
Tools	A support request reaches an engineering-only capability	Allowlist, path-containment, and denial tests
Reasoning	A hypothesis is presented as an observed fact	Evidence-contract cases with acceptable abstentions
Context	Compaction loses the source behind an earlier conclusion	Summary-lineage and source-expansion tests
Deployment	The infrastructure update succeeds but the old image keeps serving	Runtime generation and image-digest acknowledgement

I organize improvement as a closed loop: capture a redacted failure, classify the owning layer, reproduce it with a safe fixture, change the smallest responsible component, add a regression test, deploy through the normal pipeline, and watch the relevant operational signal. This prevents a retrieval problem from being “fixed” with a prompt change or an authorization problem from being hidden by a model retry.

The public version of this case study does not include performance or adoption numbers. The evaluation design is the defensible evidence I can discuss: what the system was tested to do, where it was expected to refuse, and how a failure became a durable improvement.

6. Next-step planning: make evaluation produce engineering work

An evaluation result is useful only if it changes what the team builds next. I treat next-step planning as the final stage of the pipeline rather than an informal note after testing.

The primary agent retains task-wide planning and final ownership. Delegated models receive only small, clearly scoped tasks whose outputs can be checked independently. This keeps the overall objective, evidence, and tradeoffs in one place instead of allowing several agents to make incompatible architectural decisions.

For each observed weakness, I create a bounded next step with six fields:

1. Observed failure: What happened, and at which pipeline layer?
2. Evidence: Which redacted trace, fixture, source, or runtime signal demonstrates it?
3. Hypothesis: What is the smallest plausible cause that can be tested?
4. Proposed change: Which single component should change first?
5. Success and stop criteria: What result would support the change, and what result would cause me to stop or change direction?
6. Regression coverage: Which existing privacy, authorization, retrieval, or deployment behavior must remain intact?

I then prioritize the work by user impact, security risk, recurrence, evidence quality, and implementation cost. A privacy-boundary or authorization failure outranks a stylistic answer problem. A repeatable retrieval failure outranks an anecdotal preference. A proposed model upgrade competes against prompt, retrieval, and tool fixes rather than becoming the default response to every weakness.

The resulting loop is: observe, classify, plan, implement, verify, deploy, and watch. This planning step is what turns an agent demonstration into an engineering system that can improve without losing the boundaries already established.

What one production request must reveal

Infrastructure health cannot explain why an answer was wrong. A production trace needs enough redacted information to reconstruct the decision path without turning the logging system into another sensitive-data store.

For each request, I want to identify the request and session, authenticated profile, retrieved source identifiers, model route, delegated tasks, selected tools, validated argument shapes, policy decisions, execution status, response classification, latency, and attributable model usage. Payloads that contain secrets, private conversation content, or sensitive operational data should be omitted or redacted before they reach monitoring.

This trace connects evaluation to operations. If a response used the wrong source, I can investigate routing and retrieval. If the correct tool was denied, I can inspect the profile policy. If the expected image was not serving, I can inspect deployment reconciliation. Observability becomes evidence about the pipeline rather than a large collection of unrelated logs.

The hardest choice: one interface, multiple trust boundaries

The most difficult design choice was whether to build one highly capable assistant or several narrow assistants. A single assistant is easier for employees to discover, but a universal toolset creates unnecessary authority. Separate applications make permissions clearer, but employees have to decide which system understands their question before they can ask it.

I chose one conversational experience backed by multiple profiles. The profiles share the gateway and selected knowledge services, but they do not share the same tools or execution path.

The support profile emphasizes explanation, retrieval, and evidence. It can read approved documentation and repository mirrors. General code execution and broad write access are absent. When support needs to perform a narrow write, such as proposing a shared-document update, it uses a purpose-built operation with a preview, fresh confirmation, one-time proposal, and audit record.

The engineering profile can use a broader toolset and a dedicated sandbox. That access remains separate from the controller’s state and credentials. The sandbox can be replaced without losing conversations or memory.

This design added operational complexity because profiles, policies, and tests had to stay synchronized. It was still the better tradeoff. Capability boundaries became inspectable and testable, while employees kept a consistent entry point.

How I handled failures

“The assistant failed” is not a useful operational category. I separated failures by the layer that owns the recovery.

A reasoning failure occurs when the evidence does not support a conclusion. The response remains explicit about unknowns and can request a discriminating check.

A policy denial occurs when a profile is not permitted to use a tool or path. The system returns a handoff or a narrower alternative. It does not treat a correct denial as an infrastructure outage.

A tool failure includes a connector timeout, malformed response, unavailable source, or validation error. The controller records the tool and stage that failed, while keeping sensitive payloads out of logs.

An execution failure occurs inside the isolated runner. Because the runner is disposable, the system can terminate the container, preserve bounded output, and retry only when the operation is safe to repeat.

A deployment failure occurs when infrastructure changes but the expected runtime does not converge. The deployment pipeline uses immutable images, a serialized reconciliation lease, and a runtime acknowledgement containing the expected generation and image digest. A successful infrastructure command is not enough to declare the deployment healthy.

A state failure affects conversation data, personal memory, shared knowledge, or a derived index. The recovery plan identifies which store is authoritative. Transaction-consistent backups protect primary state, and rebuildable indexes are regenerated after restore.

These distinctions also shaped monitoring. Dashboards and alerts cover service health, controller errors, and redacted incident events. They do not claim to measure answer quality. Diagnostic correctness is checked through evidence contracts and evaluation, not inferred from CPU utilization or an HTTP health check.

What I learned

The first lesson was that an AI assistant is an identity and permissions system before it is a conversational product. The answer is only trustworthy if the system can explain whose context it used, which source it treated as authoritative, and which capabilities were available.

The second lesson was that memory needs separate ownership domains. Personal continuity, shared organizational knowledge, active conversation context, and long-session retrieval solve different problems. Combining them into one store makes privacy, provenance, and correction harder.

The third lesson was that tool use needs stronger contracts than natural-language guidance. Typed inputs, path containment, read-only mounts, purpose-built writes, isolated execution, and independent policy checks make the boundary enforceable.

The fourth lesson was that production deployment needs evidence. Infrastructure as code describes the intended state. Runtime attestation shows what is actually serving requests. Both matter for a stateful assistant whose model, tools, and permissions can change independently.

The final lesson was about product design. Employees do not need to see every internal component, but the system should make its uncertainty and limits visible. A concise answer with a source, a bounded investigation, and an honest handoff are all successful outcomes when they match the evidence and authority available.

This project brought together the parts of engineering I enjoy most: backend architecture, AI integration, security boundaries, infrastructure, observability, and the careful translation of a complex system into a simple employee experience.