Agentic AI Researcher

How the Twin Forge evolved from a two-agent experiment into an evidence-driven research workflow.

My goal is to develop a proof of concept for uncertainty quantification in cancer digital twins. The proposed uncertainty layer combines particle filtering with a Kalman-family conditional update. Particles can represent nonlinear, multimodal, or discrete uncertainty, while a conditional update can handle suitable continuous states. Before any of that is useful, I need a forecast model that deserves to have uncertainty placed around it.

That requirement has shaped the research more than any particular algorithm.

My industry work in production AI and backend engineering influenced how I approached the problem. A promising model is only one part of a dependable system. Inputs, state, failure modes, tests, recovery, and monitoring all have to be explicit. Research requires the same discipline, with an additional distinction: code that runs is not necessarily evidence that supports a scientific claim.

I first explored a breast-cancer digital twin based on a reaction-diffusion partial differential equation, or PDE. The mathematical framework was promising, but the data available to me did not contain all of the patient-specific inputs and longitudinal measurements needed to reproduce the model as intended. I then moved to brain metastases, where I had longitudinal imaging work, lesion annotations, and a pretrained segmentation model from a separate NYUMets reproduction project.

The move did not make the problem simple. A patient may have several lesions. Existing lesions can grow, shrink, disappear from an image, or become difficult to match across irregular visits. New lesions may become visible between scans, but the first visible date is not necessarily the biological start time. Treatment may also occur between observations. A segmentation network can help define what is visible at a visit, but it does not automatically predict the next visit.

I needed a research process that could examine these possibilities without confusing a plausible mechanism with a validated forecast. That is why I built the Twin Forge.

Version one: the Scientist and the Engineer

The first workflow used two Hermes profiles: an Experiment Scientist and an Experiment Engineer.

The Scientist selected a bounded question, stated the success criteria, and decided what the result meant. The Engineer implemented the experiment, ran it, and returned the code, manifest, and metrics. The result then went back to the Scientist for another round of planning.

This gave the work more structure than a single open-ended agent conversation. The roles had different responsibilities, and the experiment specification created a record of what was supposed to run. A file-backed controller tracked the handoff so that completed evidence could not be silently rewritten as a different experiment.

The first loop was simple:

Scientist plans → Engineer runs → Scientist evaluates → next experiment or stop

The idea was to let the Scientist explore several routes, including a constrained PDE, a reduced-state transition, a compact temporal model, and synthetic known-truth tests for the future uncertainty layer. Hermes provided the role-specific agents and tools, while the repository held the actual research record.

What the first loop exposed

The two-agent workflow ran into problems that looked similar from the outside but had very different meanings.

A missing dependency or incompatible environment was an execution failure. Unclear provenance for pretrained weights was an evidence problem. A request to retrain a backbone or open protected outcomes was a scope decision. A model failing to outperform persistence was a scientific result. In the early workflow, each could appear as the same message: the agents stopped.

The loop also lacked an independent check between implementation and interpretation. A command completing successfully did not prove that the Engineer used the frozen data boundary, preserved the forecast origin, produced the required artifacts, or evaluated the correct endpoint. The Scientist could read the result, but the same role that wanted to advance the research was also deciding whether the handoff was trustworthy.

Another problem was memory. A long transcript could explain what happened during one turn, but it was not a durable scientific record. Without a structured account of supported findings, limitations, stopped candidates, and revisit conditions, the workflow could keep proposing small variations of an idea that had already failed.

Finally, the workflow was difficult to observe. A long-running job, an agent retry, a failed verification, and a completed negative result require different responses. I needed to know which state the research was in without continuously watching a terminal.

These were architecture problems, not prompting problems. Adding a longer instruction to the same two roles would not create independent verification, durable evidence, or operational visibility. This is where my experience with production systems became especially relevant. Reliable workflows separate responsibilities, make state visible, constrain retries, and preserve enough evidence to explain a failure after it occurs.

Version two: separating evidence from interpretation

The next version added an Experiment Verifier and made the experiment contract stricter.

The Scientist now freezes the research question, data boundary, alternatives, success criteria, and stopping rule before implementation. The Engineer runs that specification and writes a manifest and aggregate metrics. The Verifier independently checks the frozen contract, artifact hashes, tests, data boundaries, and reproducibility. Only after verification passes does the Scientist evaluate the scientific result.

Each role has a boundary:

• The Engineer cannot verify its own output.
• The Verifier cannot repair the code or decide that the research should advance.
• The Scientist cannot reinterpret an unverified run as scientific evidence.

If verification fails, the original artifacts remain unchanged. An eligible repair receives a new experiment identifier and uses a bounded recovery path. A negative scientific result can close one candidate while allowing the Scientist to test a materially different candidate. This prevents operational recovery from quietly changing the hypothesis, endpoint, or success criteria.

The file-backed controller enforces the legal state transitions. It freezes specification digests, assigns the next owner, records append-only events, tracks resource use, and stops when the campaign reaches a scientific or operational boundary. Hermes provides the role-specific reasoning and tools, but the controller and experiment artifacts determine what happened. This separation is similar to a production control plane: the agents perform the work, while an external state machine governs what transitions are allowed and records why they occurred.

Adding an Advisor without taking planning away from the Scientist

I wanted the Scientist to lead ordinary planning, much like a PhD student develops and tests ideas between advisor meetings. I did not want every experiment prescribed by a supervisory agent.

The Research Advisor therefore enters at specific points. It checks in after every third completed cycle and reviews the workflow when the Scientist reaches an impasse. The Advisor reads the original research objective and the current aggregate evidence, then recommends whether the Scientist should pursue another authorized route or stop. An advisor memo is guidance, not a new experiment specification and not permission to expand the scope.

This relationship gives the Scientist room to work while still creating a path out of repetitive or unproductive planning. It also distinguishes a research impasse from a technical failure that the Engineer should repair.

Obsidian as research memory

Hermes conversations are useful for reasoning, but they are not the source of truth. Every formal experiment has a specification, run manifest, metrics, verification report, and scientific evaluation. The controller records the state transitions and preserves the hashes of active artifacts.

I use Obsidian as the readable layer over that evidence. After a completed evaluation, the workflow creates or refreshes a learning note that records what the experiment supports, what it does not support, which assets are reusable, and what evidence would justify revisiting the idea. A candidate registry prevents the Scientist from repeating a stopped approach merely by changing a threshold, seed, or parameter.

This design lets memory compound without turning an old note into a new fact. When an Obsidian summary and a formal experiment artifact differ, the formal artifact remains authoritative.

Discord as the monitoring surface

The next improvement was visibility. I connected the workflow to a dedicated Discord research channel. A notifier reads the controller’s append-only events and active heartbeat, then reports when each role starts or completes a turn, when a handoff changes state, when a failure occurs, and when a long-running step is still active.

Discord is a window into the workflow, not the workflow itself. It does not receive patient-level data, protected images, credentials, or full agent transcripts. A notification failure does not alter the experiment state. The messages provide enough context to distinguish an active job, an execution failure, a verification failure, an advisor review, and a scientifically valid stop.

I also created separate Discord-facing bots for research status and workflow coordination. They read the repository’s current state and aggregate records rather than inventing progress from conversation history.

Why the new architecture is an improvement

The current design improves the research process in five ways.

First, it makes claims traceable. A conclusion points back to a frozen question, implementation record, verification report, and evaluation.

Second, it separates kinds of failure. The workflow can respond differently to an environment problem, an evidence problem, a failed scientific criterion, and an exhausted research route.

Third, it supports bounded autonomy. The agents can plan and run several experiments without requiring me to supervise every handoff, while campaign limits, data boundaries, resource budgets, and stopping rules remain explicit.

Fourth, it preserves negative results. A candidate that fails its frozen criteria is recorded as stopped, along with the evidence required to revisit it. The workflow cannot manufacture progress by repeatedly tuning the same failed idea.

Fifth, it makes the work observable. Obsidian explains what the research has learned across experiments, while Discord shows what the system is doing now.

What the workflow taught me about the science

The architecture did not solve the digital-twin problem by itself. It made the limits of the current evidence harder to ignore.

The experiments showed that persistence is a meaningful baseline and that a mechanistic model does not earn credibility merely by generating plausible trajectories. Synthetic experiments also separated predictive usefulness from parameter recovery. A model may produce a reasonable forecast while its growth and diffusion parameters remain poorly identified.

The most recent campaign tested whether released masks and existing metadata could support an operational MRI-visible burden forecast before attempting the particle and Kalman-family uncertainty layer. The workflow stopped because the required geometry and forecast prerequisites were not established within the authorized evidence. Without an adequate center forecast, the real-data uncertainty experiment was not admissible.

That is a useful research outcome. The system did not continue running increasingly complicated models simply because more cycles were available. It preserved the negative gates and identified what new evidence would be needed to reopen the route.

What comes next

The next step is not to place a sophisticated uncertainty method on top of an unsupported forecast. I need either stronger longitudinal evidence, a better-defined observation and geometry contract, or a different digital-twin foundation that can be evaluated against simple alternatives.

The Twin Forge is still a work in progress, but its purpose is now clearer. It is not an automated discovery claim. It is an evidence-driven research system that helps me plan, execute, verify, remember, and monitor a difficult scientific program.

That architecture gives me a better chance of producing a defensible proof of concept. It also gives me a principled way to conclude that a proposed route is not supported by the available data.

For a closer look at the monitoring layer, see Making the Twin Forge Observable.