Case overview
This was an AE (adverse event) medical-review assistant for clinical-trial data. The goal was not to replace medical experts, but to turn cross-sheet and cross-visit record checking into a traceable, reproducible, and continuously optimizable Agent workflow.
The work ran from March through August 2026. It started by validating the review experience inside Excel, then explored LobeHub, Univer components, and reusable skills. The final direction moved to Pi Agent + Skill and a local self-harness that froze inputs, candidate configurations, and evaluation. The final version reached 70%+ record-level recall on a controlled validation set without requiring large-scale row-by-row labeling.
Context and problem
Clinical-trial medical review often requires reading AE, medical history (MH), laboratory (Lab), ECG, physical examination (PE), and vital-sign (VS) forms together. The hard part is not spotting one abnormal number; it is deciding whether the records support each other:
- Do AE start and end dates align with the related measurements?
- Is a result linked to a medically plausible AE or MH record?
- Does the same measurement receive a consistent clinical-significance assessment across visits?
- Can every review concern be traced to a subject, visit, form, and source record?
- When a search finds nothing, can the system distinguish “not found in the searched scope” from confirmed missing data?
The project therefore framed output as a record-review concern, not a clinical conclusion. The system proposes follow-up prompts grounded in visible evidence, preserves source locations and limitations, and does not provide diagnosis, treatment, or definitive causality judgments.
Evolution timeline
| Period | Stage | Key work | Capability formed |
|---|---|---|---|
| Mar 2026 | Excel add-in exploration | Validated import, location, and result presentation in the familiar spreadsheet workflow | Confirmed that reviewers need to stay close to the source context |
| May–Jun 2026 | LobeHub + Univer + Skill | Tested spreadsheet components, CSV/XLSX entry, rule checks, MCP connectivity, and a reusable skill contract; extracted subject-level evidence packages before model review | Separated UI, data extraction, rule boundaries, and model judgment |
| Jul 2026 | Pi Agent + Skill | Disabled unnecessary built-in tools and exposed only fact catalog, controlled search, contract reading, experiment notes, and structured submission | Constrained the Agent to retrieved facts with inspectable tool traces |
| Jul–Aug 2026 | Self-harness optimization | Froze development and holdout sets, candidate prompt/tool/model configurations, and compared recall, precision, F1, call complexity, and cost | Replaced one-off prompt tweaking with a replayable optimization and promotion process |
Solution structure
1. Data preparation: turn the workbook into a fact catalog
The first step was not sending the complete workbook to a model. It identified sheet roles, aligned project-level field mappings, built a subject index, and preserved record locations. Each subject then received a compact evidence package containing complete AE/MH records and relevant Lab, ECG, PE, and VS candidates.
The package retained form, row, visit, record date, measurement, result, reference range, clinical significance, and explicit-link fields. Candidate selection controlled context size; it did not become a medical conclusion.
2. Agent review: fact-first, record-level output
The skill contract defined review boundaries, read-only behavior, controlled tools, and the output schema. The Agent read the contract and fact catalog first, searched one AE, measurement, or date range at a time, and produced one concern for each independently reviewable record.
Rules covered date conflicts, AE/MH traceability, measurement-to-AE fit, clinical-significance conflicts, abnormalities preceding an AE start date, AE end dates versus measurement recovery, and missing supporting records. A no-hit search carried an explicit evidence state so it could not be overstated as confirmed missing data.
3. Evaluation loop: a small, stable baseline
The project did not make large-scale annotation a prerequisite. It extracted a structured baseline from existing review materials, then used human confirmation for scope, categories, and disputed items. The evaluation pipeline had three reusable stages:
- Extract and normalize the historical-review baseline;
- Parse the Agent's record-level structured output;
- Calculate Recall, Precision, and F1 with subject, form, and concern-category keys while preserving misses, false positives, and category distribution.
The later self-harness kept development and holdout subjects fully separate. Each candidate recorded its prompt, tool set, model, input hashes, and run trace. Only candidates that passed development were sent to holdout, and a candidate had to preserve holdout recall relative to the current champion before human promotion. This separated real quality gains from simply changing to an easier sample.
My product and method contributions
- Scenario framing: narrowed “AI review” to verifiable record-consistency checks and defined the boundary of expert decisions that should not be automated.
- Interaction path: started from Excel, then explored a spreadsheet workspace, MCP, and Agent combination that lets reviewers navigate back to source records.
- Skill design: separated sheet recognition, field mapping, evidence extraction, medical rules, and output protocol to reduce duplicated model work and rule drift.
- Agent engineering: designed Pi Agent's minimal controlled tool set, record-level submission protocol, evidence states, and run traces so each concern explains what was actually seen.
- Quality system: established baseline normalization, development/holdout isolation, candidate comparison, and human promotion so recall optimization remained auditable.
- Iteration strategy: fixed missed-rule coverage first, then tightened noisy missing-data checks, and only afterward optimized atomic retrieval, output granularity, and model cost.
Results and value
- Reached 70%+ record-level recall on a controlled validation set, establishing a measurable quality baseline without large-scale row-by-row labeling.
- Built the full loop from workbook → subject evidence package → Agent review → structured result → Recall/Precision/F1 evaluation.
- Moved output from generic anomaly summaries to traceable review concerns grounded in a fact catalog and controlled retrieval.
- Reduced prompt drift, evaluation contamination, and irreproducible runs through versioned skills, input hashes, development/holdout isolation, and human promotion.
- Preserved a path for local execution, expert review, result export, and project-specific adaptation in later customer pilots.
The metric is an engineering result for a bounded data scope; it is not a claim about universal performance across all clinical studies or medical-review tasks.
Public case boundary
This case does not publish original customer materials, clinical samples, full prompts, internal evaluation files, or implementation code. The website keeps only the problem framing, system evolution, method, evaluation approach, and result definition—enough to show product judgment and Agent engineering without reconstructing the underlying project.