Agent Engineer

I build AI agents. Then I find out if they actually work.

Four projects, one shared engine. I grade them by exact matching and span overlap, not by asking another model if the answer looks right. Every number here is real, including the ones that did not go my way.

See the work github.com/7P3ng or email me

27.8% to 0.0%False positives, Quorum K=3

0.548FieldAgent detection F1

3CI-green repos, public

00 / thesis

One engine, three hard problems.

Most portfolios show demos. These show the evaluation. If an idea is solid, it should hold up across different problems, so I built one core (Quorum's core/) and ran it on three. Where it holds, the numbers are here. Where it does not, that is in the write-up too.

The work, with its evaluation

FlagshipShared kernel originartifact / 01

Quorum

Fans out one finder agent per file, then sends each finding through K=3 skeptic agents that independently try to kill it, routing each task to the cheapest model that can handle it.

The finding

K=3 verification cut false positives from 27.8% to 0.0% (95% CI [11.1, 50.0] to [0, 0]). That came with a real trade-off, recall dropped from 100% to 77.8%. On a held-out real target it found 3/3 genuine bugs with zero surviving false positives, at about $0.25 per run. 58 tests, CI green.

27.8% → 0.0%False positives

100% → 77.8%Recall trade-off

3 / 3Held-out bugs found

~$0.25Per run

Cost-routing note

Quorum routes each task across tiers (DeepSeek, Haiku, Sonnet, Opus). The harness is committed and tested, but the live multi-tier cost number is gated on an operator key, so I am not publishing one rather than fabricate it.

View repo ↗Open live ↗

Live trace UIOpen live ↗

Open Quorum ↗

Adaptive red-teamVendors Quorum core/artifact / 02

Aegis

An adaptive attacker runs a red-team loop against a target on two harmless proxies (extracting a hidden canary string, and an injection sentinel), scored by exact match with no model acting as judge. ASR means attack success rate.

The finding

A reasoning model is significantly more robust on its own (injection ASR 49.3% vs 68.1%, p=0.0012; canary 10.4% vs 21.5%, p=0.010; overall p=0.0002). But the full defense stack erases that gap (1.7% vs 2.8%, p=0.40, not significant), so good defenses matter more than model choice. Attacker adaptation rose 24.0% to 29.9%, significant only after scaling the benchmark, scaling is the honest power lever, not p-hacking. 78 tests, CI green.

49.3% vs 68.1%Injection ASR (p=0.0012)

1.7% vs 2.8%Defense erases gap (n.s.)

24.0% → 29.9%Adaptation, after scaling

78Tests, CI green

View repo ↗Open live ↗

Live demoOpen live ↗

Open Aegis ↗

Contract red-flag finderVendors Quorum core/artifact / 03

FieldAgent

Reads a real commercial contract, flags risk-bearing clauses with span, severity, and a plain-English reason, then grades itself against CUAD legal gold using span overlap.

The finding

Detection F1 = 0.548 (precision 0.741 / recall 0.435), 95% CI [0.460, 0.637] on 20 held-out contracts, +0.21 over a keyword floor. The honest part is the headline, the "agentic chunking" lift was model-specific noise, not a real edge. It looked like +0.45 on one model only because of a truncation artifact (the baseline was silently cutting its output short). A fair rerun collapses it to +0.07, and it ties on Claude Sonnet. 47 tests, CI green.

0.548F1 (P 0.741 / R 0.435)

[0.460, 0.637]95% CI

+0.21Over keyword floor

+0.45 → +0.07Lift, after fair rerun

View repo ↗Open live ↗

Live demoOpen live ↗

Open FieldAgent ↗

Skill-Tuning Council

Methodology / systems designInternal infra, no public URL

A self-improving skill orchestrator that vets its own changes before they ship. Four proxy reviewers (taste, pragmatism, intent, anti-drift) vote on every change; an adversary proposes, editors refine, a merger combines, and disagreements escalate. 576 tests.

576Council tests

AdversaryProposes a candidate change

EditorsRefine and constrain

MergerCombine the candidates

Council4-proxy vote

EscalateDisagreement triggers review

How I evaluate

Deterministic scoring

No model grades its own homework. I score with exact string matching, span overlap (how much of the predicted text region lands on the correct one), and standard statistical tests. If I cannot measure it objectively, I do not claim it.

Adversarial verification

Before a finding reaches you, skeptic agents try to shoot it down. Claims that cannot survive get dropped. In Quorum this cut false positives from 27.8% to 0.0%.

Honest nulls

A null means the effect was not real, or too small to measure. FieldAgent's apparent +0.45 gain turned out to be a measurement quirk. The real number is +0.07, so I report +0.07.

Cost-gated runs

A full Quorum scan costs about $0.25. Every run reproduces offline with a single command.

Contact

If you are building serious agentic systems, or want to know whether someone else's actually hold up, I would genuinely enjoy the conversation.

Emailthomas@thomaspeng.ca GitHubgithub.com/7P3ng ↗