Refundly

Static snapshot of a Refundly evaluation sweep (OpenRouter models + fault injection). It is read-only: the kill switch, approvals and ticket trigger show this notice instead of acting. The runs, verdicts and evidence are the real ones from that sweep. Run it yourself from the repo.

Kill switch engaged. Every mutating tool call is blocked fleet-wide; workers report blocked, never done.

Silent failure rate

–

Payments the agent reported done that PayPal says were not. Invisible to the agent's own scoreboard.

Reported–

Verified–

The overstatement: — claimed successes were false.

Computing escalation policy…

Recent requests

Payment agents score themselves on what they say they did. Refundly checks what PayPal actually did.

Built on my own research on agent reliability and silent failure. A paper I have under review is titled Silent Failure in Automated Model Adaptation, and Refundly is that thesis pointed at your money.

The problem

Rekha was double-charged by a gym app she couldn't cancel. She asks an agent to fix it; the agent reports refund issued. PayPal quietly left it pending. The agent's scoreboard stays green, and Rekha never gets her money back. Research finds this silent failure is up to 48% of failures in single-control tasks and 75.8% on AppWorld. It is the failure nobody sees.

Why today's tools miss it

Observability tools grade the agent with another model as judge. That LLM judge catches silent failure at only ~0.65 AUROC, barely better than a coin flip, because a confident wrong claim reads as success to a model too. Refundly does not ask a model. It reads PayPal's own record and compares, deterministically.

Who it benefits

Rekha, and anyone like her — double-charged, or trapped in a subscription built to be un-cancellable, who gives up because the process is exhausting. Refundly does the chore and proves the money actually moved.
The person managing a parent's money from another city, who needs to know a refund or cancellation truly went through — not that an app said so.
Small sellers on PayPal who must refund their own customers correctly and cannot afford an agent that says "done" when it isn't.
Anyone who would otherwise be silently wronged: refunds lost, cancellations that never happened, charges that quietly came back.

The vision: AI you can trust with your money, because every claimed payment is verified against PayPal, risky actions pause for you, and the whole thing is auditable evidence instead of faith in a self-report.

Roadmap tier — not part of the verification path

The result below is a controlled demonstration on one small open model. Its controls are uninformative by construction, so it is reported here as a research direction and never as evidence about production agents. State verification against the system of record remains the only ground truth Refundly acts on.

Interpretability probe: the representation–action gap

method + reproduction →

On a task built so that surface text and stated confidence carry no signal by construction, the label is still linearly recoverable from the model's internal activations. The information is present internally even when it is absent from the output. A controlled demonstration on one small open model — not a measurement of production agents, whose calibration is the 40-ticket sweep above.

Probe on activations1.00

TF-IDF surface text0.53

Stated confidence0.54

Shuffled-label null0.47

AUROC, 5-fold CV · Qwen-2.5-1.5B · label = does the read-back equal the request. A controlled result: the contrast is the point, not the 1.00. State verification stays ground truth; the probe is the white-box roadmap tier.