My best CRO win was +25%. The retest said minus 14%.

I ran a test on a quiz flow. It came back +25%. I wrote it down as a win, told the client, and moved on to the next fix.

Someone re-ran that same test later. The result was minus 14%.

Nothing had gone wrong with the tooling. The number was real the first time and real the second time. What was wrong was me: I treated a single run as a fact. That is the failure mode I built @di-atomic/cro-optimizer to stop, and it is far more common than anyone admits.

The gap nobody staffs

I already had two skills for conversion work. cro-audit diagnoses a page and ranks what is broken. cro-actions turns those findings into a sequenced plan and names the exact component to install. Between them you can find a problem and ship a fix in an afternoon.

Then the fix goes live and the loop just... stops.

Nobody owns the question that comes next. Did it work? Was the number real? Should it stay? What did we learn that makes the next audit smarter? In most programs the answer is a Slack message that says "nice, +18%" and then everyone moves on. The lift gets remembered as a fact. It gets quoted in the next pitch. It becomes the prior that shapes the next round of decisions.

A program that does that is not learning. It is compounding its own noise.

Four numbers that changed how I read a result

I built the skill around 18 real test outcomes, each with a metric and a live source. Four of them do most of the work.

About half of all tests prove nothing. An analysis of 115 A/B tests found 58 of them, 50.4%, were statistically null. So the honest default posterior for any fresh result is "probably nothing happened." If your program reports a win most weeks, that is not a good program. That is a program with a measurement problem.

Checking early turns a 5% error rate into 26%. Evan Miller's classic write-up puts a number on peeking: stop a test the moment it looks significant and your false-positive rate climbs from 5% to 26.1%. Peek ten times and a result you call "1% significant" is really about 5%. Everybody peeks. Almost nobody adjusts for it.

Wins overstate production by 20 to 50%. It is called the winner's curse, and it is structural, not sloppiness: you selected that variant because its noise happened to point up. So the raw test number is the ceiling, never the estimate. My rule is a haircut to 50-80% of raw before anything gets written down.

The headline metric can win while the business loses. A founder removed the credit-card requirement from trial signup. Signups went up 340%. Trial-to-paid went down 60%. Read only the first number and you ship a disaster and celebrate it.

What the skill actually does

cro-optimizer is guidance-only. There is no backend, no model of my own doing statistics. The reasoning is the product, and it is enforced by two scripts that run on every verdict.

The significance gate checks five things: the significance floor, duration of at least 7 days and 2 business cycles, power of at least 80% against a pre-declared MDE, no peeking, and a thin-sample warning. The causal-overreach gate checks five more: was the lift discounted, is there certainty language, is the estimate stable rather than the interim peak, were confounds checked (segment split, seasonality, novelty), and is a guardrail metric named.

Both must pass. Not "should" — a result is not a verdict until both come back clean. Where PostHog already ran the experiment I delegate the peeking and sample-ratio diagnosis to PostHog rather than rebuild it, then add the parts it does not do: the discount, the guardrail, and the write-back.

Every verdict comes out in the same shape. One of five conclusions (won, lost, inconclusive, stopped_early, invalid), the exact deploy version it refers to, both gate results, the discounted lift rather than the raw one, the guardrail metric that was checked, and a next action.

That last field matters more than it looks. If the verdict is lost, the next action is a rollback through SpiderPublish, not a note in a doc. Roughly 1 in 14 shipped variants actively hurts. Leaving one live because reverting felt like admitting a mistake is an ongoing, compounding loss.

And if the verdict survives everything, the discounted number gets written back into cro-audit's baseline as a verified brand prior. That is the loop closing. The next audit on that brand starts smarter than the last one, because it is standing on a number that was actually checked.

The rule I would keep if I could only keep one

If a brand has no analytics wired, the skill stops. It does not estimate, it does not eyeball a before-and-after, it does not offer a "directionally positive" read. It returns one instruction: wire PostHog or GA4 first.

That is the same thing I say about outbound. You can always optimize once you have a number. You cannot optimize zero. A CRO program running without measurement is not a slower program, it is a different activity entirely: it is redecorating.

Small confirmed wins beat one big redesign

The best pattern in the whole registry is not dramatic. Restroworks ran four iterations on homepage and contact messaging and form width. The lifts were 16%, then 5%, then 20%, then 7%. Together: +52% leads.

Nobody would put any one of those in a case study. Compounded and confirmed, they beat the redesign that everyone actually wants to do. That is what generateNextHypothesis is biased toward — iterate on the confirmed winner, and only escalate to a redesign when the evidence says the page is structurally wrong rather than badly worded.

@di-atomic/cro-optimizer is live on the OPVS marketplace. It composes cro-audit and cro-actions, reads from PostHog or GA4, and acts through SpiderPublish. Point it at a test result and ask "is this real?"

It will quite often tell you no. That is the feature.