Signups rose 30%. Revenue fell 12%.

At one company I worked with, the winning pricing variant cut the price 40% and lifted signups 30%. The team shipped it as the new price. At that company, the next quarter's ARR came in 12% below the prior one. They ran the arithmetic after the fact. At that company, a 30% lift on a 40% cut was a smaller business, and the arithmetic would have said so beforehand. They had A/B tested their way into a worse company, with confidence, against a metric they'd picked wrong on day one.

The test wasn't a pricing test. It was a conversion test in pricing clothing. The hypothesis was lower price lifts conversion, and the test answered it correctly. The useful hypothesis was lower price lifts revenue, which means multiplying the conversion lift by the price ratio and then subtracting whatever retention you lose downstream. The team measured the easy thing and shipped the answer.

This is the most common error in pricing experimentation, and the tools cause it. Optimizely, VWO, Statsig all report lift on a binary conversion event by default. Detecting lift in revenue takes custom event tracking, longer windows, and statistics most teams aren't set up to run. So the path of least resistance is to test conversion and declare victory when conversion moves. For a company whose business model is revenue rather than signups, that path books losses on a schedule.

Then it compounds. A cheaper price pulls more buyers. It also pulls a different buyer. The person who wouldn't pay the old price often has lower willingness to pay across the board: shorter lifetime, higher churn, more support load, weaker upgrade rate. The signup lift is real and the revenue behind each signup is thinner than the model assumed. The gap doesn't show up in the test window. It shows up two or three quarters later, in cohort curves nobody reopened because the decision already shipped.

Design the test against revenue per visitor instead, measured over a window long enough to capture the dominant retention behavior. For most SaaS products that's a quarter, not a week. The test runs longer, tolerates more short-term noise, and returns a conclusion that survives contact with the actual P&L.

The worst part is that the mistake is hard to walk back. Customers who joined at the low price are anchored to it. Raise the price toward the old level and the price-sensitive cohort churns on cue. The company has settled into a new equilibrium it can't cheaply leave. The signup lift that read as a win locked in a structurally smaller business, on a timeline nobody can speed up.

Test revenue. The signups are incidental. The math is unforgiving and almost nobody runs it before they ship.