My AI Reported a 9.9% Open Rate. The Real Number Was Half That.

The Build Log — entry two

I saw 9.9% on my dashboard one morning. In the local service world, that's decent. Not great, but enough to think "okay, this thing has legs." I almost tweeted about it.

Then I checked the math. The real open rate was roughly 5%. My system had been double-counting every Gmail user — which, in my small market, was most of them.

What broke

The email contains a tracking pixel. When someone opens it, their mail client loads the image and my server logs a hit. Simple. Except Gmail doesn't let your client load images directly from strangers. It proxies them through Google's own servers.

A single human opening an email once could generate multiple requests. Gmail pre-fetches. It refreshes the cache. The same person checking on their phone after their laptop triggers a second request. My server saw two, three, sometimes more hits for one person glancing at an email over coffee.

And the number wasn't obviously wrong. 9.9% is plausible. It sat right in the band where you nod and think "that's about right." A wrong number that looks right is far more dangerous than one that's obviously broken — because you'll act on it.

The fix

Riker added a deduplication window. The same recipient can only log one open per 24-hour period. If Gmail's proxy hits that URL five times, it's one open. The pixel is fine. The counting was naive.

The corrected count for that batch: 71 unique opens, not 142. 5.0%, not 9.9%.

The embarrassing part? I could have spotted this earlier if I'd looked at the raw request log instead of the pretty dashboard. The log was full of sequential hits from the same IP ranges, seconds apart. It looked like a machine doing machine things, not 142 humans waking up to read my email.

I didn't look because the dashboard said 9.9%, and 9.9% felt good.

The lesson

The easiest person to fool with a metric is the person who built the system that produces it. You want the number to be good. That desire is a filter, and it colors what you see.

Before you scale anything — before you spend more money, before you tell a client it's working — find the raw input that feeds the number you like. Ask if it could be counting the same event twice. Ask if a machine is generating the signal instead of a human. Ask if the metric measures the thing you actually care about, or just the thing that's easy to count.

Your one thing this week

Pick one number in your business that currently makes you feel good — open rate, conversion rate, hours saved, whatever — and spend fifteen minutes finding the raw data underneath it. Not the dashboard. The log, the CSV export, the raw event list.

Ask: could this be counting something twice? Could a machine be inflating it? Could it be measuring activity instead of outcome?

You don't need to fix it this week. Just look. The gap between the number you see and the truth underneath is usually visible in fifteen minutes, if you're willing to look.

---