Fifty seven percent of our price checks used to come back with nothing in them. Not a wrong number, not a stale one. Nothing. I noticed it first as a gap in my own charts, weeks where a crop’s line just stopped and started again somewhere else, and I went looking for why.
The answer wasn’t dramatic. Grocery websites don’t want to be scraped, so they hide prices behind session tokens, lazy-loaded scripts, and layouts that change without warning. My scrapers would ask a page for a price and the page would hand back a placeholder, a loading spinner, or a block page dressed up as a normal response. From the outside it looked like data. From the inside it was air.
What the data actually shows
Once I started logging null responses properly instead of just discarding them, the pattern was obvious. Blanks weren’t random. They clustered around specific retailers, specific times of day, and specific product categories, mostly the fresh produce pages that get redesigned more often than the pantry aisle does. Some categories were solid, giving me a clean read most days of the week. Others were patchy enough that a graph built from them would’ve been more guess than measurement.
I rebuilt the collection layer around that finding rather than pretending it wasn’t there. Retries with backoff for the pages that fail intermittently. Fallback checks against a second source when the primary one goes dark. And, importantly, I stopped filling gaps with the last known price just to make a chart look continuous. A flat line that’s actually three days of missing data is worse than an honest hole, because it tells you something moved when it didn’t. Now a missing read stays missing until I’ve actually got a real one to replace it.
The blank rate has come down a lot since then. It’s not zero, and I don’t expect it to be. Retailers keep changing their sites and I keep adapting to it. That’s the job.
What it means for your garden
Here’s why this matters if you’re not the one staring at scraper logs. Every price trend I show you, garlic creeping up before winter, tomato seedlings cheap after a warm snap, is only as good as the data underneath it. If I’d kept averaging over the gaps instead of fixing them, I’d have been quietly feeding you smoothed-over nonsense dressed up as insight, and you’d have no way to tell.
It also means my price coverage still leans on what I can actually see clearly, and right now that’s stronger for some categories than others. If you’re in Perth or Hobart, most of what feeds these trends is Brisbane and southeast Queensland shelf data, so treat the dollar figures as indicative and the direction of the move as the useful bit, wherever you’re gardening.
None of this changes what to plant this week. It just means when I do tell you garlic’s cheap or seedlings have jumped, I’ve actually checked, rather than assumed. The unglamorous part of running a data pipeline is admitting how often it was wrong before you fixed it.
