Ask HN: Are there any apps to track grocery prices in local stores?
With tariffs kicking in and imports slowing, I want to track the local impact at my grocery stores. Does anyone have any suggestions? Would scraping websites of local grocers be sufficient? Any prior art?
19 comments
This can only be done with crowdsource data. The hard parts, besides accumulating active users, are unique product identification and reputation system to prevent fake data flooding.
Autoscraping the price tags, as the sibling suggested, would be nice. However, in real retail the codes on the tags are regularly as small as possible and sometimes do not even correspond with the one on the product (don't know how this works). Means you need to scan them really close, individually. You also can't rely on placement for augmenting missing/trimmed textual data on tags with identifying the actual item on the shelf as in many places it's consistently messed up (as if on purpose).
Autoscraping the price tags, as the sibling suggested, would be nice. However, in real retail the codes on the tags are regularly as small as possible and sometimes do not even correspond with the one on the product (don't know how this works). Means you need to scan them really close, individually. You also can't rely on placement for augmenting missing/trimmed textual data on tags with identifying the actual item on the shelf as in many places it's consistently messed up (as if on purpose).
I wanted to build this a couple of years ago. My idea for solving the user problem was as follows:
- Users get to ask whether they are getting the best prices (or not) for an item.
- They do this by scanning the label with barcode.
- If anything is unclear they are asked to confirm the item, price, store, etc...
- Once confirmed they get to see price history or prices at other nearby stores.
That way you get a new datapoint per request.
You could probably solve some of the data integrity problems with some data science.
Tools are better now than they were back then so perhaps you could vibe code this in a couple of hours?
You could probably solve some of the data integrity problems with some data science.
Tools are better now than they were back then so perhaps you could vibe code this in a couple of hours?
Instead of parsing data on things on shelves with video for the data points why not just use receipts? That correlates more closely to the price data than the consistency of what's actually on the shelves, though video of the shelves would also be useful (i.e. estimating inventory/shortage).
These days I would like to pair this with a database of the supply chain of products which identifiers were parts are sourced from etc... I could then enter some preferences like "avoid products from countries engaging in genocide" and after scanning such a product it could tell me if a product failed one of my filters.
I did some research towards this end and found some existing tools that could be useful:
- https://github.com/Forceu/barcodebuddy/ (Barcode system for Grocy)
- https://world.openfoodfacts.org/ (Open Food Facts is a food products database made by everyone, for everyone. You can use it to make better food choices, and as it is open data, anyone can re-use it for any purpose.)
We tend to think that our only power is at the voting both and there it often feels negligible. I believe that we actually hold a lot of economic power through our consumption choices, especially those of us in first world economies or with higher income. Not only could this be more influential that casting a vote every four or five years, but it actually gives you the opportunity to express a much more nuanced view rather than picking one of two (or a handful of) politicians who represent a portfolio of policies that might not be that different from each other (thus offering no real choice) or where you have to pick the lesser of two evils (thus implicitly endorsing policies you disagree with). Currently this economic power is largely squandered (except for extreme cases like boycotts), like misaligned magnetic dipoles in a ferromagnet cancelling each other out in the absence of an external magnetic field. If instead we could get all the individuals to align, consumers in the economic force case or atomic or molecular magnetic dipoles in the magnetism case, then the same collection of individuals is able to produce a powerful force field with real effects.
I did some research towards this end and found some existing tools that could be useful:
- https://github.com/Forceu/barcodebuddy/ (Barcode system for Grocy)
- https://world.openfoodfacts.org/ (Open Food Facts is a food products database made by everyone, for everyone. You can use it to make better food choices, and as it is open data, anyone can re-use it for any purpose.)
We tend to think that our only power is at the voting both and there it often feels negligible. I believe that we actually hold a lot of economic power through our consumption choices, especially those of us in first world economies or with higher income. Not only could this be more influential that casting a vote every four or five years, but it actually gives you the opportunity to express a much more nuanced view rather than picking one of two (or a handful of) politicians who represent a portfolio of policies that might not be that different from each other (thus offering no real choice) or where you have to pick the lesser of two evils (thus implicitly endorsing policies you disagree with). Currently this economic power is largely squandered (except for extreme cases like boycotts), like misaligned magnetic dipoles in a ferromagnet cancelling each other out in the absence of an external magnetic field. If instead we could get all the individuals to align, consumers in the economic force case or atomic or molecular magnetic dipoles in the magnetism case, then the same collection of individuals is able to produce a powerful force field with real effects.
I am not sure how the system for most grocery stores works now, but in the ancient past there was a central authority that set the prices for individual stores (so in Australia, the Coles or Woolworths all have individualised prices per store for items except for advertised sale items).
The computer in the store that drives the POS scanners then can print out the shelf label stickers and a poor sod goes and replaces the stickers on the shelves.
This is all a big secret of course, so asking a grocery store for their prices is going to get you nowhere.
You could try scraping the websites, but often the price for online shopping is different to what is in store.
Best bet would be to offer an incentive for people to scan their receipts into your app. I don't know what kind of incentive you could build to do that, maybe for every 5 receipts you could offer a coupon or something. Not much of an idea and a massive PITA, it would also only give prices on those things that people bought.
edit: also gets complicated for multi-item discounts or combination discounts
The computer in the store that drives the POS scanners then can print out the shelf label stickers and a poor sod goes and replaces the stickers on the shelves.
This is all a big secret of course, so asking a grocery store for their prices is going to get you nowhere.
You could try scraping the websites, but often the price for online shopping is different to what is in store.
Best bet would be to offer an incentive for people to scan their receipts into your app. I don't know what kind of incentive you could build to do that, maybe for every 5 receipts you could offer a coupon or something. Not much of an idea and a massive PITA, it would also only give prices on those things that people bought.
edit: also gets complicated for multi-item discounts or combination discounts
There are barcode scanning libraries that specialise in scanning dozens or hundreds of barcodes in a single frame.
These are typically used to mass-import your competitors prices just by walking through the shop with a camera mounted on your shoulder. Or phone in a shirt pocket with the camera facing forward.
There’s a few white-label companies that offer these apps. Many of the barcode companies offer their own version of this app. They’re not usually available on the AppStores.
The reason for this is that each competitor requires slight customisation (font, price location relative to each barcode, etc) to get the best results.
These are typically used to mass-import your competitors prices just by walking through the shop with a camera mounted on your shoulder. Or phone in a shirt pocket with the camera facing forward.
There’s a few white-label companies that offer these apps. Many of the barcode companies offer their own version of this app. They’re not usually available on the AppStores.
The reason for this is that each competitor requires slight customisation (font, price location relative to each barcode, etc) to get the best results.
I thought barcodes were effectively just "guids" that you could use for a lookup table or whatever to get pricing etc. Knowing what the product _is_ wouldn't tell you what the price is - you'd need the table.
How does it actually work?
How does it actually work?
You're right, barcodes are just unique IDs individually identifying producers and products.
However, as I wrote, the vendors of these systems provide customisation services to improve the OCR of the price relative to the barcode. Location, font, etc. Even the name can be imported, and in some cases shelf space per product, thus stock estimates, etc.
However, as I wrote, the vendors of these systems provide customisation services to improve the OCR of the price relative to the barcode. Location, font, etc. Even the name can be imported, and in some cases shelf space per product, thus stock estimates, etc.
Scraping and aggregating grocery data is tougher than you'd expect. If you're determined to build something in this space, then reach out to me. I run Project Hammer - tracking Canadian grocery prices.
I am working on an app to track grocery purchases from online receipts. My goal is to track the effects of inflation on prices, and build something like Mint, but with itemized transactions from receipts.
Safeway’s site is extremely aggressive with anti scraping. Even if you simply direct load a product url a single time you’re hit with a captcha. Try it - view any product page, and the change 1 character in the product id in the url. Instant captcha. Scrapers be damned.
You think they might be checking the clickstream/referrer header?
As for buying groceries over the past year 80% of my spending has been bulk purchases at wholesale locations. So instead of buying a couple of tomatoes I would buy them by the box at $19 each.
It seems to me the way to do this is have folks scan in a receipt. Tag with any missing information (location, date) that may be missing.
Incentivize the behavior... somehow... and hope for the best.
Incentivize the behavior... somehow... and hope for the best.
I planned to do this one. Check with the law, like talk to a lawyer or someone in the consumer ministry. Prices were protected under law in our case. It wasn't legal to crowdsource it; stores had to share it willingly. I'm not sure of the Act is still there, but we ended up pivoting around it.
You can skirt laws like Uber did and sometimes change them. But be aware of the how far you can skirt them. If you're making a political statement, be extra careful.
In our case, shops were unhappy because of the business model. They overcharged on some things and made others look cheap, but to whomever they targeted, they wanted to look cheap. When we compared a grocery list between multiple shops, it turns out most had similar totals. It would have encouraged user behavior where they buy chicken in this shop, salmon there, diapers in another shop. And because all of these were loss leaders, it would have been bad for the stores overall. Also consider the protectionist environment that brings about tariffs and the current hostile climate against "tech bros", and it's a doubly bad idea - if the law doesn't get you, the mob will.
But if they share it with you willingly, it's normally legal and less work. We had a phase when implementing GST where companies would advertise that they were not raising prices. All you have to do is verify this. You can even charge an advertising fee.
You can skirt laws like Uber did and sometimes change them. But be aware of the how far you can skirt them. If you're making a political statement, be extra careful.
In our case, shops were unhappy because of the business model. They overcharged on some things and made others look cheap, but to whomever they targeted, they wanted to look cheap. When we compared a grocery list between multiple shops, it turns out most had similar totals. It would have encouraged user behavior where they buy chicken in this shop, salmon there, diapers in another shop. And because all of these were loss leaders, it would have been bad for the stores overall. Also consider the protectionist environment that brings about tariffs and the current hostile climate against "tech bros", and it's a doubly bad idea - if the law doesn't get you, the mob will.
But if they share it with you willingly, it's normally legal and less work. We had a phase when implementing GST where companies would advertise that they were not raising prices. All you have to do is verify this. You can even charge an advertising fee.
many grocery stores here don't even publish prices in a machine readable format, or only for selected products. best of luck.
Flipp ?
Just scrape Flipp ?
Your grocery store would have significant data that it feeds its e-ink displays with. Capturing those updates could be fun if you’re looking for a project…
Your grocery store would have significant data that it feeds its e-ink displays with. Capturing those updates could be fun if you’re looking for a project…