I have been working on https://www.accessmrf.com - a catalog of all American health care prices. Since there's way too much data, I am building a taxonomy and deduplicating it extensively first, with the idea that I can shrink it by ~4 orders of magnitude and make the problem tractable. I have written a bit about how I do it here: https://www.felixhaba.com/writing/simplifying-healthcare-pri...
My goal this week has been to index all insurers that publish machine readable files. For now, my project helps researchers and founders in the space access the data via API, it is not consumer-grade yet. But if you are interested in this data I would love to brainstorm ideas or share my knowledge. Shoot me a message.
I am working on https://www.accessmrf.com a catalog for all health care prices published by insurers under the Transparency in Coverage rule.
I recently wrote a blog post using Min Hashing to estimate that at least 90% of the 1.17 petabyte dataset are duplicates and I keep investigating new ways of making this dataset manageable. https://www.felixhaba.com/writing/simplifying-healthcare-pri...
I've been building out https://www.accessmrf.com/sources to index Transparency in Coverage files across insurers in a single place. Most recently I have been looking at the overlap across files using Min Hashing and found that there is a lot of duplicate data. Currently, I'm figuring out a way of making this dataset more digestible by identifying and eliminating the 90+% duplicate and ghost data.
I'm building a catalog for health care price transparency data that aggregates the rates published by all insurers, to put everything in one place and make it easier for developers / researchers to access this data. https://www.accessmrf.com/
I'm building a catalog for health care price transparency data that aggregates the rates published by all insurers, to put everything in one place and make it easier for developers / researchers to access this data. https://www.accessmrf.com/
Great job, I intend to reproduce this on a similar dataset I've been collecting!
I will say, it would be great to see the color labeling done on domain url alone, to see how much of the topography of the map is driven simply by the different formatting characteristics of the websites you're gathering data from.
My goal this week has been to index all insurers that publish machine readable files. For now, my project helps researchers and founders in the space access the data via API, it is not consumer-grade yet. But if you are interested in this data I would love to brainstorm ideas or share my knowledge. Shoot me a message.