Ask HN: Claude Skills and Bs4 for intelligent scraping

1 ポイント投稿者 poch4nga9 か月前2 コメント

while building https://polyscores.xyz (soccer betting for polymarket) as a side project, I was scraping a number of data sources that feed the front end and my decision making during live games. but the problem that I have is that there is so many different data models and parts of the source websites that I am scraping - I don't want to have a dedicated solution for all them. I want something that can do all at once with a simple prompt

While doing all this scraping without any agent help, seeing claude skills' (I am an anthropic power user) ability to run scripts - I want to build a generic scraper that can do a number of things:

1) create data models based on the data it sees

2) create target data stores, like a postgres table based on a model from 1

3) scrape the data using claude agent sdk + playwright mcp

4) dump the data into the storages it creates

this can be utilised for so many different problems. but looking at the scope of the problem, this might be its own company :D

my questions are more of a market discovery:

- if people know of anyone building something similar?

- is someone from the audience would find it useful?

I think I will build a small version for myself without 1 and 2 but would love to know what people think

3 コメント

DatMule9 か月前

careful anthropic, well all top, but I have direct copyright violations oct22/'25 documented when they settled copyright violations for $1.5B usd on oct09/'25. Wouldn't say it if I didn't have screenshot and recording proof. just a heads up is all tho. Claude does Code well. usually.

poch4nga9 か月前

sorry I don't get this at all

ceaseless_dev8 か月前