HackerTrans
トップ新着トレンドコメント過去質問紹介求人

tfederman

no profile record

投稿

Ask HN: Is archive.org a good place for structured data?

3 ポイント·投稿者 tfederman·9 か月前·2 コメント

Ask HN: Any recommendations for help with (engineering) career presentation?

2 ポイント·投稿者 tfederman·昨年·1 コメント

Show HN: Fountain of RSS – 30k active feeds and code for sourcing more

github.com
2 ポイント·投稿者 tfederman·昨年·3 コメント

コメント

tfederman
·9 か月前·議論
It's not a big data set that lends itself primarily to analysis, it's more like content. For example, a list of all US Presidents with a lot of metadata or text content fields about them collected/combined from different sources, cleaned, corrected, annotated, etc. (Pretend Wikipedia has only a subset of these fields and considers broadening them out of scope.)

As for Github, the data would still be under "my" account and I'm thinking about more of a platform that doesn't depend on one person. Maybe I would manage day to day version control in Github but I'd want to promote occasional releases to be more official and not reliant on my account.
tfederman
·昨年·議論
In the GPT-2 era I created CouldReads, a big data set of generated book titles/synopses trained on thousands of e-books. It was a fun project in the naivete of 2020 but it's less amusing now.
tfederman
·昨年·議論
A while back I wrote up a way to turn the big Wikipedia XML dump into a database. Not a generic table with articles but thousands of tables, one for each article "type". I'm not sure if this is still the best way to go about it.

https://feder001.com/exploring-wikipedia-as-a-database-part-...
tfederman
·昨年·議論
That looks like a crowdsourced project for turning arbitrary sites into RSS which is very cool, but I don't see a way to get a large RSS data set out of it. And with about 5000 sources (I think) it's not as large as what I was hoping for, but it could be a good complementary source.
tfederman
·昨年·議論
RSS reader through Bluesky custom feeds: https://github.com/tfederman/stroma-news

Bluesky API library spun off from the other project: https://github.com/tfederman/pysky

Haven't really started it yet, but a master list of RSS feeds and the code I used to source them: https://github.com/tfederman/huge-rss-list

And also a new project to fetch all links seen in the Bluesky firehose and gather metadata to build a database of sites and pages at a more granular level than the domain. For example, is account X posting video links from one YT channel or many?