HackerTrans
TopNewTrendsCommentsPastAskShowJobs

tfederman

no profile record

Submissions

Ask HN: Is archive.org a good place for structured data?

3 points·by tfederman·9 maanden geleden·2 comments

Ask HN: Any recommendations for help with (engineering) career presentation?

2 points·by tfederman·vorig jaar·1 comments

Show HN: Fountain of RSS – 30k active feeds and code for sourcing more

github.com
2 points·by tfederman·vorig jaar·3 comments

comments

tfederman
·9 maanden geleden·discuss
It's not a big data set that lends itself primarily to analysis, it's more like content. For example, a list of all US Presidents with a lot of metadata or text content fields about them collected/combined from different sources, cleaned, corrected, annotated, etc. (Pretend Wikipedia has only a subset of these fields and considers broadening them out of scope.)

As for Github, the data would still be under "my" account and I'm thinking about more of a platform that doesn't depend on one person. Maybe I would manage day to day version control in Github but I'd want to promote occasional releases to be more official and not reliant on my account.
tfederman
·vorig jaar·discuss
In the GPT-2 era I created CouldReads, a big data set of generated book titles/synopses trained on thousands of e-books. It was a fun project in the naivete of 2020 but it's less amusing now.
tfederman
·vorig jaar·discuss
A while back I wrote up a way to turn the big Wikipedia XML dump into a database. Not a generic table with articles but thousands of tables, one for each article "type". I'm not sure if this is still the best way to go about it.

https://feder001.com/exploring-wikipedia-as-a-database-part-...
tfederman
·vorig jaar·discuss
That looks like a crowdsourced project for turning arbitrary sites into RSS which is very cool, but I don't see a way to get a large RSS data set out of it. And with about 5000 sources (I think) it's not as large as what I was hoping for, but it could be a good complementary source.
tfederman
·vorig jaar·discuss
RSS reader through Bluesky custom feeds: https://github.com/tfederman/stroma-news

Bluesky API library spun off from the other project: https://github.com/tfederman/pysky

Haven't really started it yet, but a master list of RSS feeds and the code I used to source them: https://github.com/tfederman/huge-rss-list

And also a new project to fetch all links seen in the Bluesky firehose and gather metadata to build a database of sites and pages at a more granular level than the domain. For example, is account X posting video links from one YT channel or many?