Hey HN! Just wanted to share a side project I've been hacking on. It's a news ingestion tool that's ridiculously cheap to run - we're talking less than a buck for a million articles.
Why I built it
I got fed up with the insane costs of existing news ingestion services. Plus, I wanted something I could tweak and scale easily. So I rolled up my sleeves and built this bad boy.
How it works
The secret sauce is a Lambda architecture combined with SQS queues. Here's the gist:
Lambda functions scrape news sources and dump article metadata into SQS queues.
Another set of Lambdas processes these queues, fetching full article content.
Processed articles get stored in S3, with metadata in DynamoDB.
Why it's so damn cheap
Lambdas only run when there's work to do. No idle servers burning cash.
SQS queues act as a buffer, smoothing out traffic spikes without over-provisioning.
We're leveraging AWS's generous free tier limits across multiple services.
The result? Pennies on the dollar compared to traditional setups or SaaS solutions.
Tech stack
AWS (Lambda, SQS, S3, DynamoDB)
Python (because life's too short for verbose code)
BeautifulSoup & Newspaper3k for content extraction
Open invitation
The repo is public, and I'd love for you all to check it out, star it if you like it, and maybe even contribute. Whether it's adding new features, optimizing performance, or just fixing my inevitably messy code, all PRs are welcome!
https://github.com/Charles-Gormley/IngestRSS
Let me know what you think! Has anyone else tackled this problem differently? I'm always down to learn new approaches or optimizations.
P.S. If this picks up steam, I might write a more detailed blog post about the architecture and cost breakdown. Let me know if that's something you'd be interested in!
Also this is my first post on hacker news so hopefully I'm not breaking any rules.
do you think that ~25% fees on every in app transaction ( Total is 30% but 3% is used for credit card transactions.) is maybe too much to cover these software development costs?
A counter to this would be game store sites like Steam have in 30% fees but steam doesn't have a monopoly on the games store. Apple has a monopoly on the app store.
Apple has been taking a lot of actions within the last year against the developer community:
1) Retaliations in the anti-competitive lawsuits against Apple in US (surrounding fees) :
* You must request an entitlement now from apple to link to an external payment provider in app. You must still give apple a 27% even if you use external payments.
* If Apple feels a lack of compliance with the entitlement program they have the right to look through financial books and bill for if they think their cut was missed.
2) Retaliations in the anti-competitive lawsuits against Apple in EU (surrounding fees) :
* PWAs. PWAs were starting to become a form of protest for developer to get around Apple's closed source system. PWAs don't utilize Apple's SDKs. Sure Apple may have some points surrounding the technical inefficiencies surrounding PWAs but it is suspicious that these rules have taken place during these lawsuits.
3) Beeper
* Everything going on with Beeper enforces the point that Apples wants to have a stranglehold on their monopoly
Optional 4) The insane markup of the Vision Pro relative to competitors. Making it difficult for devs to test new products.
All of these exemplify Apple's use of their monopoly on the individuals who support their ecosystem (developers). There is strong consumer demand for Apple so their is a strong incentive to develop with Apple. Being an early-stage founder myself I am finding it very difficult to convince myself that developing on Apple for proving product market fit is the way to go.
These rules are hurting the smaller developer teams who can't afford to pay these fees and can't pay for a compliance guy in the beginning to look over Apple specific polices. PWAs being canceled is just another example of Apple over-using their power to hurt the little guy.
Lambda functions scrape news sources and dump article metadata into SQS queues. Another set of Lambdas processes these queues, fetching full article content. Processed articles get stored in S3, with metadata in DynamoDB.
Why it's so damn cheap
Lambdas only run when there's work to do. No idle servers burning cash. SQS queues act as a buffer, smoothing out traffic spikes without over-provisioning. We're leveraging AWS's generous free tier limits across multiple services.
The result? Pennies on the dollar compared to traditional setups or SaaS solutions. Tech stack
AWS (Lambda, SQS, S3, DynamoDB) Python (because life's too short for verbose code) BeautifulSoup & Newspaper3k for content extraction
Open invitation The repo is public, and I'd love for you all to check it out, star it if you like it, and maybe even contribute. Whether it's adding new features, optimizing performance, or just fixing my inevitably messy code, all PRs are welcome! https://github.com/Charles-Gormley/IngestRSS
Let me know what you think! Has anyone else tackled this problem differently? I'm always down to learn new approaches or optimizations. P.S. If this picks up steam, I might write a more detailed blog post about the architecture and cost breakdown. Let me know if that's something you'd be interested in!
Also this is my first post on hacker news so hopefully I'm not breaking any rules.