The unwritten laws of engineering at Stedi(stedi.com)
stedi.com
The unwritten laws of engineering at Stedi
https://www.stedi.com/blog/the-unwritten-laws-of-engineering-at-stedi
53 comments
Is there any a priori reason to pay attention to this? Is this company known for its excellent engineering? Or are they just another bunch of opinionated nerds?
I only heard about them when they did layoffs and got rid of a lot of developers.
I interviewed there a few years ago, so they've been on my radar since.
I think they push the bounds of "serverless" for a complex data project. They did discover a bug in IAM (also documented on their blog).
But, really, it's another bunch of opinionated nerds. Isn't that most of HN?
I think they push the bounds of "serverless" for a complex data project. They did discover a bug in IAM (also documented on their blog).
But, really, it's another bunch of opinionated nerds. Isn't that most of HN?
It says it at the top of the post:
> We are hiring across multiple engineering, product, and design roles right now, so we wanted to post this publicly to give a sense of what it’s like to work here. If this resonates with you, we would love to hear from you.
> We are hiring across multiple engineering, product, and design roles right now, so we wanted to post this publicly to give a sense of what it’s like to work here. If this resonates with you, we would love to hear from you.
I appreciate that many people have reservations about hyper-focusing on product development and utilising the serverless ecosystem. I understand that it may appear working with coarser primitives is constraining and unproductive, and for unfamiliar practitioners, like anything else, there's a learning curve. Having been involved in three startups that adopted the AWS serverless mantra, I've come to argue in its favour.
Serverless architectures require designs that accommodate these constraints upfront, and there's a learning curve to finding the best primitives to adopt. It's akin to building with Lego rather than a 3D printer—you're working within predefined limits. Once you understand the optimal patterns for utilising these constraints, they become guardrails that reduce development and operational costs over time.
In my experience, the bar for requiring "always on" is higher than we might realise. Also, the choice between serverless and traditional elastic computing models isn't always binary. Excellent options like AWS Fargate serve as a practical middle ground.
Of course, this approach is only sometimes applicable. There might be a tipping point where it might make sense to consider renting servers with more traditional hosting solutions. Is optimising operational costs by investing capital in rented or physical infrastructure a strategic decision post-startup?
Serverless architectures require designs that accommodate these constraints upfront, and there's a learning curve to finding the best primitives to adopt. It's akin to building with Lego rather than a 3D printer—you're working within predefined limits. Once you understand the optimal patterns for utilising these constraints, they become guardrails that reduce development and operational costs over time.
In my experience, the bar for requiring "always on" is higher than we might realise. Also, the choice between serverless and traditional elastic computing models isn't always binary. Excellent options like AWS Fargate serve as a practical middle ground.
Of course, this approach is only sometimes applicable. There might be a tipping point where it might make sense to consider renting servers with more traditional hosting solutions. Is optimising operational costs by investing capital in rented or physical infrastructure a strategic decision post-startup?
Is there a self-hosted serverless program that works well for learning in a safe environment? Yeah, I get the contradiction - I mean, Minio is to Amazon S3 as this thing would be to Amazon Lambda. I've done cool things with serverless functions before, but never at scale.
Defeats the purpose in my view, but to answer your question check out Openfaas or explore Erlang, much if the serverless architecture is Actors forced into Cloud Native primitives. The Holy Grail in my view are actors without knowing or caring how they map to princesses.
KNative?
> Once you understand the optimal patterns
Do you have some recommendations where to start studying such patterns? I am a bit overwhelmed by all the possibilities and (cost) dangers of cloud computing.
Do you have some recommendations where to start studying such patterns? I am a bit overwhelmed by all the possibilities and (cost) dangers of cloud computing.
I've started reading this book and I rate it highly so far:
Serverless Development on AWS [0].
I agree that finding comprehensive, non-sales-oriented best practices can be challenging. One useful resource is Serverless Land from AWS [1], which offers a collection of practical examples and articles.
Mastering serverless involves more than just wiring up functions—it's about adopting Infrastructure as Code (IaC) principles. Over time, you'll develop reusable patterns and tools like Terraform modules that encapsulate best practices such as asynchronous Lambdas with EventBridge subscriptions, SQS Dead Letter Queues (DLQ), and appropriate error handling configurations.
For a deeper dive into the Serverless philosophy explicitly applied to AWS, I recommend checking out Yan Cui's blog [2]. Yan provides insightful articles on serverless architectures and patterns.
I suggest exploring the Reactive Manifesto [3] for principles and best practices in building responsive, scalable systems if you're interested in asynchronous messaging patterns.
[0] https://www.oreilly.com/library/view/serverless-development-... [1] https://serverlessland.com [2] https://theburningmonk.com [3] https://www.reactivemanifesto.org
I agree that finding comprehensive, non-sales-oriented best practices can be challenging. One useful resource is Serverless Land from AWS [1], which offers a collection of practical examples and articles.
Mastering serverless involves more than just wiring up functions—it's about adopting Infrastructure as Code (IaC) principles. Over time, you'll develop reusable patterns and tools like Terraform modules that encapsulate best practices such as asynchronous Lambdas with EventBridge subscriptions, SQS Dead Letter Queues (DLQ), and appropriate error handling configurations.
For a deeper dive into the Serverless philosophy explicitly applied to AWS, I recommend checking out Yan Cui's blog [2]. Yan provides insightful articles on serverless architectures and patterns.
I suggest exploring the Reactive Manifesto [3] for principles and best practices in building responsive, scalable systems if you're interested in asynchronous messaging patterns.
[0] https://www.oreilly.com/library/view/serverless-development-... [1] https://serverlessland.com [2] https://theburningmonk.com [3] https://www.reactivemanifesto.org
Having become an "expert" in Serverless: I hate it.
I would rather stick with EC2s at this point.
I would rather stick with EC2s at this point.
Can you explain more? I'm very interested in what you have to say about serverless.
My frenetic reply is here: https://news.ycombinator.com/item?id=40193356
They seem to be written down right there in this blog post by the organization itself.
This list feels to me like a tactical laundry list of how they want to achieve some goals at this company through technical practices. Which will and should change as technology changes.
But the “what” and the “why” seem to be mostly implied. There are mentions of quality and security, but it is really unclear. I don’t know what this company is trying to actually achieve. As an aside, lists like this are intriguing to me because they seem to value uptime over capability. I get the impression they would be perfectly happy shipping nothing for long periods.
But the “what” and the “why” seem to be mostly implied. There are mentions of quality and security, but it is really unclear. I don’t know what this company is trying to actually achieve. As an aside, lists like this are intriguing to me because they seem to value uptime over capability. I get the impression they would be perfectly happy shipping nothing for long periods.
> I get the impression they would be perfectly happy shipping nothing for long periods.
I wouldn't say _happy_, but we were certainly _willing_ to do this for long periods of time to get the right foundation built (for reference, we built for 5 years before launching). This is made explicit in the opening:
> As a result of following the framework, product development may sometimes come to a screeching halt. The work itself may be tedious and frustrating. Progress may take many multiples of the time it would take using other methods. > [...] > That's the difference between principles and preferences: principles are what you do regardless of the cost – everything else is just a preference.
As for this item:
> But the “what” and the “why” seem to be mostly implied.
This is outlined in another linked post: https://www.stedi.com/blog/excerpts-from-the-annual-letter
I wouldn't say _happy_, but we were certainly _willing_ to do this for long periods of time to get the right foundation built (for reference, we built for 5 years before launching). This is made explicit in the opening:
> As a result of following the framework, product development may sometimes come to a screeching halt. The work itself may be tedious and frustrating. Progress may take many multiples of the time it would take using other methods. > [...] > That's the difference between principles and preferences: principles are what you do regardless of the cost – everything else is just a preference.
As for this item:
> But the “what” and the “why” seem to be mostly implied.
This is outlined in another linked post: https://www.stedi.com/blog/excerpts-from-the-annual-letter
> Security is job zero.
This is really vague point. What's security? You can spend millenia securing anything without clear target. ssh with root:toor is probably not secure. What about ssh with 12-chars password? Shall we disable passwords altogether? Shall we change port? Shall we implement port knocking? Shall we investigate ssh source code, recompile it, remove anything not needed in this particular server? Shall we rewrite ssh from the scratch using Rust? Shall we build new language with better security focus?
This is really vague point. What's security? You can spend millenia securing anything without clear target. ssh with root:toor is probably not secure. What about ssh with 12-chars password? Shall we disable passwords altogether? Shall we change port? Shall we implement port knocking? Shall we investigate ssh source code, recompile it, remove anything not needed in this particular server? Shall we rewrite ssh from the scratch using Rust? Shall we build new language with better security focus?
> Use managed, serverless (defined as ‘usage-based pricing with little to no capacity planning required’) primitives wherever possible, even at the cost of functionality.
Erk. I get where they're coming from, but it feels like they're choosing lower AWS costs, but in doing so, incurring higher dev costs.
But maybe, that's a trade-off that makes sense for them. Would like to have more background on this balancing act - e.g., what "functionality" means to them.
The other concern I'd have is the risk of unanticipated spend, serverless techs can scale up really well, and your costs can also scale up hand-in-hand.
I guess it depends on how bursty their product is.
Erk. I get where they're coming from, but it feels like they're choosing lower AWS costs, but in doing so, incurring higher dev costs.
But maybe, that's a trade-off that makes sense for them. Would like to have more background on this balancing act - e.g., what "functionality" means to them.
The other concern I'd have is the risk of unanticipated spend, serverless techs can scale up really well, and your costs can also scale up hand-in-hand.
I guess it depends on how bursty their product is.
I feel you are reading it backwards. They are increasing infra costs and decreasing eng development cost - they promote using dynamodb over making your own key value store. And if that doesn't work for some reason, then use open source, maybe foundationdb or redis. And if that can't work, then build it. Building it will take longer and system stability will almost always be worse comparatively.
When I hear "dynamodb or maybe foundationdb or redis" that sounds like three very different projects.
for sure. They are all key value stores, but they are all better tailored for different purposes. The idea is to accept the limits of the building blocks you are choosing and figure out how to support your product that way. The alternative is to build the custom datastore solution which _will_ be harder than the org expects. Everyone wants a magic datastore that does everything and usually when an org tries to build their own, it fails to scale out for a multitude or reasons. I have definitely seen it work and work well -- but it always took way more effort than planned. If it is not a core competency, it is probably not worth the investment to build your own.
My money is on them getting neither lower AWS costs nor lower dev costs. They're just paying out both ends because "cloud" is a thought-terminating cliché at this point.
Yeah, that's where my "Erk" came from. But I don't know what their product looks like under the hood so can't really comment further, just that "even at the cost of functionality" got my antennae tingling.
Ditto the bit where "As a last resort, we write our own code". That feels like a lot of pain and YAML trying to cobble together other people's expensive code.
Ditto the bit where "As a last resort, we write our own code". That feels like a lot of pain and YAML trying to cobble together other people's expensive code.
"AWS is expensive" is a thought-terminating cliche as well.
AMZN ain't making its profit selling books and toothbrushes.
It's all the yummy egress and all the other markups you're paying for.
It's all the yummy egress and all the other markups you're paying for.
Who are all these people egressing gb/mo/customer? Does everyone except me work at Netflix?
I agree it's a weird thing to obsess over. I have never seen egress mentioned, let alone approach the leaderboard in terms of costs.
I'm not sure I believe it, but they could reasonably make profit on economies of scale that individual organizations wouldn't get managing their own hardware - merged hardware utilization, sharing engineering efforts, standard hardware procurement, etc.
What does this "automate updating of dependencies" mean? Is this something that happens in some ecosystem where updating things is so common that it needs to be automated? What ecosystem does it refer to? JS? Or is it more general? And what does it mean to update dependencies automatically and in real time? Does it mean for some web based products that you deploy versions with updated dependencies and immediately roll that out to production? What does it mean for other kinds of products (Maybe they don't have any)?
Nowadays there are automated tools like Dependabot (0) or RenovateBot (1) that make it simple to keep dependencies up to date. I can imagine the need originated from the JS ecosystem, but from a security standpoint it makes sense for almost any stack.
0: https://github.com/dependabot
1: https://docs.renovatebot.com/
0: https://github.com/dependabot
1: https://docs.renovatebot.com/
Creating PRs with suggested outputs is one thing. But automating it all the way to deployment seems a bit much
I think the idea is that for minor updates or patches, any potential breakages should be caught by the build pipeline or rolling deployments with automated rollback strategies (if you’re at a scale where this is feasible). Major updates will probably fail in the pipeline and require manual intervention either way.
I don’t think it makes sense for every project, but if recovery options are cheap then I don’t see anything that speaks against it.
I don’t think it makes sense for every project, but if recovery options are cheap then I don’t see anything that speaks against it.
Even in presence of an xz like attack?
Sure, why not? Are you suggesting that having a human in the loop, robotically bumping the version numbers of your dependencies would have mitigated it?
Lots of humans upgraded lots of dependencies without noticing, I doubt whoever is doing it in your org is special enough to be the one who would have caught it. And if they are, they should be working in security research, not bumping dependency versions in package.json.
Lots of humans upgraded lots of dependencies without noticing, I doubt whoever is doing it in your org is special enough to be the one who would have caught it. And if they are, they should be working in security research, not bumping dependency versions in package.json.
I didn't mean human to update your dependencies but rather a security decision, whether auotnated or human, approves that dependabot recommends.
I bump versions of maybe 2-4 dependencies per year, for a 200 man-year project. It's not exactly overwhelming. But then it's not a JS project either.
I'm not sure how updating dependencies a few times a year makes you safer from a well-hidden supply chain attack.
Also, not sure what JS has to do with the xz attack.
The argument I was responding to is that automating your dependency updates somehow makes you more vulnerable to a supply chain attack.
I could see an argument that waiting X days from a dependency release to when you pull it in gives you a little time for other people to find issues. But that's orthogonal to whether you update dependencies automatically or manually, or whether you do it once/year or every day.
Also, not sure what JS has to do with the xz attack.
The argument I was responding to is that automating your dependency updates somehow makes you more vulnerable to a supply chain attack.
I could see an argument that waiting X days from a dependency release to when you pull it in gives you a little time for other people to find issues. But that's orthogonal to whether you update dependencies automatically or manually, or whether you do it once/year or every day.
What I meant was that for maybe 3-4 packages total in a year, automation doesn't matter much. It's not a big cost. Risk wise you can be pretty conservative if you are in a high-risk scenario.
But usually I find that bumping dependencies is a detective work because unlike JS where you can depend on multiple versions of the same package, for .NET you can't. So if you update package A, there is a risk that you also update a transitive dependency C, which package B depends on but on the previous version. So even for what looks like trivial updates it's often a chore. Which is why I'm happy to have just a handful of dependencies.
But usually I find that bumping dependencies is a detective work because unlike JS where you can depend on multiple versions of the same package, for .NET you can't. So if you update package A, there is a risk that you also update a transitive dependency C, which package B depends on but on the previous version. So even for what looks like trivial updates it's often a chore. Which is why I'm happy to have just a handful of dependencies.
I'm not familiar with stedi, but the healthiest company I worked in ticked all these same boxes and points. These are all solid rules/guides/laws.
I was the CISO at a healthcare company for several years and still work in security engineering. "There is nothing more important than securing our systems" sounds good but falls over when you look closely.
Of course you should design and implement secure systems. You owe that to your customers. Still, that's meaningless until you decide what "secure" means. How valuable is your data to your customers and to potential attackers? Who's likely to target you? What level of risk are you willing to accept?
And make no mistake: you do accept a certain level of risk. If your systems are valuable enough, you're going to get the attention of enemy nations who will attack you. How much are you willing to invest to protect against North Korea or Russia? Short of financing a formal proof of every bit of software you run on, from web frontend down to microcode, you're trusting that someone else did their job correctly. And will you run that unprecedentedly secure code on a server farm inside Fort Knox?
Job one in security is deciding how much you need to invest to reach the level of security you need to achieve. If you're storing healthcare data, that's going to be a whole lot of time and money, and even then good luck if a malicious nation wants in. If you're storing grocery store shopping lists for a thousand people, and you're investing $10,000,000 to secure it, please call me because I want in on that action.
Looks like Stedi's in a field where security matters an awful lot and it's great to hear that they take it seriously. However, I'd love to hear what that actually means in practice.
Of course you should design and implement secure systems. You owe that to your customers. Still, that's meaningless until you decide what "secure" means. How valuable is your data to your customers and to potential attackers? Who's likely to target you? What level of risk are you willing to accept?
And make no mistake: you do accept a certain level of risk. If your systems are valuable enough, you're going to get the attention of enemy nations who will attack you. How much are you willing to invest to protect against North Korea or Russia? Short of financing a formal proof of every bit of software you run on, from web frontend down to microcode, you're trusting that someone else did their job correctly. And will you run that unprecedentedly secure code on a server farm inside Fort Knox?
Job one in security is deciding how much you need to invest to reach the level of security you need to achieve. If you're storing healthcare data, that's going to be a whole lot of time and money, and even then good luck if a malicious nation wants in. If you're storing grocery store shopping lists for a thousand people, and you're investing $10,000,000 to secure it, please call me because I want in on that action.
Looks like Stedi's in a field where security matters an awful lot and it's great to hear that they take it seriously. However, I'd love to hear what that actually means in practice.
[deleted]
I love the product. I love the service. HOWEVER, I’ve been doing this for a while to know that at some point, Stedi will bought…Stedi will be sold (notice my ordering of bought/sold) and the product and service will become another complicated and expensive mess.
I'll take the other side of that bet.
Skimmed through it and didn't really see any focus on products or users. Red flag IMHO.
Security is #0. Okay, security against what?
Security is #0. Okay, security against what?
Why is Stedi and Techcrunch article glossing over autoscaling EC2 and Elastic Beanstalk, which also scale on demand?
Can't speak to EB, but for EC2:
Not the new hotness? Longer startup times, which can hurt if you scale to zero? Requires building a golden image which is more overhead plus you are responsible for OS bugs?
Not the new hotness? Longer startup times, which can hurt if you scale to zero? Requires building a golden image which is more overhead plus you are responsible for OS bugs?
How is it unwritten when we are just reading written words on these laws?