I find this infra perspective so fascinating, as a career-long product platform engineer and solidly staff+. I don't argue that sean's writing (which is amazing) is a little overly mercenary in framing, but I'll pick a few choice sections I don't agree with here too. tldr I think the "product cares about speed; infra cares about leverage" staff engineer archetype is a false dichotomy we shouldn't encourage.
"In the product environments Sean describes, where goals pivot quarterly and features are often experimental, speed is the ultimate currency. You need to ship, iterate, and often move on before the market shifts." -- I disagree that speed is the ultimate currency! A great product org also respects long term leverage, it's just _always_ harder to argue for. But it's best to build a strong portfolio of going fast (where needed), going slow (where high leverage), and if everyone agrees your "going slow" led to huge returns you get the best of all worlds. Frankly it's a sign of a relatively junior product engineer if they are myopically focused on speed at the cost of all else.
"But the more powerful return is systemic innovation. If you rotate teams every year, you are limited to solving acute bugs that are visible right now. Some problems, however, only reveal their shape over long horizons." -- Extremely true, and this is _equally or more true_ in product domains. My most valuable contributions have come from sitting in a product area long enough to generalize 5 micro optimizations into the macro engineering leverage we needed to drive an order of magnitude more value from the same engineering input.
"For some engineers, navigating this [high visibility driven] chaos is a thrill. For those of us who care about system stability, it feels like a trap." -- my protip for prospective staff engineers is to _never_ say you only care about [speed, stability]. In most cases you must care about both, and it's worth advertising yourself as such. If you self select out of companies that only care about [pure stability/pure ship velocity] there should be a valuable balance to strike and staff engineers are in a unique place to enshrine that balance in engineered systems.
"In a product organization, you often need to impress your manager’s manager. In an infrastructure organization, you need to impress your customers’ managers." -- surely we can agree impressing customer stakeholders is even more important in (healthy) product orgs :) But it's a curious claim!
Great article, wonderful to hear more nuanced and deep discussion of the practice of extremely senior IC engineering. Kudos!
Finally, I cannot wait! This sure sounds like it should remove the terrifying guesswork of wondering what the actual quota values are, detecting when you're approaching a limit, and starting to automate or at least alleviate the pain of changing them.
Judging just by the blog post, this might well be the best quality of life improvement of 2019.
In the console, I see at least some services (I care about Step Functions) list as "showing default quotas only". I wonder if this means there will be a long tail of adoption before individual services show actual current values?
Strong agree -- the distribution was a solid deal when it happened and didn't require a paywall. I just gave up on them (http://www.locallyoptimal.com/blog/2019/03/24/publish-indepe...) now that they're forcing you to choose no-paywall or medium distribution. If I'm not wanting a paywall I may as well use my own site and be totally free.
It's a shame, I found the editing interface pretty convenient (image embedding in particular) and conceptually the Medium-run publications make a ton of sense.
Inherited a codebase that ran nearly all revenue-critical operations, and operated at its core on some of the most metaclassy/dynamic tools Python has at its disposal.
Luckily I work in a tech company where the fact that nobody knew this code and it took years to effectively ramp up on was argument enough that it should go away.
Actual removal was a much messier story. It had tangled deeply into adjacent systems so you couldn't "just replace it". We are in the later phases of something like the Strangler Pattern (https://docs.microsoft.com/en-us/azure/architecture/patterns...) where we built higher-level interfaces over the top and gradually re-implemented the underlying functionality without using any of these custom frameworks.
That said, it's a long term project that is easy to lose steam on. It's been very important to regularly revisit our goals and how we're attacking them...AWS has released services that fundamentally changed our approach (for the better) in the years since this effort started and we've probably cut off at least a year from the overall effort by adopting those instead of continuing on the original course.
I wrote up some of these ideas about accomplishing big projects that span years at https://medium.com/@scott_triglia/ask-a-tech-lead-i-have-to-.... The parts about regularly re-evaluating the next steps in your course of action were directly inspired by this project I just described.
Great to see open source tooling around Step Functions. Personally I'm still holding out for someone taking the States language spec (https://states-language.net/spec.html) and turning out higher-level tooling for writing/inspecting/manipulating state machines.
Agree w/ bpicolo (hi Ben!) that the tooling still can come a long way for those who can't use Lambdas directly. This code is interesting, but it misses my major use case for Step Functions -- running a workflow across several services transparently.
If I had to guess the future, I suspect that tools like Step Functions will become more and more crucial as we realize that in a FaaS world the job of coordination is complex and essential.
What makes you say Step Functions isn't ready for prime time? We've been using it (and SWF which it is based off) for about a year at decent scale and been generally very happy with it.
So I ask this mostly out of horrified curiosity -- is there any negative press or revelation that could actually sink Uber at this point? What would it take for the company to fail?
We've seen:
* Systematic flouting of laws in public
* Even further hidden avoiding of laws
* Sexism complaints across the board
* What appears to be a pervasive toxic culture, inspired (if not explicitly encouraged) straight from the CEO's behavior
Very much looking forward to an upcoming project with Step Functions at $DAYJOB. I suspect there are many use cases for turning complicated intersections of batch jobs and blobs of code into discrete, independent tasks like SFN requires.
Particularly interested in composing higher and lower level state machines for better separation of concerns, but haven't yet come up with a compelling use case :)
Is it possible to accept an activity task with no timeout, store the state in a DB, then when your event occurs, load state from the DB and send the task_success signal? Not convenient, but seems feasible..
As someone currently at the "prove it's a good idea" stage of moving production from SWF to SFN, I'm happy to hear it's been such a good experience in practice!
I think I'm mostly excited to stop locally representing/maintaining/enforcing the state machine -- it's been a huge source of toil for very little benefit. Hoping the cheapness of a new state machine will encourage splitting them when it makes sense instead of inflating/complicating an existing workflow...
How are the latency gains in practice? Specifically the activity -> decision -> activity interstitial latency in SFN vs the old SWF style.
By far the most frustrating part of blogging for me (though certainly not the only one) is the assumption that no one will ever see it. I can probably solve this by options I dislike -- giving up ownership to Medium or obnoxiously cross-posting to all social media platforms -- but the end result is I am discouraged before I even start.
Oddly I love speaking at conferences...maybe because some nonzero audience is more or less guaranteed?
"In the product environments Sean describes, where goals pivot quarterly and features are often experimental, speed is the ultimate currency. You need to ship, iterate, and often move on before the market shifts." -- I disagree that speed is the ultimate currency! A great product org also respects long term leverage, it's just _always_ harder to argue for. But it's best to build a strong portfolio of going fast (where needed), going slow (where high leverage), and if everyone agrees your "going slow" led to huge returns you get the best of all worlds. Frankly it's a sign of a relatively junior product engineer if they are myopically focused on speed at the cost of all else.
"But the more powerful return is systemic innovation. If you rotate teams every year, you are limited to solving acute bugs that are visible right now. Some problems, however, only reveal their shape over long horizons." -- Extremely true, and this is _equally or more true_ in product domains. My most valuable contributions have come from sitting in a product area long enough to generalize 5 micro optimizations into the macro engineering leverage we needed to drive an order of magnitude more value from the same engineering input.
"For some engineers, navigating this [high visibility driven] chaos is a thrill. For those of us who care about system stability, it feels like a trap." -- my protip for prospective staff engineers is to _never_ say you only care about [speed, stability]. In most cases you must care about both, and it's worth advertising yourself as such. If you self select out of companies that only care about [pure stability/pure ship velocity] there should be a valuable balance to strike and staff engineers are in a unique place to enshrine that balance in engineered systems.
"In a product organization, you often need to impress your manager’s manager. In an infrastructure organization, you need to impress your customers’ managers." -- surely we can agree impressing customer stakeholders is even more important in (healthy) product orgs :) But it's a curious claim!
Great article, wonderful to hear more nuanced and deep discussion of the practice of extremely senior IC engineering. Kudos!