Tobias Lutke still writes code for Shopify(changelog.com)
changelog.com
Tobias Lutke still writes code for Shopify
https://changelog.com/podcast/416#transcript-45
78 comments
> The post is long gone, but it was entitled "The Secret to memcached"
The Internet (Archive) remembers:
* https://web.archive.org/web/20120125060458/http://blog.leets...
The Internet (Archive) remembers:
* https://web.archive.org/web/20120125060458/http://blog.leets...
This technique is the way Rails Fragment Caching works, and enables Russian Doll caching
Those terms should help you find more writing about this technique that I always forget isn’t obvious outside the Rails ecosystem
Those terms should help you find more writing about this technique that I always forget isn’t obvious outside the Rails ecosystem
DHH helped to popularise it with this post: https://signalvnoise.com/posts/3113-how-key-based-cache-expi...
It's been hugely influential to the way I think about scaleable pages. It's a similar idea to the cache-busting techniques well-known with URLs (https://example.com?style.css=1234 etc), but applied to page fragments recursively, where a change in any component also "revs"/"bumps" its parent components.
It's been hugely influential to the way I think about scaleable pages. It's a similar idea to the cache-busting techniques well-known with URLs (https://example.com?style.css=1234 etc), but applied to page fragments recursively, where a change in any component also "revs"/"bumps" its parent components.
I disagree completely.
Once you're in a management role, let alone a CEO role, your job not to write software. You defer defer implementation and architecture completely to your team. Your job is to hire and build that team and set high-level company direction.
The more power you have over your co-implementors' careers -- and the more junior they are -- the less comfortable they are standing up to your architectural or technical decisions. This stifling of collaboration can cause you to make poor decisions you wouldn't otherwise, and creates an uncomfortable dynamic.
This is why I don't believe in TLM ("tech lead manager") roles, also, fwiw.
Wanna code? Do it on your own time, Tobi! :)
Once you're in a management role, let alone a CEO role, your job not to write software. You defer defer implementation and architecture completely to your team. Your job is to hire and build that team and set high-level company direction.
The more power you have over your co-implementors' careers -- and the more junior they are -- the less comfortable they are standing up to your architectural or technical decisions. This stifling of collaboration can cause you to make poor decisions you wouldn't otherwise, and creates an uncomfortable dynamic.
This is why I don't believe in TLM ("tech lead manager") roles, also, fwiw.
Wanna code? Do it on your own time, Tobi! :)
You got the point provided that you are concerned about code quality. This might be an area where a CTO lacks skills over time compared to his developers.
Becoming a CTO myself within an 80 developers area, I found it invaluable to code infrequently still. It is quite simple: better decisions, better products.
To me, non-coding CTOs or most CEOs that do not understand Computer Science, it seems that they try to learn a foreign language by sending in their assistant to attend classes.
I mean this quite literally. My team can communicate to me in other terms and a different language than they would have in a simplified ELI5/KPI driven manner. And vice versa, I challenge them to think beyond day to day business.
It is not a lack of trust. It is about better understanding and better helping your team focus on the essential parts. And vice versa.
Becoming a CTO myself within an 80 developers area, I found it invaluable to code infrequently still. It is quite simple: better decisions, better products.
To me, non-coding CTOs or most CEOs that do not understand Computer Science, it seems that they try to learn a foreign language by sending in their assistant to attend classes.
I mean this quite literally. My team can communicate to me in other terms and a different language than they would have in a simplified ELI5/KPI driven manner. And vice versa, I challenge them to think beyond day to day business.
It is not a lack of trust. It is about better understanding and better helping your team focus on the essential parts. And vice versa.
Generally true, but not always true. Shopify seem to be doing just fine with this model. If things change, maybe this model will also change.
I’ve only once worked under a manager who also coded, and I had extremely mixed feelings. On the one hand, he was a brilliant engineer who I learned a ton from, and who set a really solid technical direction for the team.
On the other hand, I was personally unhappy a lot of the time. On one occasion he pretty much tore apart one of my designs and I felt fairly humiliated (with reason, as a few other senior engineers confirmed in private conversations).
Did my design deserve to die horribly? I don’t know. The PR had already been accepted, but maybe the accepter was wrong. In any case it felt really crappy to be criticized so heavily by someone who in theory supports me.
On the other hand, I was personally unhappy a lot of the time. On one occasion he pretty much tore apart one of my designs and I felt fairly humiliated (with reason, as a few other senior engineers confirmed in private conversations).
Did my design deserve to die horribly? I don’t know. The PR had already been accepted, but maybe the accepter was wrong. In any case it felt really crappy to be criticized so heavily by someone who in theory supports me.
You know, it's easy to be an asshole when you're brilliant. But that's why soft skills are important, especially in leadership roles.
I tried my best to be a supporting lead, everywhere i was in that role. I believe in leading by example, but i recognize the necessity to _let_ people do things themselves and not just hog power because "i know best". Let people implement imperfect designs. They'll own it, learn, and fix them in time. Your job is to lead, to give direction and not let total crap be put into prod, not keep people under your "benevolent dictator" heel.
I tried my best to be a supporting lead, everywhere i was in that role. I believe in leading by example, but i recognize the necessity to _let_ people do things themselves and not just hog power because "i know best". Let people implement imperfect designs. They'll own it, learn, and fix them in time. Your job is to lead, to give direction and not let total crap be put into prod, not keep people under your "benevolent dictator" heel.
I tend to disagree with the "live and let die" approach.
Delegating responsibilities isn’t abandoning responsibilities and when deviation happens, getting hands dirty is good stewardship to me.
The main factor to me is the team size and subsequently the organization of said responsibilities delegation. My personal context sweet spot is around a 50 persons team.
The main factor to me is the team size and subsequently the organization of said responsibilities delegation. My personal context sweet spot is around a 50 persons team.
There's also good and bad ways to give feedback. It's important to find ways to give honest feedback that isn't crushing your reports's confidence. It's more of a question of when to step in with an important critique that whether to at all. There are some pieces of code and systems that aren't critical enough or won't be stressed by a junior learning through a bad design decision. And critical system where you absolutely have to step in and make sure everything is in the best shape possible.
50 persons is a small organization, not a team!!
I agree about responsibilities part, however, if you don't let your people grow by simply vetoing the ideas, they won't grow. Discussing and openness to your subordinates' ideas is not a bad thing in my opinion. Letting people make proof-of-concepts and fail is not abandonment of responsibilities. Letting people implement their bad ideas in production is.
I agree about responsibilities part, however, if you don't let your people grow by simply vetoing the ideas, they won't grow. Discussing and openness to your subordinates' ideas is not a bad thing in my opinion. Letting people make proof-of-concepts and fail is not abandonment of responsibilities. Letting people implement their bad ideas in production is.
> 50 persons is a small organization, not a team!!
Well, I refer to the whole company as a team. This may be my SME bias :)
Well, I refer to the whole company as a team. This may be my SME bias :)
That's a sign of a good CxO, IMHO - thinking of the whole org as a team. Respect :)
As CEO or CTO, it;'s crucial you understand your business and if you're in a tech business, you better understand the tech to an appropriate level too. Coding can be a part of that
I believe in dog-fooding, using your own products. That holds for programmers and UI designer, who should use the software they build, but also for architects and CTOs: they should experience the effect of their decisions, and taking part in the design and coding can help.
[deleted]
He’s great. I used his Money (for managing money values as cents in the db), Liquid and DelayedJob in a bunch of projects.
Now I run my own Shopify store, selling my software.
Now I run my own Shopify store, selling my software.
Thanks for sharing this. I've used this (I call it a cache buster) for static assets in a CDN, but never considered it for other things. I'd love to read the article if it can be dug up. It's not clear how that unique ID is managed and am curious to learn more.
You can often go further than that with some database trickery.
We once needed to cache a list of objects that was unacceptably slow serialize the JSON for. We needed to cache them in memcache, but also needed reliable cache invalidation without having to instrument every ingestion point with the logic. Here's what we ended up doing.
1. Every record gets an int field titled "index" or similar.
2. Every time a record is inserted or updated, index is changed to the next value from a sequence
3. The cache key for memcache includes the number of records and the maximum index from all the records returned (including any intermediate objects if joins are involved)
4. Infinite TTL for all cache keys, but the entire cache is LRU guaranteeing that stale records are eventually removed.
This setup works because any conceivable operation done would change the cache key. If new records are inserted or deleted, count and index change. If a record is updated, then the max index changes. Even deleting and adding a new record will change the index, meaning that no matter what happens you'll get a new key and the system will continue to cache appropriately.
We once needed to cache a list of objects that was unacceptably slow serialize the JSON for. We needed to cache them in memcache, but also needed reliable cache invalidation without having to instrument every ingestion point with the logic. Here's what we ended up doing.
1. Every record gets an int field titled "index" or similar.
2. Every time a record is inserted or updated, index is changed to the next value from a sequence
3. The cache key for memcache includes the number of records and the maximum index from all the records returned (including any intermediate objects if joins are involved)
4. Infinite TTL for all cache keys, but the entire cache is LRU guaranteeing that stale records are eventually removed.
This setup works because any conceivable operation done would change the cache key. If new records are inserted or deleted, count and index change. If a record is updated, then the max index changes. Even deleting and adding a new record will change the index, meaning that no matter what happens you'll get a new key and the system will continue to cache appropriately.
From inside of Shopify I’ve heard about a lot of shit stories. For their employees, it may be less ideal to have a CEO messing around with code.
From the outside though, the company does extremely well.
Nonetheless, CEOs are generally not supposed to „make“ but to manage. There has to be a tradeoff somehow.
From the outside though, the company does extremely well.
Nonetheless, CEOs are generally not supposed to „make“ but to manage. There has to be a tradeoff somehow.
We call them cache keys. We use a hash of entirely arbitrary data you want a cache item to invalidate on when that data changes.
For example, a cached snippet of HTML for a related products section should invalidate if the product IDs change, so you hash the Item ID with the product IDs and bam, you got a cache key that will let you know when you need to reprocess that snippet.
Put that bad boy as a child of another snippet and you can cache the whole lot. It's cache keys all the way down!
Caveat: not worth the engineering effort for simple systems, just use an FPC and call it a day.
For example, a cached snippet of HTML for a related products section should invalidate if the product IDs change, so you hash the Item ID with the product IDs and bam, you got a cache key that will let you know when you need to reprocess that snippet.
Put that bad boy as a child of another snippet and you can cache the whole lot. It's cache keys all the way down!
Caveat: not worth the engineering effort for simple systems, just use an FPC and call it a day.
I think he was an active contributor to RoR in those days
>> I can't think of a better way to actually maintain a clear understanding of what's going on at a place like Shopify.
Talking to customers would be one better way to maintain said clear understanding.
Running your own Shopify store (which Tobias also does) could potentially be another.
Talking to customers would be one better way to maintain said clear understanding.
Running your own Shopify store (which Tobias also does) could potentially be another.
There are numerous comments that essentially are saying "he shouldn't be doing this." That's shortsighted. You don't have insights into how this is working in practice for them. He, together with the other people at the company, have built an offering that is serving and satisfying millions of people. If there are problems with it, they are his problems to deal with. Why are you so hot and bothered about it? Focus on building your own cool shit. Be productive.
For folks who don't know, Tobias Lutke is a CEO of Shopify. https://en.wikipedia.org/wiki/Tobias_L%C3%BCtke
BTW, I am not surprised. A technical, programmer, CEO can always write code. Nothing wrong with that. It's just like an exercise. It is amazing that is able to do that.
BTW, I am not surprised. A technical, programmer, CEO can always write code. Nothing wrong with that. It's just like an exercise. It is amazing that is able to do that.
He’s also a self-proclaimed StarCraft 2 fan. He’s donated cash to tournament prize pools and recently hired beloved former pro TLO of Team Liquid to Shopify.
https://esportsobserver.com/dario-wunsch-joins-shopify/
He's also a lurker on news.yc: https://news.ycombinator.com/threads?id=xal
How is this a good thing?
All it shows is complete lack of trust and complete desire for control.
Shopify is an awful company to work for unless you’re a junior dev who likes nerding out and refactoring code over and over again.
Source: worked there
All it shows is complete lack of trust and complete desire for control.
Shopify is an awful company to work for unless you’re a junior dev who likes nerding out and refactoring code over and over again.
Source: worked there
While you were downvoted for being negative about this and your style of message, I think there is a kernel of truth, and I don't think we should be so eager to dismiss the experiences of people who have actually worked at a place, versus what is essentially a form of PR for the company.
I suspect the problem is the lack of specificity: it's hard to judge from the comment whether there are likely to be legitimate grievances or not.
Indeed, reading their post history shows this, however:
https://news.ycombinator.com/item?id=24878130
https://news.ycombinator.com/item?id=24878130
Sorta fits inline with the fact that the CEO still spends time writing code (half-joke)
Interesting perspective, I have talked to two very seniors engineers there who love the place(one worked there 7 years, other I am not sure). One of the things they like(and I would agree) is that Shopify does rewrite old stuff when they move to something new, so there are no leftover unmaintainable untouchable .
BTW what it shows is that, he loves to code. He already built an incredibly successful company. Whatever is that he does works. I love writing code too been doing it since I was 14. One of the reason I do not want to move to Engineering Management is the expectation of abandoning it.
BTW what it shows is that, he loves to code. He already built an incredibly successful company. Whatever is that he does works. I love writing code too been doing it since I was 14. One of the reason I do not want to move to Engineering Management is the expectation of abandoning it.
I’ve never met him, but having read his blog (before he took it down around the time Shopify IPO’ed) and his work in libraries like Liquid, Rails, and others, he’s an engineer/programmer at heart. Nice to see he’s able to hold onto some of that even after the journey his company has been through these 14-15 years.
I initially had that "oh no, CEOs shouldn't be coding" response too. For anyone who didn't read the transcribed segment of a podcast linked:
This sounds entirely like a founder who still solves problems with code, but then stands back and lets his company form a team around it if it's valuable.
No inside knowledge or anything, this just doesn't sound...bad.
> we started to see performance slowly degrading in terms
> of response times on the server, and eventually we kind
> of had to do something about it [...] Interesting story -
> the initial commits for that applications were Tobi
> himself
This sounds entirely like a founder who still solves problems with code, but then stands back and lets his company form a team around it if it's valuable.
No inside knowledge or anything, this just doesn't sound...bad.
Playing the devil’s advocate here: he surely doesn’t need to do this. I assume there are enough great devs at Shopify. So why does he do it? Because it’s fun? Because he can relax? Maybe his time as CEO would be better spent not coding?
I worked at a SaaS company where the founder CEO had a sales background. He'd hop into the inside sales (read: smile and dial) team desk once a quarter or so and practice pitching cold-to-warm-ish leads over the phone.
It was super motivating for the reps, mostly junior in that role, to see their CEO doing it and remaining empathetic to what they're facing every day.
Not sure if that's like... the best reason in the world when he could be doing other more important things, but it was definitely a morale booster for the company to see him in the pit once in a while.
It was super motivating for the reps, mostly junior in that role, to see their CEO doing it and remaining empathetic to what they're facing every day.
Not sure if that's like... the best reason in the world when he could be doing other more important things, but it was definitely a morale booster for the company to see him in the pit once in a while.
Recently I've heard this episode where the Shopify developer casually mentions the fact that their CEO Tobin Lutke has made some commits to their code. In my opinion they mention this in order to show that Shopify is not your ordinary top-down corporate environment.
It's also good PR. That's why it's also mentioned by FB employees that Mark Zuckerberg has also made some commits to the their codebase.
BTW I
It's also good PR. That's why it's also mentioned by FB employees that Mark Zuckerberg has also made some commits to the their codebase.
BTW I
What better way to keep tabs on the state of the engineering fundamentals than actually try to write code?
post title is SUPER misleading. Here's the only paragraph in the transcript that mentions Tobias:
> Interesting story - the initial commits for that applications were Tobi himself, who took it upon himself to start something, and as a prototype gets something up and running and make it as lean as possible to get started. And then eventually, that became a team, and we picked it up, and that became the project that we’re working on.
The Founder-CEO is in the commit history. Duh. So what? That doesn't mean he's writing code right now.
> Interesting story - the initial commits for that applications were Tobi himself, who took it upon himself to start something, and as a prototype gets something up and running and make it as lean as possible to get started. And then eventually, that became a team, and we picked it up, and that became the project that we’re working on.
The Founder-CEO is in the commit history. Duh. So what? That doesn't mean he's writing code right now.
Mark Zuckerberg once said that he does not write code anymore because they have a rule at Facebook, that says if you write the code, you have to maintain the code. I think that's a rational choice for a company of that size. Thus I'm not convinced Lutke's ongoing coding is that clever.
I don’t know about Shopify specifically, but in general I think this is a red flag. Are you going to be able to leave honest feedback on your CEO’s PR? can your CEO run a company and also continue to keep up with best practices for your language/tooling? Will your CEO go along with code style consensus, when he/she hasn’t had time to participate in any of the conversations?
Check out "No Rules Rules" by Reed Hastings and Erin Meyer. If you can't leave (constructive) feedback/criticism to your CEO, you're working at a sick company.
Yes, a high school intern did that actually - https://news.ycombinator.com/item?id=20490964
Have worked in a couple of jobs where the CEO or CTO still raised PRs, and they were really treated no differently. There was definitely a lot of code style changes requested, because as you say that knowledge might be out-of-date.
I suspect it might be cultural - I've definitely got the impression that in some countries CEOs see themselves as "gods" who should not be questioned by the minions.
I suspect it might be cultural - I've definitely got the impression that in some countries CEOs see themselves as "gods" who should not be questioned by the minions.
[deleted]
Guy Raz interviewed him on How I Built This last year, great listen
https://www.npr.org/2019/08/02/747660923/shopify-tobias-l-tk...
https://www.npr.org/2019/08/02/747660923/shopify-tobias-l-tk...
He also gave a talk a long time ago I really liked. Seems the actual video is gone now. :(
https://www.infoq.com/presentations/lutke-rockstar-memcachin...
https://www.infoq.com/presentations/lutke-rockstar-memcachin...
This is such an interesting perspective. 99% of engineering managers dont or cannot write software any more usually as a rule.
"shouldn't" is also debatable to include, because managing itself is a full-time job
He's not exactly an engineering manager, but he's definitely part of that 1% who don't meet your rule, so it actually fits within the standard perspective.
Well, it doesn't matter if they can or cannot. It's not their job to write code. Unless it's a very small team.
Where do you get the 99% number from?
Thin air. This is the internet after all.
For a moment I confused this gentleman's name with that of the character in Arrested Development (the "never nude" Tobias Fünke?)
But why or how on Earth would a person like Tobias Lutke find the time to do this? Are there other people who run multibillion dollar public companies who poke around in the codebase?
But why or how on Earth would a person like Tobias Lutke find the time to do this? Are there other people who run multibillion dollar public companies who poke around in the codebase?
Bill Gates was famously coding as a public company CEO and revising a lot of code he was unsatisfied with. I recall rumours he was politely urged to stop it.
"Ultimately he had to force himself to stop revising and perfecting his peers’ work. “I had to say to myself, ‘Ok, we’re going to ship code that I didn’t edit,’” he said. “And that was hard for me, but I kinda got over that.”" https://www.cnbc.com/2018/05/10/bill-gates-quit-this-bad-hab...
One source suggests he last created production code in 1989, three years after MS went public (https://www.quora.com/When-did-Bill-Gates-last-write-code-fo....)
"Ultimately he had to force himself to stop revising and perfecting his peers’ work. “I had to say to myself, ‘Ok, we’re going to ship code that I didn’t edit,’” he said. “And that was hard for me, but I kinda got over that.”" https://www.cnbc.com/2018/05/10/bill-gates-quit-this-bad-hab...
One source suggests he last created production code in 1989, three years after MS went public (https://www.quora.com/When-did-Bill-Gates-last-write-code-fo....)
To be honest, creating an Operating System, the ecosystem surrounding it, etc. is technologically much more complicated than a lot of the stuff that web-based companies are doing nowadays. Besides, Microsoft was definitely a much more technical company that sold a purely technical product, so it is not entirely inconceivable that founders were deeply technical in a way that today's founders aren't.
I also imagine that there was much less 'adult supervision' of Microsoft. They probably did not take (any?) VC money, did they?
The scene today is very different, so I wonder how such a thing happens in modern-day startups. As an investor, I would be alarmed at such egregious misallocation of valuable founder/C-suite time and brainpower.
EDIT: I have read many an anecdote (from Joel Spolsky, for example), wherein Bill Gates was said to have pored over the minutest specification in documents spanning hundreds of pages and understood all the subtlety involved. Once again, how on Earth did Bill find the time?
I also imagine that there was much less 'adult supervision' of Microsoft. They probably did not take (any?) VC money, did they?
The scene today is very different, so I wonder how such a thing happens in modern-day startups. As an investor, I would be alarmed at such egregious misallocation of valuable founder/C-suite time and brainpower.
EDIT: I have read many an anecdote (from Joel Spolsky, for example), wherein Bill Gates was said to have pored over the minutest specification in documents spanning hundreds of pages and understood all the subtlety involved. Once again, how on Earth did Bill find the time?
I heard an anecdote where he received a technical briefing from a security team about a product in about 2003 and asked them meaningful, challenging technical questions in reply.
imo Shopify is a company barely in control of its codebase.
I hope he can continue to do his craft forever tbh.
Good memory: Lutke's blog is responsible for a lightbulb moment of mine back in 2007, regarding cache invalidation. The post is long gone, but it was entitled "The Secret to memcached" -- the advice, which is incredibly obvious in retrospect but wasn't to me at the time, is to manage cache invalidation by adding a unique ID to the cached asset, and letting the cache fill and expire items on its own. Going into it, I had assumed I should be removing cached items in the code that was utilizing the cache. I was ignorant. I still am, but less so about memcached. Thanks, Tobias!