Lessons from Stripe(markmcgranaghan.com)
markmcgranaghan.com
Lessons from Stripe
https://markmcgranaghan.com/lessons-from-stripe
132 comments
Three small businesses can easily end up bidding against each other on Google Ads for the same keywords, higher and higher, until they drive each other to the point of having precisely zero profit left to them, and all of the profit going to Google.
Making money is all well and good, but this is not empowering anyone.
Making money is all well and good, but this is not empowering anyone.
Much of small business is about geographic monopoly where simply putting yourself in front of customers that didn't know about you before is wealth creating.
That's a good point, though I have to wonder how often is there an actual micro-monopoly?
I imagine that two dry-cleaning places a mile away from each other would end up in competition eventually, and Google will drain all the profits from both.
Something is fundamentally unbalanced about this.
I imagine that two dry-cleaning places a mile away from each other would end up in competition eventually, and Google will drain all the profits from both.
Something is fundamentally unbalanced about this.
Coffee shops can be make or break depending on the corner of the intersection they are on so it can be a little bit ridiculous in terms of how small the geography is. I agree re: Google capturing the spread. Always want to control the means of distribution for your own business, ultimately.
You can argue that about any limited resource for which firms are in competition, like land.
Unlike most resources though, Google ads take into account a quality score, so if one of those merchants is lower quality than the rest e.g. users go back from the store to keep searching more often, then they'll have to pay more and all other things being equal, the more useful merchants will outcompete them.
In practice AdWords does enable many businesses that would otherwise find it too hard to reach a niche audience, and wouldn't be able to rank organically for a keyword because they have no inbound links.
Unlike most resources though, Google ads take into account a quality score, so if one of those merchants is lower quality than the rest e.g. users go back from the store to keep searching more often, then they'll have to pay more and all other things being equal, the more useful merchants will outcompete them.
In practice AdWords does enable many businesses that would otherwise find it too hard to reach a niche audience, and wouldn't be able to rank organically for a keyword because they have no inbound links.
Land is similar, yes. Still one could get a 5-year lease and plan their business around it. AdWords rates can go up in price any minute, and if they don't they create a false sense of security which is shattered when they finally do.
Could they enable a new business? They could, but as soon as they enable two businesses of the same kind in one place they will also take all the profits from both.
Could they enable a new business? They could, but as soon as they enable two businesses of the same kind in one place they will also take all the profits from both.
But that's not really how it works. It's not winner take all with the auctions. And if there are 3 businesses bidding, all 3 are going to get placement even if they all 3 have large enough budgets to buy all available impressions. Because there are more than one space in search results and other AdWords placements.
Even without that consideration, all 3 would also need to outbid one another while budgeting enough to take 100% of the traffic.
Even without that consideration, all 3 would also need to outbid one another while budgeting enough to take 100% of the traffic.
Ads are definitely not bad for Google and its employees, because they're a money printing machine for a company that's essentially an online ad monopoly.
Whether this is good for the small business merchants is questionable, since it turns out your ex-employer was literally stealing the money those companies/individuals were earning through ads: https://news.ycombinator.com/item?id=17103280. In the linked thread you can even read some testimonials from small merchants which you and many of the engineers were "helping".
Ads are definitely bad for regular humans though:
* they waste their computing resources and make their devices slow & use up the battery
* they serve malware (Google got hit too)
* they spy on them
* they distract and manipulate them
* etc
Ironically, they're also bad for the software industry, since now customers are used to getting things for "free", so many are no longer paying for software, undermining the revenue stream of... small business merchants.
With friends and helpers like these, business merchants would probably rather get punched in the face.
Whether this is good for the small business merchants is questionable, since it turns out your ex-employer was literally stealing the money those companies/individuals were earning through ads: https://news.ycombinator.com/item?id=17103280. In the linked thread you can even read some testimonials from small merchants which you and many of the engineers were "helping".
Ads are definitely bad for regular humans though:
* they waste their computing resources and make their devices slow & use up the battery
* they serve malware (Google got hit too)
* they spy on them
* they distract and manipulate them
* etc
Ironically, they're also bad for the software industry, since now customers are used to getting things for "free", so many are no longer paying for software, undermining the revenue stream of... small business merchants.
With friends and helpers like these, business merchants would probably rather get punched in the face.
thanks for this interesting perspective, I haven’t really thought about ads from this point of view before.
However, does the desire of advertisement buyers to convert justify the profiling and tracking of peoples behaviors to the extent that it is happening?
You don’t seem to provide any reasonable justification beyond that not only big but also small companies profit from this state of affairs.
This doesn’t make the power and behavior of google any less creepy or undesirable.
However, does the desire of advertisement buyers to convert justify the profiling and tracking of peoples behaviors to the extent that it is happening?
You don’t seem to provide any reasonable justification beyond that not only big but also small companies profit from this state of affairs.
This doesn’t make the power and behavior of google any less creepy or undesirable.
The justification is that better ads convert more, as he mentioned in his post. Making ads more efficient means a small, niche business can effectively spend money on ads that drive business. If you can't target well, then giant companies whose audiences are very broad are the only ones who can justify paying for ads. That's why TV advertising is dominated by huge companies - Budweiser/Chevy/etc. want everyone to see their ads, but John's vintage typewriter store will only spend money on ads if they can target customers in their geographic area with a very specific interest at a reasonable price.
Of course, the big part of your question is "to the extent that it is happening" - there's definitely an argument to be made that they're collecting too much info, but my point is merely to say that this kind of information collection allows for the kind of granular audience targeting that is disproportionately beneficial to small, niche businesses.
Of course, the big part of your question is "to the extent that it is happening" - there's definitely an argument to be made that they're collecting too much info, but my point is merely to say that this kind of information collection allows for the kind of granular audience targeting that is disproportionately beneficial to small, niche businesses.
Yeah, thank you for your second paragraph because that‘s exactly the point where things go off the rails. We could have granular geo-location and context (i.e., keywords) based targeting to satisfy smaller customers. Did anybody specifically complain about this when that was still the state of affairs?
Business is a relative project where your profit doesn‘t really depend on the absolute capabilities that you are using but your relative advantage over your competitors. If fine-grained targeting wasn‘t available then small businesses would do fine without it. So the real problem is the competition between google, facebook, amazon and others which leaves users as the products. We would probably need some like an arms control treaty to tame these behemoths and make this situation somewhat sustainable in the long run.
Business is a relative project where your profit doesn‘t really depend on the absolute capabilities that you are using but your relative advantage over your competitors. If fine-grained targeting wasn‘t available then small businesses would do fine without it. So the real problem is the competition between google, facebook, amazon and others which leaves users as the products. We would probably need some like an arms control treaty to tame these behemoths and make this situation somewhat sustainable in the long run.
That’s actually really interesting to hear, although tbh I strongly disagree.
Serious question: did you and fellow googlers ever have discussions like “what if that argument is actually not true? What if it’s just a convenient rationalisation?” I think even if you dismissed that POV, it’s a sign of a healthy culture that you can question the mission, even if you then decide it’s right.
Serious question: did you and fellow googlers ever have discussions like “what if that argument is actually not true? What if it’s just a convenient rationalisation?” I think even if you dismissed that POV, it’s a sign of a healthy culture that you can question the mission, even if you then decide it’s right.
I recently had my onsite for Engineering Manager role at Stripe. Stripe interview process is lengthy, I met 14-15 people. There's a mini onsite (3 45 minute interviews) if you cleaer that one there will be a full day onsite which consists
1:1 Management Roleplay, Technical Roleplay, Technical Discussion, 3 behavioral interviews
there is a lot of overlap between mini onsite and full day onsite.
After meeting so many folks and spending so much time, the feedback you get is minimal. In my case the recruiter informed that everything was great but the team did not feel the spark.
1:1 Management Roleplay, Technical Roleplay, Technical Discussion, 3 behavioral interviews
there is a lot of overlap between mini onsite and full day onsite.
After meeting so many folks and spending so much time, the feedback you get is minimal. In my case the recruiter informed that everything was great but the team did not feel the spark.
> the feedback you get is minimal
I've been in hiring roles for 15+ years and I've interviewed somewhere around 1000 candidates. I don't work at Stripe, but I can probably explain why you didn't get feedback.
Giving candidates feedback is really fraught. First, even honest feedback is unlikely to be satisfying. It's entirely possible that your recruiter told you exactly what the team's feedback was. "Your answers were fine, but we want better" or "... but candidate X was stronger" is super-common, but very unsatisfying. Second, while some candidates take feedback well, most don't. When I've been specific -- e.g. "your answers to questions about software architecture fell short" or "you didn't clearly demonstrate experience solving production issues" -- most candidates argue or get angry. People are really, really bad about taking criticism well. I've even had two candidates threaten to sue me, and one threaten violence. These are obviously outliers, but they do happen.
Finally, the upside is limited: to the company, what's the value of feedback? We've already decided you're not a fit, so what value is feedback to us? Especially if, as I've explained, you're unlikely to get much value from it either?
The best way to get feedback about your prospects as a job candidate isn't from someone a company you're interviewing with. Instead, ask someone you trust to conduct a mock interview with you and give you feedback. You'll be more to be able to listen if it's someone you know has your best interest in mind; they'll be more likely to give honest feedback if they know you'll take it well.
I've been in hiring roles for 15+ years and I've interviewed somewhere around 1000 candidates. I don't work at Stripe, but I can probably explain why you didn't get feedback.
Giving candidates feedback is really fraught. First, even honest feedback is unlikely to be satisfying. It's entirely possible that your recruiter told you exactly what the team's feedback was. "Your answers were fine, but we want better" or "... but candidate X was stronger" is super-common, but very unsatisfying. Second, while some candidates take feedback well, most don't. When I've been specific -- e.g. "your answers to questions about software architecture fell short" or "you didn't clearly demonstrate experience solving production issues" -- most candidates argue or get angry. People are really, really bad about taking criticism well. I've even had two candidates threaten to sue me, and one threaten violence. These are obviously outliers, but they do happen.
Finally, the upside is limited: to the company, what's the value of feedback? We've already decided you're not a fit, so what value is feedback to us? Especially if, as I've explained, you're unlikely to get much value from it either?
The best way to get feedback about your prospects as a job candidate isn't from someone a company you're interviewing with. Instead, ask someone you trust to conduct a mock interview with you and give you feedback. You'll be more to be able to listen if it's someone you know has your best interest in mind; they'll be more likely to give honest feedback if they know you'll take it well.
>> What's the upside for the company?
That I'd apply with you again - if I enjoyed the process and the specific feedback either suggests to me that I didn't articulate well or I need to be better prepared. If the process is bad or you treat me like cattle or you provide no feedback, I don't know if the problem is you or me and I don't think I'd try to work for your company again.
That I'd apply with you again - if I enjoyed the process and the specific feedback either suggests to me that I didn't articulate well or I need to be better prepared. If the process is bad or you treat me like cattle or you provide no feedback, I don't know if the problem is you or me and I don't think I'd try to work for your company again.
I think for some glaring problems, feedback is helpful. But in a lot of cases, companies have no clue if they are really hiring the best candidate. It's a crap shoot whether they realize it or not.
But if you have 50 candidates that are all qualified for the job, you are basically just looking for random reasons to exclude candidates until you narrow it down to a few.
But if you have 50 candidates that are all qualified for the job, you are basically just looking for random reasons to exclude candidates until you narrow it down to a few.
The hiring process isn't just about the candidates you make end up hiring. Everyone that comes in contact with it is going to form an impression of your company--positive or negative.
Clearly there's some kinds of feedback that is likely to leave a negative impression (see those that threatened violence!) but on the hand putting someone through days and days of interviews and then saying "we didn't feel a spark" also is going to leave a negative impression.
The rare interviews where I received concrete actionable feedback afterwords, even though it still stung I have a more positive impression than those where I got some pabulum (much less the ones that ghosted me).
Clearly there's some kinds of feedback that is likely to leave a negative impression (see those that threatened violence!) but on the hand putting someone through days and days of interviews and then saying "we didn't feel a spark" also is going to leave a negative impression.
The rare interviews where I received concrete actionable feedback afterwords, even though it still stung I have a more positive impression than those where I got some pabulum (much less the ones that ghosted me).
Not to minimize your frustration, but if you put yourself on the other side of the hiring table, I could see "spark" as a coded word for "soft skills." The good news is, it appears that because they spent a significant amount of time with you and they had that many interviewers involved, they were very seriously considering your offer.
An EM role is going to be less about your technical competency and more on things you talked to them about. If you think back to what you were asked and the stories you shared with the hiring folks, maybe try to reflect on your lessons and behaviors in your stories. Were you considering your organization's top line goals when you pushed your team into finishing a project. Were you considering your direct reports' own career goals. Were you displaying empathy and asked the right questions to understand some bigger meaning? It's hard man to see any of that on your own, I know, it sucks, but the recruiter is busy talking to 100s of people and if you aren't going to be a sure bet, they have to cut their ties and move on.
An EM role is going to be less about your technical competency and more on things you talked to them about. If you think back to what you were asked and the stories you shared with the hiring folks, maybe try to reflect on your lessons and behaviors in your stories. Were you considering your organization's top line goals when you pushed your team into finishing a project. Were you considering your direct reports' own career goals. Were you displaying empathy and asked the right questions to understand some bigger meaning? It's hard man to see any of that on your own, I know, it sucks, but the recruiter is busy talking to 100s of people and if you aren't going to be a sure bet, they have to cut their ties and move on.
If I put myself on the other side of the hiring table, I would have the common decency to give someone at least a little bit of feedback if they gave up more than a full day of their time to wrestle themselves through our convoluted hiring process.
Employees doing the evaluation are usually separated from that communication. They probably wrote down fairly detailed notes each into an interview management system and talked about it in a 30m meeting.
But it's up to the recruiting dept to deliver the news / feedback, and because it's a large legal liability trap to actually give feedback and since the recruiters don't have the technical background to understand the feedback in the first place, usually tell candidates not much at all. On top of that, most recruitment departments are bad at most companies because of structural issues in being a recruiter in a first place.
To really solve this, you need to remove the legal liability trap. The industry works around it with backchannel communication of the feedback, but that usually means you have to know people on the team interviewing you.
But it's up to the recruiting dept to deliver the news / feedback, and because it's a large legal liability trap to actually give feedback and since the recruiters don't have the technical background to understand the feedback in the first place, usually tell candidates not much at all. On top of that, most recruitment departments are bad at most companies because of structural issues in being a recruiter in a first place.
To really solve this, you need to remove the legal liability trap. The industry works around it with backchannel communication of the feedback, but that usually means you have to know people on the team interviewing you.
>> everything was great but the team did not feel the spark.
That's a really poor response. I can understand that many of us would have that same implicit gut feeling, but recruitment, and especially supposedly exemplar recruiters like Stripe, have an obligation to articulate their reasons for decisions in a much more grounded framework.
When I say obligation I mean primarily to their internal stakeholders (to candidates would be nice, but let's be realistic here); the team's primary objective is to define & put process around finding, assessing and hiring people, not optimize for efficiently applying bias and "gut feeling" quickly to large pools of people.
That's a really poor response. I can understand that many of us would have that same implicit gut feeling, but recruitment, and especially supposedly exemplar recruiters like Stripe, have an obligation to articulate their reasons for decisions in a much more grounded framework.
When I say obligation I mean primarily to their internal stakeholders (to candidates would be nice, but let's be realistic here); the team's primary objective is to define & put process around finding, assessing and hiring people, not optimize for efficiently applying bias and "gut feeling" quickly to large pools of people.
It's unfortunate, but it's rare for any company to give direct feedback when passing on a candidate. It creates legal exposure for very little benefit to the company since they've already decided not to invest further. As a candidate, I hate getting rejections with no feedback, but I understand.
Wow!
I wonder how much of interview overhead it would be for Sr Enggrs and Managers. Multiple rounds of interviews like this is overkill.
I wonder how much of interview overhead it would be for Sr Enggrs and Managers. Multiple rounds of interviews like this is overkill.
No kidding. When I hear about long grueling hiring processes like this, I wonder what the hiring company is finding out in the 8th hour of interviews that they didn't in the 7th. Or in the case of one job my sister applied to: the 8th interview!
In this case, maybe that one person, at the end of a long day, who didn't feel the spark?
In this case, maybe that one person, at the end of a long day, who didn't feel the spark?
Maybe to get internal buyin?
It's not so easy to flip the bozo-bit and write someone off if you were involved with the interview process to hire someone.
It's not so easy to flip the bozo-bit and write someone off if you were involved with the interview process to hire someone.
Roleplay?
I can't imagine doing that in an interview, especially with people I don't know... is it as embarrassingly cringey as it sounds?
I can't imagine doing that in an interview, especially with people I don't know... is it as embarrassingly cringey as it sounds?
Yes, it was cringey. In both the role plays the actors were either fresh out of college with 1 or 2 years experience at Stripe and I am not sure if they were really calibrated to do the leadership interviews.
I had a similar experience at Stripe and it was the low point in an otherwise (what I felt) great interview.
I was interviewing for a forward-deployed/field engineer role and they expected me in an hour to come up with a nuanced strategy, down to SOAP vs REST with timelines and specific commitments, on how I'd do a large-scale payment integration with a partner. The interview was a disaster. Not only was it clear that my interviewer had little real-world experience in the sort of environment we were roleplaying (with internal politics, tons of legacy, teams of varying quality and type of skills, etc) but it seemed they were looking for something really specific I just couldn't grasp.
It went horribly and I didn't get an offer. So maybe a counterpoint to an otherwise great company with great process. To their credit, they said they didn't really have a bar for this (newish) role. But still, it was amateurish and all-around bad.
I was interviewing for a forward-deployed/field engineer role and they expected me in an hour to come up with a nuanced strategy, down to SOAP vs REST with timelines and specific commitments, on how I'd do a large-scale payment integration with a partner. The interview was a disaster. Not only was it clear that my interviewer had little real-world experience in the sort of environment we were roleplaying (with internal politics, tons of legacy, teams of varying quality and type of skills, etc) but it seemed they were looking for something really specific I just couldn't grasp.
It went horribly and I didn't get an offer. So maybe a counterpoint to an otherwise great company with great process. To their credit, they said they didn't really have a bar for this (newish) role. But still, it was amateurish and all-around bad.
Maybe I'm wrong about this but it seems odd for a company to hire externally for engineering manager except in extremely rare cases? From experience in a few companies I can't think of people who were hired directly to that role (e.g. a people manager of engineers). I basically only see external ICs and senior engineering leader hires and internal promotion of engineers to engineering managers.
Hiring externally for any role at any company is very normal (e.g., FANG companies hire external EMs all the time).
In my experience, when you start with a team of great engineers who all love to build it's surprisingly rare that one of them wants to stop building and start managing. I'd actually say most of the time, by which I mean more than 50% of the time, an engineering manager is hired rather than promoted. Obviously this is anecdotal.
>internal promotion of engineers to engineering managers
That assumes you have ICs who want to and have the skills to go into management. Promoting people for the sake of filling a chair leads to really bad dynamics.
Moreover, newly promoted managers lack experience and if all you have is the blind leading he blind then it tends to create issues. You need some seasoned people in the mix to provide guidance.
That assumes you have ICs who want to and have the skills to go into management. Promoting people for the sake of filling a chair leads to really bad dynamics.
Moreover, newly promoted managers lack experience and if all you have is the blind leading he blind then it tends to create issues. You need some seasoned people in the mix to provide guidance.
I think it's tricky to hire managers externally, because it's a hard skill to measure -- you can't really manage on a whiteboard. But, side-grading ICs to management isn't always ideal, so you should be open to hiring managers externally.
Same here experiences here. Found it to be super frustrating.
This is what happens when you start to overpay your HRs
I don't know, this super optimism as a value might lead to situations where something is going clearly in the wrong direction, but people get afraid of pointing it out due to fear of going against the company values, and getting natural overly optimistic personalities out of their delusional high and back into reality.
From the engineering friends I know at Stripe, you'd be correct: there's a strong cultural push to not question the status quo / prevailing wisdom.
On the tech side, they continue to build on top of a broken foundation (MongoDB), resulting in millions of man hours wasted dealing with the complete lack of transactions. Mongo now has transactions, but last I heard Stripe was still running a very outdated version and spending absurd amounts of time dealing with issues that transactions would have entirely avoided. If you suggested that maybe it'd be worth changing course to something like Postgres because of the insane amount of work being wasted, you'd be shut down for not being optimistic enough.
On the tech side, they continue to build on top of a broken foundation (MongoDB), resulting in millions of man hours wasted dealing with the complete lack of transactions. Mongo now has transactions, but last I heard Stripe was still running a very outdated version and spending absurd amounts of time dealing with issues that transactions would have entirely avoided. If you suggested that maybe it'd be worth changing course to something like Postgres because of the insane amount of work being wasted, you'd be shut down for not being optimistic enough.
The fact that Stripe ever used MongoDB is dubious enough, but that they've endured on it well past the fad's expiry is reason enough for me to avoid doing business with them in the future.
Handling money is serious business and if one insists on using the unproven flavor-of-the-week to do so, they'd better make sure it has at least the fundamental requirements to accomplish its task, e.g., transactions/MVCC. Stripe apparently failed to do so, and has chosen to propagate that failure for many years.
One can make excuses for the spunky startup with a handful of employees needing to save time by using what they know or whatever, but Stripe is well past that point, and serious people should've taken over by now. I'd expect "replace Mongo with a real database" to be near the top of the todo list for any serious people.
Handling money is serious business and if one insists on using the unproven flavor-of-the-week to do so, they'd better make sure it has at least the fundamental requirements to accomplish its task, e.g., transactions/MVCC. Stripe apparently failed to do so, and has chosen to propagate that failure for many years.
One can make excuses for the spunky startup with a handful of employees needing to save time by using what they know or whatever, but Stripe is well past that point, and serious people should've taken over by now. I'd expect "replace Mongo with a real database" to be near the top of the todo list for any serious people.
> The fact that Stripe ever used MongoDB is dubious enough, but that they've endured on it well past the fad's expiry is reason enough for me to avoid doing business with them in the future.
Just wait until you find out what technology actual banks use!
Just wait until you find out what technology actual banks use!
COBOL, Fortran, mainframes, and other 70s-era tech are much more trustworthy than new-fangled VC landgrabs like MongoDB. As long as the banks haven't transitioned everything to Cassandra or whatever, I don't think we'll have an issue there.
> Handling money is serious business
How do you say this without knowing that governments around the world feel the same way and so heavily regulate this space? Do you know any publicly available recommendation from these regulators against MongoDB?
> replace Mongo with a real database to be near the top of the todo list
This is where the businesses separate themselves from technology enthusiasts. Let's assume this is made the top priority, in your opinion, what task immediately follows? Or what former priority does it replace?
How do you say this without knowing that governments around the world feel the same way and so heavily regulate this space? Do you know any publicly available recommendation from these regulators against MongoDB?
> replace Mongo with a real database to be near the top of the todo list
This is where the businesses separate themselves from technology enthusiasts. Let's assume this is made the top priority, in your opinion, what task immediately follows? Or what former priority does it replace?
> Do you know any publicly available recommendation from these regulators against MongoDB?
If there isn't one, there should be.
>Let's assume this is made the top priority, in your opinion, what task immediately follows? Or what former priority does it replace?
Mongo jeopardizes the overall security and reliability of the system, thus exposing Stripe to serious liability, so I'd say replacing it is part of the basic expectation from any production-level computer system. Obviously I don't have a copy of "Important Stripe Person's Priority List", but I'd assume such basic functionality is part of an implicitly-assumed functional baseline, and that Stripe would expect said important people to escalate and develop a plan to handle such a fundamental flaw ASAP.
If we were just tracking people's favorite cat videos or something, it'd be one thing. Attacks targeting malicious data modification and exploitation of races wouldn't really be that much of a concern. It's not great to have that attack surface but not necessarily the end of the world if someone's favorite cat videos get added to their account 1000 times. But when we're dealing with money, this kind of thing needs to be taken seriously, and tacking on a big pile of crap in front of the datastore is not a serious way to address its fundamental shortcomings, at least not when there are many better, proven options on the market.
Just in case someone thinks I'm being an alarmist here, this has already happened; multiple bitcoin exchanges have been pwned due to their reliance on Mongo's flawed semantics. [1] I can only assume that systems like Stripe haven't been attacked similarly because it would provoke the ire of much bigger fish, and why do that when you can knock over smaller bitcoin exchanges?
[1] http://hackingdistributed.com/2014/04/06/another-one-bites-t...
If there isn't one, there should be.
>Let's assume this is made the top priority, in your opinion, what task immediately follows? Or what former priority does it replace?
Mongo jeopardizes the overall security and reliability of the system, thus exposing Stripe to serious liability, so I'd say replacing it is part of the basic expectation from any production-level computer system. Obviously I don't have a copy of "Important Stripe Person's Priority List", but I'd assume such basic functionality is part of an implicitly-assumed functional baseline, and that Stripe would expect said important people to escalate and develop a plan to handle such a fundamental flaw ASAP.
If we were just tracking people's favorite cat videos or something, it'd be one thing. Attacks targeting malicious data modification and exploitation of races wouldn't really be that much of a concern. It's not great to have that attack surface but not necessarily the end of the world if someone's favorite cat videos get added to their account 1000 times. But when we're dealing with money, this kind of thing needs to be taken seriously, and tacking on a big pile of crap in front of the datastore is not a serious way to address its fundamental shortcomings, at least not when there are many better, proven options on the market.
Just in case someone thinks I'm being an alarmist here, this has already happened; multiple bitcoin exchanges have been pwned due to their reliance on Mongo's flawed semantics. [1] I can only assume that systems like Stripe haven't been attacked similarly because it would provoke the ire of much bigger fish, and why do that when you can knock over smaller bitcoin exchanges?
[1] http://hackingdistributed.com/2014/04/06/another-one-bites-t...
Transactions aren't the only solution for that problem space, and using them can introduce other (likely even harder-to-solve) problems with distributed systems.
FWIW, having worked with the Stripe API for ~5 years and put all our sales through them for the lifetime of the business, I wouldn’t have guessed they used MongoDB. The API is mostly excellent, and it has always presented a consistent view of the world to our side.
The same cannot he said for a number of other services we use, including those from some companies with supposedly very good tech credentials. We’ve had to back out of integrations due to the clear evidence that they are non transactional and/or eventually consistent.
The same cannot he said for a number of other services we use, including those from some companies with supposedly very good tech credentials. We’ve had to back out of integrations due to the clear evidence that they are non transactional and/or eventually consistent.
You couldn't tell what backend they use just from their API unless there's some errant exceptions leaking information. Their test API might give us some clues. It can only purge your data one record at a time, at about 1.5 records a second, and locks your API credentials for however long it takes to crash or finish iterating through each object. I don't know what storage engine would have that constraint but they've stuck with it for years and simulate a fast test API with a whole other piece of software instead of just having a fast test API.
What storage options don't have the ability to delete all your records at once quickly? If it was MongoDB there would surely be an index attaching the data to your account. Same with PostgreSQL.
What storage options don't have the ability to delete all your records at once quickly? If it was MongoDB there would surely be an index attaching the data to your account. Same with PostgreSQL.
You can make all those mistakes with a traditional RDBMS. Iterating over objects to delete them one at a time with database round trips is also a common mistake people make with ORMs(along with many other things that could have been done in one statement).
I’ve worked with Stripe enough to have a hunch that it was Mongo. Never really thought much about it until it was confirmed here just now. There’s a lot of immutability in their system, tons of it makes sense but some of it smells of a weak data store.
You are laying down some hefty assumptions here and I'm not sure they are fair.
You seem to indicate that Stripe has consistency issues with their payment APIs. Have you seen this? Having worked directly on these APIs (note, I said ON and not WITH, I worked at Stripe on these products), it would greatly surprise me to see non-trivial consistency issues with their data model that surrounds money movement. Stripe might be on Mongo, but assuming that they have consistency issues because of that is quite a leap. Remember, Stripe is regulated federally and in most states, as well as in their international jurisdictions, so the claim you are making would be one regulators might be interested in if there is any veracity.
You also seem to be insisting that there are no "serious people" at Stripe, and that might be true! At least, I hope no one has mistaken me for a "serious person". But what does "serious" mean here and why can only "serious people" work on financial infrastructure? Moreover, why do "serious people" not see Mongo as a "real database"? In fact, what is a "real database" to you, other than something that has transactions?
I'm very interested in your answer here; while I had my issues with Mongo in my 2.5 years at Stripe, most of the issues I encountered I would lump into the "this is a trade-off consequence" bucket, as Mongo solved a whole lot of other issues for me. The other main bucket I would have used was "distributed database problem". I know Stripe had no plans, nor should they, to move to a non-distributed data store.
No one is forcing you to use Stripe, but I would suggest that attacking Stripe for their technology usage is not very useful.
You seem to indicate that Stripe has consistency issues with their payment APIs. Have you seen this? Having worked directly on these APIs (note, I said ON and not WITH, I worked at Stripe on these products), it would greatly surprise me to see non-trivial consistency issues with their data model that surrounds money movement. Stripe might be on Mongo, but assuming that they have consistency issues because of that is quite a leap. Remember, Stripe is regulated federally and in most states, as well as in their international jurisdictions, so the claim you are making would be one regulators might be interested in if there is any veracity.
You also seem to be insisting that there are no "serious people" at Stripe, and that might be true! At least, I hope no one has mistaken me for a "serious person". But what does "serious" mean here and why can only "serious people" work on financial infrastructure? Moreover, why do "serious people" not see Mongo as a "real database"? In fact, what is a "real database" to you, other than something that has transactions?
I'm very interested in your answer here; while I had my issues with Mongo in my 2.5 years at Stripe, most of the issues I encountered I would lump into the "this is a trade-off consequence" bucket, as Mongo solved a whole lot of other issues for me. The other main bucket I would have used was "distributed database problem". I know Stripe had no plans, nor should they, to move to a non-distributed data store.
No one is forcing you to use Stripe, but I would suggest that attacking Stripe for their technology usage is not very useful.
> what is a "real database" to you, other than something that has transactions?
How about "Atomicity, Consistency, Isolation, Durability"? That would be a good start for a database.
A nearby comment suggests lack of set-based operations (e.g. delete), so that would be another nice thing to add.
How about "Atomicity, Consistency, Isolation, Durability"? That would be a good start for a database.
A nearby comment suggests lack of set-based operations (e.g. delete), so that would be another nice thing to add.
I'm currently dealing with precisely this exact scenario and database. Dated Mongo, homegrown ORM, a closet full of opaque hacks that are apparently all related to some MongoDB feature, etc. And I don't know if this is a cultural issue with MongoDB users, but my god the database has its hooks throughout the stack, all the way to the UI. It is amazing to see shit like this in 2019 and I've seen some heaps over the past few decades in this field.
But the optimism point resonated. Before reading your comment, I actually reflected on whether I am being pessimistic. I'll tell you why: at this point in the game, cleaning up the software would effectively require a rewrite. And the company is in no position to do this. They do not have the talent, leadership, or effective recruiting necessary to undertake a meaningful second system effort.
So I think I've come to the realization that the job is a poor fit and I should move on, but while I am here it actually wouldn't help pointing out the obvious issues with the system. There really is no point.
But the optimism point resonated. Before reading your comment, I actually reflected on whether I am being pessimistic. I'll tell you why: at this point in the game, cleaning up the software would effectively require a rewrite. And the company is in no position to do this. They do not have the talent, leadership, or effective recruiting necessary to undertake a meaningful second system effort.
So I think I've come to the realization that the job is a poor fit and I should move on, but while I am here it actually wouldn't help pointing out the obvious issues with the system. There really is no point.
It is always amazing to me that people downvote actual anecdotes that highlight the arguments being made, while at the same time upvote rampant speculation that has no basis in fact.
Having worked at Stripe, these exact examples came up more than once. I can verify their authenticity.
Having worked at Stripe, these exact examples came up more than once. I can verify their authenticity.
Wouldn't be easier to just upgrade mongodb, which is no longer broken? Unless its api is not backward compatible?
From what I understand there have been various attempts at this, as well as other data storage solutions, but they always run into issues. MongoDB is being used as a database and data warehouse and there are always complications.
Updating any non-trivial layer in your stack usually gets you into "complications" - that alone must never be an acceptable reason to be stuck on old versions, especially if there are serious problems with them.
You simply chew through those complications. Again and again. And try to optimize your application for less complications in the future, if possible.
You simply chew through those complications. Again and again. And try to optimize your application for less complications in the future, if possible.
I agree in the sense that small problems which have technical solutions should be pointed out and overcome. However, in later-stage startups and in larger companies, trying to change the direction might not be the wise thing to do. This highly depends on your position, but as an individual contributor and even as someone in the lower management lane (manager, director), your job most of the time is implementing the CEO's vision, whatever that is. If you find yourself disagreeing with that vision, it might be best to change jobs.[1]
In smaller startups, where you might be able to steer more aggressively, you should still ask yourself whether the company is clearly in the wrong direction only from your perspective or if there are factors which you don't have any insight into (no matter how close you are to founder(s)) which would re-contextualize the issue.
Not disagreeing, but these are some caveats which I found useful to keep in mind.
[1] - There was a story about someone from Apple(?) here a while ago who was let go after a lunch with <someone important> where he expressed his dissatisfaction with the company's direction. He then took up japanese caligraphy. Maybe someone remembers and can link it here.
In smaller startups, where you might be able to steer more aggressively, you should still ask yourself whether the company is clearly in the wrong direction only from your perspective or if there are factors which you don't have any insight into (no matter how close you are to founder(s)) which would re-contextualize the issue.
Not disagreeing, but these are some caveats which I found useful to keep in mind.
[1] - There was a story about someone from Apple(?) here a while ago who was let go after a lunch with <someone important> where he expressed his dissatisfaction with the company's direction. He then took up japanese caligraphy. Maybe someone remembers and can link it here.
Having thought about this issue a lot previously, I think it's important to not overgeneralize based on vague value statements.
Optimism can possibly mean anything from not blocking new initiatives through year-long corporate approval processes to cult-like you-have-to-drink-our-kool-aid internal communication that shuts down any and all critical voices and oversells what a company is doing and where it is currently at even to their own employees internally.
For me personally, optimism is a good thing when it means not taking issues as set in stone, genuinely believing that the future will be better than now and that one has some amount of control over that, and trying things out that might not work instantly - all while retaining one's critical thinking and being honest about the facts.
I have never met anyone who works at stripe or heard an inside take, but from the interviews with the founders I've heard, they don't strike me as the kind of people to push the delusional kind of "optimism" and shut down critical voices.
Optimism can possibly mean anything from not blocking new initiatives through year-long corporate approval processes to cult-like you-have-to-drink-our-kool-aid internal communication that shuts down any and all critical voices and oversells what a company is doing and where it is currently at even to their own employees internally.
For me personally, optimism is a good thing when it means not taking issues as set in stone, genuinely believing that the future will be better than now and that one has some amount of control over that, and trying things out that might not work instantly - all while retaining one's critical thinking and being honest about the facts.
I have never met anyone who works at stripe or heard an inside take, but from the interviews with the founders I've heard, they don't strike me as the kind of people to push the delusional kind of "optimism" and shut down critical voices.
I completely agree with you as taken verbatim.
However, what they are really talking about is that "why will it fail" is a "conversation ender", and "how might it work" is a "conversation continuer".
Those are probably bad terms, but hopefully you get my drift. One promotes conversations to end, and one encourages conversations to continue. I would personally promote that viewpoint instead to keep people from doing exactly what you are talking about.
However, what they are really talking about is that "why will it fail" is a "conversation ender", and "how might it work" is a "conversation continuer".
Those are probably bad terms, but hopefully you get my drift. One promotes conversations to end, and one encourages conversations to continue. I would personally promote that viewpoint instead to keep people from doing exactly what you are talking about.
I think, in the case of brevity, the article probably glossed over some important parts of this topic. (Which rightly could be their own article.)
There's a clear difference between giving an unusual or unconventional idea a chance; versus allowing an idea guy to let his imagination run away.
In the former, giving an unusual or unconventional idea a chance, is how progress is made. Fundamentally, a startup is always taking a chance on an unusual or unconventional idea. (When I worked for a successful startup, a lot of people at networking events thought what we were doing was weird and unconventional. It wasn't.)
In the latter case, when an idea guy is allowed to let his imagination run away, it just kills the company. Either he's wasting all his time, or even worse, wasting everyone else's time. (These are the kind of people who's whine that "we're supposed to be optimistic" when everyone in the room clearly rejects their ideas.)
There's a clear difference between giving an unusual or unconventional idea a chance; versus allowing an idea guy to let his imagination run away.
In the former, giving an unusual or unconventional idea a chance, is how progress is made. Fundamentally, a startup is always taking a chance on an unusual or unconventional idea. (When I worked for a successful startup, a lot of people at networking events thought what we were doing was weird and unconventional. It wasn't.)
In the latter case, when an idea guy is allowed to let his imagination run away, it just kills the company. Either he's wasting all his time, or even worse, wasting everyone else's time. (These are the kind of people who's whine that "we're supposed to be optimistic" when everyone in the room clearly rejects their ideas.)
[deleted]
> super optimism
I finally came to the conclusion ( for now ) that these sort of super optimism persist precisely and only because they are Startups.
Optimism as a Culture only works when it happens from those who held power to all the way down. It doesn't really work bottom up. When the company Culture is toxic, or bureaucratic as in every large companies, no optimism is going to save you.
In Startup, everyone is in it together, and it is mostly a flat or very thin structure. There is a huge transparency in vision and Goal. Everyone knew what their others are doing, and as long as everyone is fighting for it there will always be a fighting chance.
I finally came to the conclusion ( for now ) that these sort of super optimism persist precisely and only because they are Startups.
Optimism as a Culture only works when it happens from those who held power to all the way down. It doesn't really work bottom up. When the company Culture is toxic, or bureaucratic as in every large companies, no optimism is going to save you.
In Startup, everyone is in it together, and it is mostly a flat or very thin structure. There is a huge transparency in vision and Goal. Everyone knew what their others are doing, and as long as everyone is fighting for it there will always be a fighting chance.
Sure. But maybe that means one could reword its "we're going headfirst into the wall" prophecy into something more constructive, like "I think we could avoid this pitfall, here's my plan".
[deleted]
How do you teach optimism in the way Stripe is looking for though?
As someone with a hefty amount of anxiety and depression weighing down on me most of the time, I often feel like I'm the only person in the room who can't get excited about things, but I don't want to be the weight bringing everyone else down. Is there not a danger of this value becoming a proxy for "people with depression and similar mental health problems aren't welcome at Stripe"?
As someone with a hefty amount of anxiety and depression weighing down on me most of the time, I often feel like I'm the only person in the room who can't get excited about things, but I don't want to be the weight bringing everyone else down. Is there not a danger of this value becoming a proxy for "people with depression and similar mental health problems aren't welcome at Stripe"?
Optimism doesn't always manifest in excitement. It's mostly a believe that a target can be achieved, even though how to achieve it isn't always clear. And it applies outside of work as well. I have depression episodes, and I sympathize with those who are unable to overcome it and so take their lives, but one of the reasons I'm still here is my optimism that every bout will eventually end, and I'll be happy and usual again.
I have a similar problem. I'm always pessimistic, stressful and anxious. After my job interview for a machine learning job, HR gave me a personaly test. It turns out that they were literally horrified by the results and told me that every response I made was the exact opposite they were looking for. I got the job nevertheless but mental issues do become a barrier in one's career. I knowingly try to be alone in the office, don't take any breaks and don't talk to anyone unless it's necessary. Haven't heard a complaint yet. But I would suggest to the opposite, communication is important.
That's interesting, what was the point of the personality test? I mean it's crappy if you can 'fail' it, so in that sense I'm glad it's not an actual hurdle, but then why give a candidate the test in the first place? If it's to know how best to work with you, then they should save it for after you've been hired.
They said they wanted to see if I was a good fit for the organization's culture. Technical interview went really well, so they did not cared that much about the test results. But they requested me to take the test again hoping I would get better results. What a joke :)
Teach or acquire? Talk to people about it often. Keep an open mind. Take note when someone exhibits the value in a way you admire and ask yourself: how can I emulate that next time I feel pessimistic about something?
I think teach. Sometimes people need some hand-holding for this stuff, otherwise the hopelessness takes over, e.g "I can never be that person, I might as well give up".
Edit: I know that this is largely the role of therapists and mentors, but I think there's a tendency in startups to expect people to teach themselves the skills they need to do their jobs, rather than provide training and guidance. So i'm wondering if this is the kind of thing that could be trained.
Edit: I know that this is largely the role of therapists and mentors, but I think there's a tendency in startups to expect people to teach themselves the skills they need to do their jobs, rather than provide training and guidance. So i'm wondering if this is the kind of thing that could be trained.
> "people with depression and similar mental health problems aren't welcome at Stripe"
That an interesting question. Would you like to work in a tandem with a person with mental health problems? Is it the same as asking "people with chronic fatigue and similar health issues are not welcome at company specialized at manual labor" ?
That an interesting question. Would you like to work in a tandem with a person with mental health problems? Is it the same as asking "people with chronic fatigue and similar health issues are not welcome at company specialized at manual labor" ?
I work with several people who suffer from a range of mental health problems, and most of my closest friends can be described the same way. So you could even argue that it's my preference.
As for whether it's the same question as the chronic fatigue thing, i'm not sure. I think largely it comes down to how negative their perceived impact on their colleagues is?
As for whether it's the same question as the chronic fatigue thing, i'm not sure. I think largely it comes down to how negative their perceived impact on their colleagues is?
Stripe is a business.
The goal of a business is profit, because without it, it dies.
Are you able to bring profit, moreso than other candidates? You're hired.
That's it, nobody gives a shit about depression and if some company decides on arbitrary qualities that are unrelated to profit in regards to hiring, it'll be beaten by a company that focuses on profits.
Depression, feelings and values are great topics to be discussed in romance novels, tv shows, dinner parties and other leisurely activities that you may engage in, after you bring value to society you're a part of, or else society will cease to exist.
The goal of a business is profit, because without it, it dies.
Are you able to bring profit, moreso than other candidates? You're hired.
That's it, nobody gives a shit about depression and if some company decides on arbitrary qualities that are unrelated to profit in regards to hiring, it'll be beaten by a company that focuses on profits.
Depression, feelings and values are great topics to be discussed in romance novels, tv shows, dinner parties and other leisurely activities that you may engage in, after you bring value to society you're a part of, or else society will cease to exist.
You're not really addressing the core question here, it was never really about the profitability a depressed person can bring, it was about whether hiring based on a signal of an optimistic personality is unnecessarily exclusionary.
Values-based hiring isn't a magic bullet, it can lead to problems such as having a monoculture. And as I alluded to before, it's possible for hiring too rigidly against a value to unintentionally become a proxy for exclusionary bias.
Values-based hiring isn't a magic bullet, it can lead to problems such as having a monoculture. And as I alluded to before, it's possible for hiring too rigidly against a value to unintentionally become a proxy for exclusionary bias.
[deleted]
It's interesting to hear an internal perspective of how "great" Stripe's recruiting processes is.
As an external applicant, I would say it was better than most, but still an internal recruiter shit-show. Week-long windows between email responses, dropped balls, and even no-shows for scheduled calls. Seeing this contrast makes me wonder if Stripe employees are just high on their own supply?
Recruiting is hard. I can only imagine it gets harder at scale. I'd bet that Stripe's best recruiting is done through employee referrals... and that is probably building a mono-culture.
As an external applicant, I would say it was better than most, but still an internal recruiter shit-show. Week-long windows between email responses, dropped balls, and even no-shows for scheduled calls. Seeing this contrast makes me wonder if Stripe employees are just high on their own supply?
Recruiting is hard. I can only imagine it gets harder at scale. I'd bet that Stripe's best recruiting is done through employee referrals... and that is probably building a mono-culture.
Obviously anecdotal, but when I interviewed with Stripe earlier this year I saw both sides of this - I had a recruiter ghost me for almost a month and a number of times important information about interviews got lost in an email chain. It was about as messy as I would expect from any company that size trying to arrange interviews.
That said, the interview process itself, once it was set up, was one of the best interview processes that I've participated in. The PDF they send about the questions and scope of each interview helped me know how to prepare. Each interview had a specific goal and angle on determining if I was a good engineer, and at the end of the day, I felt like we all had a good idea of each other. I ultimately didn't get the offer, but because the process was robust, I didn't have a regret of "I could have done better" or "These questions were unfair and I should have gotten the job". I was confident that Stripe had a good measure of my skills and mindset, and that they'd simply decided I wasn't a good fit.
It's hard to really ask for more than than from the process itself. I can't say if it's effective for them, but at least to me, it seemed like Stripe had thought more about the interview process than most companies I've ever been with or interviewed with. There's obviously going to be implementation issues and problems around the specific people who have to implement it, but the plan itself seems well conceived.
That said, the interview process itself, once it was set up, was one of the best interview processes that I've participated in. The PDF they send about the questions and scope of each interview helped me know how to prepare. Each interview had a specific goal and angle on determining if I was a good engineer, and at the end of the day, I felt like we all had a good idea of each other. I ultimately didn't get the offer, but because the process was robust, I didn't have a regret of "I could have done better" or "These questions were unfair and I should have gotten the job". I was confident that Stripe had a good measure of my skills and mindset, and that they'd simply decided I wasn't a good fit.
It's hard to really ask for more than than from the process itself. I can't say if it's effective for them, but at least to me, it seemed like Stripe had thought more about the interview process than most companies I've ever been with or interviewed with. There's obviously going to be implementation issues and problems around the specific people who have to implement it, but the plan itself seems well conceived.
I had an engineer leave my team (regrettable attrition, one of the best engineers I worked with) and accept a position at Stripe. I'm reading your description and impression of the interview process and it 100% matches what he told me about his experience. He managed to convince me to consider Stripe as my next career move.
I hate to pile on but I absolutely agree. It's always curious to me that I have seen multiple posts on HN parroting how great Stripe's recruiting is, when my experience was lackluster at best, and a shit show at worst. As an outsider, "effort" and "thoughtfulness" would have been the last words I would have used to describe Stripe's recruiting process.
When I interviewed with them. I got the same amount of long periods of no contact and people showing up late (if showing up at all) to interview calls. The things the recruiter told me about the role and interview process completely contradicted the things that the team members told me. I was told that the third interview would be a coding interview, and then when the interviewer showed up (late to the interview, of course), it turned out I had prepared for nothing and the interview was actually just a behavioral.
They weren't even able to tell me if the role was pre-sales (and commission based) or if the role was post-sales (and not commission based). The leadership team had apparently not decided yet, but still wanted to hire people for this role. This is a pretty fundamental characteristic about a job, and I honestly don't know how you can recruit for a position without even knowing the basic compensation structure.
In the end, the recruiter told me that they wanted to move forward with scheduling the final round, then rescheduled my interview several times, then cancelled it outright (but assured me they still wanted to interview once they found a good time), and then eventually the recruiter totally ghosted me without scheduling the interview and never responded to further emails.
When I interviewed with them. I got the same amount of long periods of no contact and people showing up late (if showing up at all) to interview calls. The things the recruiter told me about the role and interview process completely contradicted the things that the team members told me. I was told that the third interview would be a coding interview, and then when the interviewer showed up (late to the interview, of course), it turned out I had prepared for nothing and the interview was actually just a behavioral.
They weren't even able to tell me if the role was pre-sales (and commission based) or if the role was post-sales (and not commission based). The leadership team had apparently not decided yet, but still wanted to hire people for this role. This is a pretty fundamental characteristic about a job, and I honestly don't know how you can recruit for a position without even knowing the basic compensation structure.
In the end, the recruiter told me that they wanted to move forward with scheduling the final round, then rescheduled my interview several times, then cancelled it outright (but assured me they still wanted to interview once they found a good time), and then eventually the recruiter totally ghosted me without scheduling the interview and never responded to further emails.
Commissions and coding? What kind of role was this?
Yes, exactly my confusion. From what little they were able to tell me about the role, it was treated 100% as an engineering role and supposedly followed their engineering interview process (though one of the team members told me the role actually required very little programming), but was technically part of the sales team.
Sales engineer by the sound of it
Please, they prefer to be identified as "solutions architects".
I don't know why you put that in scare-quotes. Solutions Architect is a common role in large software companies, and different than Sales Engineer.
I'll agree that it was better than most.
I still feel like they wasted my time. I told them my compensation expectations up front, went through the entire interview process, and then got an "non-negotiable" offer about 30% under what I told them.
They tried to justify it based on a very optimistic valuation of their equity. I'm already working at a unicorn that skimps on salary with the promise of the equity being worth something, and it would have been a significant cut in take-home pay to accept their offer.
I still feel like they wasted my time. I told them my compensation expectations up front, went through the entire interview process, and then got an "non-negotiable" offer about 30% under what I told them.
They tried to justify it based on a very optimistic valuation of their equity. I'm already working at a unicorn that skimps on salary with the promise of the equity being worth something, and it would have been a significant cut in take-home pay to accept their offer.
The bar for "recruiting" is laughably low in this industry. Many companies seriously consider "have a clueless tech sourcer 2 years out of an econ degree telemarket job openings with aggressive email spam funnel right into a hazing-style round of phone interviews and on-sites" to be "recruiting."
It's very tough to take seriously the bemoaning about engineering hiring difficulty from the likes of pg and others when this is the case.
To the extent companies are investing in the process, it seems to be look for more ways at how to reject people. Now companies want some technical project phone screen AND an algorithms on site AND a "design" review AND some sort of weird personality thing where they grill you on working with past managers and whether you mentored people. Find 99 reasons to ding you. By the way, the design review is usually run by someone 2 years out of school who works on their Jenkins configs and thinks the most important thing is that you magically read their mind and know the right "questions" to ask them.
It's just barely tangentially related to actually working in the industry, which has the result it's actually biased against senior engineers, because ironically the standard for being hired as a senior engineer is actually being better at these interview games that juniors have more recently spent time preparing for.
A lot of companies quite frankly look at what Google did, think "we're just as good as Google", and more or less copy what they do except add even more junk. These companies don't spend much of any time getting to know candidates, they will run some "events" that appeal to people with excessive free time, again mostly juniors. They haven't invested a hundredth of the massive PR push that Google has (Google literally got a box-office movie with Owen Wilson made to recruit), and they CERTAINLY don't want to offer Google-style RSU compensation packages.
After all this "we have the best people" crap, "we have this rube golderberg style interview to filter out all the riff-raff" crap, they want to offer market salaries and some weak ISO package you will have zero chance of valuing accurately and leave 18 windows open for VCs to screw you.
So, given all this, should you waste your time interviewing at companies like Stripe?
It's very tough to take seriously the bemoaning about engineering hiring difficulty from the likes of pg and others when this is the case.
To the extent companies are investing in the process, it seems to be look for more ways at how to reject people. Now companies want some technical project phone screen AND an algorithms on site AND a "design" review AND some sort of weird personality thing where they grill you on working with past managers and whether you mentored people. Find 99 reasons to ding you. By the way, the design review is usually run by someone 2 years out of school who works on their Jenkins configs and thinks the most important thing is that you magically read their mind and know the right "questions" to ask them.
It's just barely tangentially related to actually working in the industry, which has the result it's actually biased against senior engineers, because ironically the standard for being hired as a senior engineer is actually being better at these interview games that juniors have more recently spent time preparing for.
A lot of companies quite frankly look at what Google did, think "we're just as good as Google", and more or less copy what they do except add even more junk. These companies don't spend much of any time getting to know candidates, they will run some "events" that appeal to people with excessive free time, again mostly juniors. They haven't invested a hundredth of the massive PR push that Google has (Google literally got a box-office movie with Owen Wilson made to recruit), and they CERTAINLY don't want to offer Google-style RSU compensation packages.
After all this "we have the best people" crap, "we have this rube golderberg style interview to filter out all the riff-raff" crap, they want to offer market salaries and some weak ISO package you will have zero chance of valuing accurately and leave 18 windows open for VCs to screw you.
So, given all this, should you waste your time interviewing at companies like Stripe?
I previously worked at Stripe. Mark's post mirrors my experience internally as a hiring manager, and as a candidate. But as you said, at scale things get harder; I'm sure the experience does vary and some percentage of people come out with experiences that don't match the ideal.
My own experience was that Stripe was able to take me from a prospect to a hire in less than 30 days. My interviews and calls were all on Friday, and I would hear back on Monday with results and schedule the next round for that very same week.
My own experience was that Stripe was able to take me from a prospect to a hire in less than 30 days. My interviews and calls were all on Friday, and I would hear back on Monday with results and schedule the next round for that very same week.
I work at a far larger company that scaled from a smaller company.
Most people do not show up to interviews late and the process is a 1hr phone screen + 1 day onsite + 30m decision meeting. From start to end, if the candidate was available, took about 1 week. That part is not a scaling issue, but a cultural issue where it's OK to be late or flake on interviews.
Most people do not show up to interviews late and the process is a 1hr phone screen + 1 day onsite + 30m decision meeting. From start to end, if the candidate was available, took about 1 week. That part is not a scaling issue, but a cultural issue where it's OK to be late or flake on interviews.
30 days seems exceptionally long, unless that was driven by your timeline?
Given I had a job while interviewing, I was driving it as fast as I was comfortable. But the part that was important to me as a candidate is that I was getting feedback within 1 business day, and getting the next step scheduled for later that same week.
The reading didn't seem to over-emphasize greatness, it simply raised the prospect that they may be doing more things right than the average Sylicon company.
That may or may not be true and a single annecdotal evidence brought fourth based on a very specific actions that have led a single individual to feel uncomfortable is hardly enough to downplay the points that have been raised.
That may or may not be true and a single annecdotal evidence brought fourth based on a very specific actions that have led a single individual to feel uncomfortable is hardly enough to downplay the points that have been raised.
> When considering ideas, we think “how might it work?” is more interesting than “why will it fail?”
Engineering involves critical thinking. You should have a firm belief that what you wish to create can be created. However, that is simply a foundation upon which to throw as many "why will it fail?" permutations as you can generate. This isn't pessimism, it's the culling of error, like a sculpture from a block of marble.
Engineering involves critical thinking. You should have a firm belief that what you wish to create can be created. However, that is simply a foundation upon which to throw as many "why will it fail?" permutations as you can generate. This isn't pessimism, it's the culling of error, like a sculpture from a block of marble.
Some clarification on the "why will it fail?" question. It's different and less constructive than "how will it fail?" which asks for failure modes and mitigations. It's a superior discussion to "why will it fail?" That said, "why will it fail?" is also inferior to the pre-mortem discussion starter: "why did it fail?"
Agreed. I recently read Fernando Corbato's Turing Award speech, "On Building Systems That Will Fail" [0] Corbato drove development of Multics and its predecessor, CTSS, laying the foundation for much of current operating systems practice.
He states his main thesis as 'the question to ask when designing such systems is not: "if something will go wrong, but when will it?"' I do not think he asks this in a defeatist sense (look at what he built!), but in a realist sense. He claims that, for a sufficiently complicated system built on new ideas, that neither rigor in design nor thoroughness in testing is sufficient to account for all of the unintended consequences of the system's design. Without considering the negative consequences of new designs, one is apt to learn them by hard experience.
[0] http://larch-www.lcs.mit.edu:8001/~corbato/turing91/
He states his main thesis as 'the question to ask when designing such systems is not: "if something will go wrong, but when will it?"' I do not think he asks this in a defeatist sense (look at what he built!), but in a realist sense. He claims that, for a sufficiently complicated system built on new ideas, that neither rigor in design nor thoroughness in testing is sufficient to account for all of the unintended consequences of the system's design. Without considering the negative consequences of new designs, one is apt to learn them by hard experience.
[0] http://larch-www.lcs.mit.edu:8001/~corbato/turing91/
The difference being having a more positive/helpful/constructive mindset.
With "how might it work" you're proposing solutions to the potential "why will it fail" problems rather than dumping more obstacles in the way.
"this will work if we do X because it will fail if Y"
vs
"this isn't going to work because Y"
I think the prior would be a more pleasant experience.
With "how might it work" you're proposing solutions to the potential "why will it fail" problems rather than dumping more obstacles in the way.
"this will work if we do X because it will fail if Y"
vs
"this isn't going to work because Y"
I think the prior would be a more pleasant experience.
It's just good risk management to ask the "why will it fail" question to find out the failure boundary of a system. Know where and when the breaking points are give everyone more confident in the system.
There's a context-dependent trade-off between analyzing, using the currently available information and what can be reasonably be deduced from that, and experimenting.
One criticism I've heard from former Stripe employees is that many of the people in high positions are Stripe are young with no management training, especially in the non-engineering parts of the company. So it sets up for a lot of problems, and cultural problems because they don't know how to manage people or large organizations well.
Sounds like basically every successful startup in Silicon Valley.
Certainly I heard the same from friends at FB and Google back in the day.
Certainly I heard the same from friends at FB and Google back in the day.
I'm impressed that Stripe has been able to maintain it's culture over the years, as it grows in headcount and size. I recall reading this blog from 2012 (although I read it in 2014): https://blog.alexmaccaw.com/stripes-culture
And the consistency between then and now is fairly impressive. Ambition and Optimism seem like codified values that have emerged over time. But the one thread that is consistent is their thoughtfulness on hiring.
It should be no surprise to anyone here though, given that their interviewing doesn't consist of the standard 4-5 leetcode questions, but rather much more thoughtful rounds like a bug squash, architecture, and lots of hands on coding - algorithm rounds exist, but kept to a minimum.
Not to say that this is good and should be the new standard - just that it is extremely thoughtful and was going against the grain. Wish more companies would think about their hiring practices from first principles and didn't just copy Google
And the consistency between then and now is fairly impressive. Ambition and Optimism seem like codified values that have emerged over time. But the one thread that is consistent is their thoughtfulness on hiring.
It should be no surprise to anyone here though, given that their interviewing doesn't consist of the standard 4-5 leetcode questions, but rather much more thoughtful rounds like a bug squash, architecture, and lots of hands on coding - algorithm rounds exist, but kept to a minimum.
Not to say that this is good and should be the new standard - just that it is extremely thoughtful and was going against the grain. Wish more companies would think about their hiring practices from first principles and didn't just copy Google
Unless you actually work there, how are you so confident that Stripe's actual culture has been "maintained" outside of reading company sanctioned blog posts/think pieces...? Or that the actual culture has ever even closely mirrored the one that has been outwardly portrayed?
Glassdoor is not perfect, but I think I get more insight into the reality on the ground looking through recent reviews than a blog post, and at least from what I can see recent reviews over the last few months are extremely mixed.
Glassdoor is not perfect, but I think I get more insight into the reality on the ground looking through recent reviews than a blog post, and at least from what I can see recent reviews over the last few months are extremely mixed.
Stripe's Glassdoor profile consistently perplexes me. For a company that large that has been around as long as it has to only have 81 reviews is very strange. There is no easy way with Glassdoor to get a true measure of # reviews per total active FTEs per year but it feels much lower than other companies that are both younger and smaller.
With a staff level under a few hundred, it isn't that strange. People only tend to complain more about negative experiences vs. positive and the internet lurker rule (%99 lurk, %1 write content) explains the rest of the numbers.
How does one build a strong startup culture like Stripe? I'm looking to strike out on my own, and while I know there are other fundamental concerns I should be worried about, this is something that I continue to dwell on. A strong team can make or break a company.
I'd appreciate any advice or reading material.
I'd appreciate any advice or reading material.
I don't know if this is right, but from the outside, it looks like a company where the tech team has the most seats at the table.
Versus sales, or some separate product management team running the show. That model seems to work well with what Stripe does. It probably doesn't work well in many other niches.
Versus sales, or some separate product management team running the show. That model seems to work well with what Stripe does. It probably doesn't work well in many other niches.
> Reading material
Patrick Lencioni has written a few good books on company culture. Some of them are in the "business novel" style of The Phoenix Project and Critical Chain, which has the benefit of working well in audiobook format. If you prefer more straightforward nonfiction, then read The Advantage.
Patrick Lencioni has written a few good books on company culture. Some of them are in the "business novel" style of The Phoenix Project and Critical Chain, which has the benefit of working well in audiobook format. If you prefer more straightforward nonfiction, then read The Advantage.
check out
Lecture 11 - Hiring and Culture, Part 2 (Patrick and John Collison, Ben Silbermann) https://www.youtube.com/watch?v=H8Dl8rZ6qwE
and perhaps https://www.infoq.com/presentations/stripe-culture-2016/ though that's more corporate.
Lecture 11 - Hiring and Culture, Part 2 (Patrick and John Collison, Ben Silbermann) https://www.youtube.com/watch?v=H8Dl8rZ6qwE
and perhaps https://www.infoq.com/presentations/stripe-culture-2016/ though that's more corporate.
Make money.
Most problems in business are not problems if there is enough money to go around.
By figuring out how to make money, you solve most other problems along the way.
Most problems in business are not problems if there is enough money to go around.
By figuring out how to make money, you solve most other problems along the way.
Optimism I believe does have a very positive effect on both productivity of the people and how much they love working in a place. But in my experience overdoing it might make your employees start developing this "This does not make sense at all, why are these people living in dreams?" mindset.
From chats I had, I could feel some of my colleagues also had a hesitation towards the goals etc. It was interesting to see people become frustrated especially if they have been used to being slightly pessimistic and cautious due to their background(title(Finance background), nationality(mostly non-US and UK people) etc.)
Overall very nice article though, I do believe moving fast in hiring is the most yield/effort you can do to hire great people. Best companies I worked for(above one included) moved extremely fast in hiring, worst take 2-3 weeks just to reply to an email let alone scheduling an onsite. Wish decision makers in recruiting knew this.
p.s. love the Stripe Docs and Technical blog posts, this one specifically was very informative: https://stripe.com/en-de/blog/api-versioning
Overall very nice article though, I do believe moving fast in hiring is the most yield/effort you can do to hire great people. Best companies I worked for(above one included) moved extremely fast in hiring, worst take 2-3 weeks just to reply to an email let alone scheduling an onsite. Wish decision makers in recruiting knew this.
p.s. love the Stripe Docs and Technical blog posts, this one specifically was very informative: https://stripe.com/en-de/blog/api-versioning
I have seen a lot more problems come from an excess of optimism, than an dearth of it. You should not be so optimistic that you don't think about how to roll back changes if they turn out to be a bad idea, for example. Backups and rollback plans and code reviews and etc. are all fundamentally pessimistic. An excess of optimism can lead to no plan B. Every startup pivot relied upon NOT having so much optimism that you will not believe a pivot is required.
Not to say that optimism is all bad, but it's not all good either, and in the U.S. corporate world I see a lot more problems from an excess of optimism than from the reverse.
Not to say that optimism is all bad, but it's not all good either, and in the U.S. corporate world I see a lot more problems from an excess of optimism than from the reverse.
I had the pleasure of working with Mark for ~3.5 years at Heroku just prior to him joining Stripe. Since leaving Stripe, he’s been working on research projects at Ink & Switch, one of which they recently chose to focus on as a stand-alone business: https://museapp.com/
Most values and keeping the team healthy, reflects the vision of founders I suppose.
... and why have you left stripe ??? :/
You can feel good about a workplace and still leave (need a change, better opportunities elsewhere etc)
Not directed at author but I looked at his bio after reading the article. He spent his time mostly as an engineering manager and yet in his bio has:
"I led engineering teams at Stripe"
Do managers in software companies actually think they lead teams? I've seen or heard "led engineering" or variants so much from mid level management that I wonder if my perspective is skewed.
Seems crazy to me.
"I led engineering teams at Stripe"
Do managers in software companies actually think they lead teams? I've seen or heard "led engineering" or variants so much from mid level management that I wonder if my perspective is skewed.
Seems crazy to me.
Two reasons it shouldn’t sound crazy:
1. Just grammatically, managers need some verb to use. If he said “I managed Engineering,” it would be really difficult to argue that isn’t what an engineering manager does. But good managers lead, so they prefer the word lead over manage, even though the meaning in context is quite similar.
2. You seem to be implying that managers suck and don’t know anything about engineering, so how could they claim leadership?
Speaking from my career path, most eng managers come out of engineering and are pretty technical. At Atlassian (where I’m a front-line Engineering Manager), you have to earn your way to Senior Engineer before you can join the management track, and the managers for several steps up the ladder are accountable for technical outcomes.
It’s very common in our office for the front line managers (aka Team Leads) to lead whiteboarding and pair a lot during the early phases of a project to make sure it’s built on a sound design, and to be similarly involved throughout projects to ensure they’re tracking to completion on schedule and hitting our quality bar.
My understanding is Stripe works very similarly. So, yeah, I buy that he “led” engineering teams there.
1. Just grammatically, managers need some verb to use. If he said “I managed Engineering,” it would be really difficult to argue that isn’t what an engineering manager does. But good managers lead, so they prefer the word lead over manage, even though the meaning in context is quite similar.
2. You seem to be implying that managers suck and don’t know anything about engineering, so how could they claim leadership?
Speaking from my career path, most eng managers come out of engineering and are pretty technical. At Atlassian (where I’m a front-line Engineering Manager), you have to earn your way to Senior Engineer before you can join the management track, and the managers for several steps up the ladder are accountable for technical outcomes.
It’s very common in our office for the front line managers (aka Team Leads) to lead whiteboarding and pair a lot during the early phases of a project to make sure it’s built on a sound design, and to be similarly involved throughout projects to ensure they’re tracking to completion on schedule and hitting our quality bar.
My understanding is Stripe works very similarly. So, yeah, I buy that he “led” engineering teams there.
> But good managers lead, so they prefer the word lead over manage, even though the meaning in context is quite similar.
I would actually argue that good managers _support_ their teams, they don't lead them.
A tech lead or architect should be leading the engineering team, and the manager should be supporting the team's efforts and clearing the path for them.
I would actually argue that good managers _support_ their teams, they don't lead them.
A tech lead or architect should be leading the engineering team, and the manager should be supporting the team's efforts and clearing the path for them.
I don't think it's uncommon to have a 'lead of team leads' called an engineering manager. They're typically the people who were good enough at being team leads that they advance to making sure all the team leads are good at it too.
Mark led engineering teams at Stripe. That is a fact that your assumptions cannot disprove.
Do you qualify the person with the most merged PR's as the leader of the engineering team then?
Someone has to have the vision and the strength to keep things outside of that vision out of the scope of work and know how to say `No`.
Someone has to have the vision and the strength to keep things outside of that vision out of the scope of work and know how to say `No`.
Yes? Why would you think otherwise?
I think his (somewhat murky) point was the there is a frequently a distinction between 'managing' and 'leading' a team, particularly a technical one.
Managing is generally considered a role that involves HR/scheduling/career development type tasks, i.e. soft skills.
Leading, alternatively, is perceived as a harder technical role. Sometimes involving overall architecture and design depending on the role. More hard skills.
Managing is generally considered a role that involves HR/scheduling/career development type tasks, i.e. soft skills.
Leading, alternatively, is perceived as a harder technical role. Sometimes involving overall architecture and design depending on the role. More hard skills.
It would be difficult to be an effective engineering manager if one was not technical. Otherwise any people manager could be an engineering manager. Engineering managers don't necessarily get their hands dirty, but they need to understand what's really going on with their teams and their progress in order to make informed decisions.
> "Ambitious problems for Stripe look like enabling more internet businesses and supporting entrepreneurs in more countries, not getting more ad clicks."
I used to have a similar mindset until I actually ended up working on Ads at Google. There are hundreds of thousands (if not millions) of small business merchants, mom and pop stores, etc that rely on ads (be that Google, Facebook, Instagram, whatever) to run their businesses. In fact, these are the vast majority of ad buyers! The huge multi-billion dollar corporations are far and few between.
That's what drove many of the engineers (including myself). A small increase in conversion ratio (be that through better ranking, filtering, query interpretation, etc) meant a huge deal to these merchants and in some cases, could be the difference between life and death for some of these small companies. And at the scale Google operates at, very small incremental changes = huge impact. The engineers and projects, at least in my team, were driven by exactly the same goals mentioned here - to "enable more businesses" and "support entrepreneurs" all across the world.
It's easy to get sucked into the hype and blanket statements of "ads are bad" (I was guilty of that too). I've since left Google, but that experience really internalized for me that things like this are never so black and white.