Airbnb Engineering(airbnb.io)
airbnb.io
Airbnb Engineering
http://airbnb.io/
63 comments
Seems like it is getting to be a trend to write your own workflow engine and open source it. Spotify has luigi, LinkedIn has Azkaban and Airbnb has Airflow. I wonder at what market capitalization companies start rolling their own language like .net, go and swift?
How is that in any way surprising? The more resources you have, the more custom-tailored you can make your tools. And I can't imagine it's a shock that data workflows aren't a completely-solved-for-all-usecases problem at this point...
Don't forget Chronos.
http://mesos.github.io/chronos/
http://mesos.github.io/chronos/
Isn't Chronos more of a distributed scheduler ("fault tolerant cron") than a workflow engine?
Distributed job scheduler is, in a larger sense, a workflow engine. It will need to understand what is the next job to trigger, handle different situation where job fails\timeouts and do the retry if specified.
which was also started at Airbnb.
drake[0] from factual falls into this category as well.
[0] - https://github.com/Factual/drake
[0] - https://github.com/Factual/drake
Airflow reminded me a lot of taskflow from the OpenStack project.
[deleted]
Hey, isn't a DSL your own language, strictly speaking? So any internal company-specific XML schema etc...
"Engineering Culture" photo makes me think of a call center. Why would I want to do something as brain intensive and doesn't do well with disruption as software engineering in the environment depicted?
I'm fine with down voting, I still ask the question, how many engineers would be happy with the closeness of workstations portrayed? As someone who was reached out to by a couple of recruiters for companies w/ similar setups in the last couple of days, I don't see this as a selling point. Maybe it is just an image, but down votes and no comments to contrary say a lot.
No, you don't get it, how else are people supposed to exchange ideas and make the synergy and lots of the serendipity if they aren't thrown into a room with a bunch of cafeteria tables? It's not like people have other ways of communicating these days. <-- That is sarcasm HN.
I'm right there with you jmspring. I would not be happy in that environment and I happily pass on each and every company with floor plans that would make my job harder.
I'm right there with you jmspring. I would not be happy in that environment and I happily pass on each and every company with floor plans that would make my job harder.
> I'm fine with down voting
Wow. So brave!
Wow. So brave!
I'd like to understand why as knowledge workers a portrayed environment where you have minimal space between you and an open environment it a good working environment? The photo used by the AirBnB add portrays an extreme of the popular open office concept.
In the last two years I've worked in "open office as portrayed", a more isolated open office, as well as at home. I know my preference.
I still ask why is this a motivator on the cover of the digital equivalent of a glossy magazine?
This has nothing to do with the company (I use airbnb regularly), but the portrayal of environments.
A justification, I get emails regularly, I have no commute. The recruiters that reach out to me I simply ask, "so, I have no commute, can you encourage me to further engage with that first hurdle." So far, none can. Simple question, I've got a great job, sell me on what you have. Reality is most established companies at this point just want bodies to fill positions.
Disclaimer - I've passed on two founding CTO positions (over the last 2-3 years) because my risk averseness wasn't in sync at the time asked.
I see this post as a recruiting glossy and asked my questions as such.
In the last two years I've worked in "open office as portrayed", a more isolated open office, as well as at home. I know my preference.
I still ask why is this a motivator on the cover of the digital equivalent of a glossy magazine?
This has nothing to do with the company (I use airbnb regularly), but the portrayal of environments.
A justification, I get emails regularly, I have no commute. The recruiters that reach out to me I simply ask, "so, I have no commute, can you encourage me to further engage with that first hurdle." So far, none can. Simple question, I've got a great job, sell me on what you have. Reality is most established companies at this point just want bodies to fill positions.
Disclaimer - I've passed on two founding CTO positions (over the last 2-3 years) because my risk averseness wasn't in sync at the time asked.
I see this post as a recruiting glossy and asked my questions as such.
Because we aren't respected the way other knowledge workers are. Look at the abuse heaped on us by the press or the anti-tech worker protests in SF.
Shut up "coder" and get back to work. You don't need a private office. After all, you aren't a doctor, a lawyer, or even an MBA. Also, don't forget about all of those stock options we gave you in lieu of a market-rate salary; those could potentially make you a millionaire! Quit your whining.
Something tells me that photo makes it look worse than it is because it was taken with a long lens.
What I don't get about these kinds of jobs is, its already built. Its already scaled. You already created the architecture.
How interesting could those kinds of jobs really be?
And if you really need or want world class talent, why would you limit the pool to people who want to commute into San Francisco?
How interesting could those kinds of jobs really be?
And if you really need or want world class talent, why would you limit the pool to people who want to commute into San Francisco?
I remember that an engineer we tried to hire at Google in 2000 turned down the offer for the same reason -- he thought that all of the interesting problems were probably already solved :)
The reality is typically the opposite. Most startups are technically straightforward to start with and only get really advanced with scale.
The reality is typically the opposite. Most startups are technically straightforward to start with and only get really advanced with scale.
Exactly. Currently working a project that started out as a simple django application with Mongo db (don't ask me, wasn't my decision). It used Parse, heroku, and some other platforms. I'm in the process of building what will replace parse (file storage) using Go, Redis, postgre, and nginx. Then when in done with that, the main api will get turned over to the same basic architecture, though with other parts to balance load, etc. This is more complex because now we have to do a lot more maintenance. Which means that more code will get written to automate things. More programs talking to each other and more different parts moving at different speeds. I love it, but it's complex. And then you realize you are in charge of doing it all. :)
That does sound like exciting greenfield though; sounds like a refactoring/scale out project.
It is exciting! I'm in charge of scaling a business to support an explosion of users. I say business because I'm also building / extending the sale & marketing systems, the internal communications, and some other stuff I have planned out. It's a lot of work but we need it and good tools help me move quickly. I love scaling businesses as a whole.
Most startups are not Google.
True, but most startups don't get as far as AirBNB have, either.
I don't work for Airbnb, but most top notch engineering organizations I know have far more unsolved problems than solved problems.
Each additional magnitude of scale also brings a new wave of engineering challenges and opportunities.
I'd be shocked if this didn't apply to Airbnb. This is just as much a teaser of what's to come as it is a shrine to what 's been built.
(I personally have don't have the SF requirement nor any comment on that - Kaggle fully supports and promotes a distributed workforce)
Each additional magnitude of scale also brings a new wave of engineering challenges and opportunities.
I'd be shocked if this didn't apply to Airbnb. This is just as much a teaser of what's to come as it is a shrine to what 's been built.
(I personally have don't have the SF requirement nor any comment on that - Kaggle fully supports and promotes a distributed workforce)
There are obvious huge ideas that AirBNB is not close to tackling yet.
Right now, you essentially have to be a part time hotelier if you're renting out one or two AirBNB room and a full time hotelier if you're renting out more. The intersection of people who both have room free and are willing to be hoteliers is pretty narrow. I know people who go on vacations of 3+ months at a time with their rooms empty and leave $10,000 of revenue off the table simple because it would be too much hassle to figure out how to effectively AirBNB or craigslist the room.
What I want is a magical team of professionals who enter my house an hour after I've left for vacation to find my house in whatever messy state it's in. They clean, tidy, professionalize and convert my house into something that's rentable to other people and I don't have to think about my house while I'm away, everything is handled for me. An hour before I get home from my vacation, that same team of professionals restores my house back to exactly the way it was. I'd even be willing to split the revenue 70/30 AirBNB's way if they could handle this because the alternative is $0 for me.
Moving up the value chain and taking over more of the travel experience is also something AirBNB is uniquely capitalized to do. AirBNB is experimenting with this with it's neighbourhoods feature and tentatively trying to become a more complete travel solution. But you could imagine AirBNB being an ad platform for other travel service providers for example. aka: pre-commit to enough museum tickets/sports games/restaurant reservations etc. before you arrive in town and we'll throw in the room for free.
Alternatively, AirBNB could start owning more of the travel vertical. You could imagine AirBNB airport shuttles that drop you off at your AirBNB door (with your key and Wifi credentials provided by the shuttle) if density in a region gets high enough.
It doesn't take much to imagine how much is left to be built of AirBNB as well as pretty much any other technology company. Talk to anyone in any company and they'll tell you they're still only 10% of the way there.
Right now, you essentially have to be a part time hotelier if you're renting out one or two AirBNB room and a full time hotelier if you're renting out more. The intersection of people who both have room free and are willing to be hoteliers is pretty narrow. I know people who go on vacations of 3+ months at a time with their rooms empty and leave $10,000 of revenue off the table simple because it would be too much hassle to figure out how to effectively AirBNB or craigslist the room.
What I want is a magical team of professionals who enter my house an hour after I've left for vacation to find my house in whatever messy state it's in. They clean, tidy, professionalize and convert my house into something that's rentable to other people and I don't have to think about my house while I'm away, everything is handled for me. An hour before I get home from my vacation, that same team of professionals restores my house back to exactly the way it was. I'd even be willing to split the revenue 70/30 AirBNB's way if they could handle this because the alternative is $0 for me.
Moving up the value chain and taking over more of the travel experience is also something AirBNB is uniquely capitalized to do. AirBNB is experimenting with this with it's neighbourhoods feature and tentatively trying to become a more complete travel solution. But you could imagine AirBNB being an ad platform for other travel service providers for example. aka: pre-commit to enough museum tickets/sports games/restaurant reservations etc. before you arrive in town and we'll throw in the room for free.
Alternatively, AirBNB could start owning more of the travel vertical. You could imagine AirBNB airport shuttles that drop you off at your AirBNB door (with your key and Wifi credentials provided by the shuttle) if density in a region gets high enough.
It doesn't take much to imagine how much is left to be built of AirBNB as well as pretty much any other technology company. Talk to anyone in any company and they'll tell you they're still only 10% of the way there.
Believe me, I wish that were the case.
The truth is that every year is way bigger (and thus harder) than the previous year. We have a graph that shows traffic over time vs response time over time, and thankfully those lines go in opposite directions, but it's only due to the continuous work of dozens of engineers as we discover new ways that large amounts of transactional traffic and data can play mischievously together. And while doing that we also have to move the needle on product: if you look at a screenshot of Airbnb two years ago and compare it to today, it's very different (especially on the native apps and mobile web). And the screenshot you'd be looking at is probably just on the guest side, which is simpler: on the host side, we've done even more that's invisible to the casual traveler, like automatic pricing recommendations based on geographic and seasonal data, major investments into fraud detection and prevention, global currency exchange, etc. Not to mention all of the tooling, monitoring, and alerting that needs to happen to support all of that growth.
None of this stuff is free and usually it requires rethinking how old systems work and interact with each other, especially as we do more complex calculations with larger amounts of data. To be honest I'd say we're generally about a year away from hitting the wall, every year. We've made a lot of improvements (when I joined almost four years ago we were maybe... two months away from hitting a wall? Super scary), but if we all just phoned it in the growth in usage would overwhelm us quickly.
Re: San Francisco — yeah, it's definitely tough to recruit out of a single geographic pool. SF is probably the best area to do it in, since there's a large cluster of really great engineers here, but there's no doubt we miss great talent that won't or can't relocate. It takes a lot of work to do remote engineering offices well: how do you make sure that the engineers in satellite offices feel connected to HQ? How do you make sure they work on meaningful projects, but also don't miss out on context and information that comes from humans naturally chatting to the people that are around them? These aren't unsolvable problems (and a number of more-established, older companies have already solved them to some degree, and even some startups that are trying the Moneyball-style approach), but as a team we haven't done it yet. We do have remote non-engineering offices, so it's not impossible to imagine some day having a distributed eng team, but today we're just in SF. Some day, maybe? It takes work but there's certainly payoff for extending your reach.
The truth is that every year is way bigger (and thus harder) than the previous year. We have a graph that shows traffic over time vs response time over time, and thankfully those lines go in opposite directions, but it's only due to the continuous work of dozens of engineers as we discover new ways that large amounts of transactional traffic and data can play mischievously together. And while doing that we also have to move the needle on product: if you look at a screenshot of Airbnb two years ago and compare it to today, it's very different (especially on the native apps and mobile web). And the screenshot you'd be looking at is probably just on the guest side, which is simpler: on the host side, we've done even more that's invisible to the casual traveler, like automatic pricing recommendations based on geographic and seasonal data, major investments into fraud detection and prevention, global currency exchange, etc. Not to mention all of the tooling, monitoring, and alerting that needs to happen to support all of that growth.
None of this stuff is free and usually it requires rethinking how old systems work and interact with each other, especially as we do more complex calculations with larger amounts of data. To be honest I'd say we're generally about a year away from hitting the wall, every year. We've made a lot of improvements (when I joined almost four years ago we were maybe... two months away from hitting a wall? Super scary), but if we all just phoned it in the growth in usage would overwhelm us quickly.
Re: San Francisco — yeah, it's definitely tough to recruit out of a single geographic pool. SF is probably the best area to do it in, since there's a large cluster of really great engineers here, but there's no doubt we miss great talent that won't or can't relocate. It takes a lot of work to do remote engineering offices well: how do you make sure that the engineers in satellite offices feel connected to HQ? How do you make sure they work on meaningful projects, but also don't miss out on context and information that comes from humans naturally chatting to the people that are around them? These aren't unsolvable problems (and a number of more-established, older companies have already solved them to some degree, and even some startups that are trying the Moneyball-style approach), but as a team we haven't done it yet. We do have remote non-engineering offices, so it's not impossible to imagine some day having a distributed eng team, but today we're just in SF. Some day, maybe? It takes work but there's certainly payoff for extending your reach.
[deleted]
So AirBnB has hit a lot of high notes in a really short span of time, but just because their product rocks today doesn't mean it will stay that way forever. You still need a product and engineering team for coming up with fresh ideas to stay relevant.
Maybe its AirBnB's primary product, but it made me think of the old Mapquest vs Google Maps "war" and how Google Maps eventually won out with continuously improving on the core product, even thinking waaay outside the box by strapping cameras on top of moving cars to construct a huge VR map of the world. I'm sure when Mapquest was new most people were thinking it was already a perfect product for what it did.
Maybe its AirBnB's primary product, but it made me think of the old Mapquest vs Google Maps "war" and how Google Maps eventually won out with continuously improving on the core product, even thinking waaay outside the box by strapping cameras on top of moving cars to construct a huge VR map of the world. I'm sure when Mapquest was new most people were thinking it was already a perfect product for what it did.
[deleted]
There are always a lot of optimisations and improvements to do.. It might not be the most interesting job out there, but I bet there's plenty to see.
I don't know about the rest of their stack, but their Machine Learning team is pretty good. They were using PMML the last time I spoke to them. And a little bit of Scalding as well. Preventing fraud at scale is quite a challenging exercise.
Airbnb clearly has product aspirations beyond home/room/yurt rentals.
A lot of negative reactions here. My impression of engineering at Facebook has been the work ethic seen in Josh Perez's "Alt" Flux library for React.
He's been very responsive and helpful to my own projects in the same problem space.
Maybe lots of people are in better situations than I'd expect, but working at Airbnb on projects similar to that on a platform tons of people use sounds cool to me!
He's been very responsive and helpful to my own projects in the same problem space.
Maybe lots of people are in better situations than I'd expect, but working at Airbnb on projects similar to that on a platform tons of people use sounds cool to me!
FWIW, your "meet the nerds" segment is fantastic, original marketing. I expect it to catch on. Above all one wants interesting, inspiring, admirable coworkers.
"Meet Our Engineers" would've been much better copy.
AirBnB seems to like to refer to themselves as Nerds... it's all over their site.
Maybe they seek other people who prefer that term too. Personally, I write Job Descriptions with the title "Engineer", and refer to my teammates as "devs" or "engs". But if that's their culture that's their culture, no?
Maybe they seek other people who prefer that term too. Personally, I write Job Descriptions with the title "Engineer", and refer to my teammates as "devs" or "engs". But if that's their culture that's their culture, no?
They should call it AirBnE.
Wait, as a quirky nerd that learned BASIC at 5, I can't fathom not wanting to spend the weekend with my peers hacking on projects and yes, just maybe, changing the world. But not before I meme-bomb Larry and start a nerf war! Being an adult is neat.
can't tell if this is brilliant sarcasm
[deleted]
Wow, a weekend hackathon where employees worked all night (https://www.airbnb.com/careers/departments/engineering). Capital is getting smarter every year! Now you can make workers think that they were the ones who decided to do unpaid overtime.
(And if it wasn't "required", don't kid yourself---of course it was required.)
(And if it wasn't "required", don't kid yourself---of course it was required.)
I used to be an engineer at Airbnb. It's not, in fact, the case that people are doing "unpaid overtime". My favorite example of this: there are a great number of people who work there who were MORE than willing to take a free Saturday/Sunday and host their OWN hackathon with their work-friends when the company-supported dates proved inconvienient for a lot of us. While it's true that a lot of the projects were company focused, a great deal were purely "stupid shit nobody needs and general shenanigans" projects. It's not the kind of place where people feel forced into working in a silly way. It's an awesome place full of rather clever people.
/endrant
/endrant
> It's not the kind of place where people feel forced into working in a silly way.
After reading your post, I'm having a hard time seeing how this is the case. Perhaps people are not directly forced into doing weekend "hackathons", but I'd imagine there is quite a bit of indirect pressure to show up for these things for fear of not being seen as a team player. Although, maybe my own personal experiences have just made me too cynical.
After reading your post, I'm having a hard time seeing how this is the case. Perhaps people are not directly forced into doing weekend "hackathons", but I'd imagine there is quite a bit of indirect pressure to show up for these things for fear of not being seen as a team player. Although, maybe my own personal experiences have just made me too cynical.
Do you really not see the power dynamic at play here?
Perhaps it was required, and employees were tricked. But you weren't there and you don't know.
I personally love my craft and often throw myself into it over the weekends, and outside of standard working hours. Since I also care a lot about what my company is working on (I am an employee/fellow, not a founder), that can include job-related work. Nobody at the company is asking me to do it, and I'm not doing it out of pressure or guilt. It's simply more fun for me than many other activities.
A healthy life outside of work and programming is important too. I don't spend every weekend at the keyboard, and I do try to sleep 8 hours a night. When I don't feel like going into the office, I don't. I'm my own boss and if I choose to the contribute to the world through a company I didn't happen to found, but do believe in, that's my prerogative. I don't know anyone at AirBnB, so I don't know, but I imagine the same may well have been true there too.
I personally love my craft and often throw myself into it over the weekends, and outside of standard working hours. Since I also care a lot about what my company is working on (I am an employee/fellow, not a founder), that can include job-related work. Nobody at the company is asking me to do it, and I'm not doing it out of pressure or guilt. It's simply more fun for me than many other activities.
A healthy life outside of work and programming is important too. I don't spend every weekend at the keyboard, and I do try to sleep 8 hours a night. When I don't feel like going into the office, I don't. I'm my own boss and if I choose to the contribute to the world through a company I didn't happen to found, but do believe in, that's my prerogative. I don't know anyone at AirBnB, so I don't know, but I imagine the same may well have been true there too.
All-nighters' worth of unpaid overtime and crowded, noisy, privacy-free open office working conditions that would make anyone who isn't a 20-something long for the days of the once-denigrated cube farm. Really shows how little respected we are.
"Meet the Nerds"
That made me cringe a little. Especially when the rest of the page seemed pretty professional (save for the now industry-standard cartoon icons everywhere).
[deleted]
The "nerds" make 120-200k a year. The 25 year olds that went to Arizona State and studied drinking, getting laid, and "business" make up to 300k a year doing enterprise sales for startups.
The current culture is, "hey thanks NERD for geeking out on that distributed systems BS, really helped me out on my last sales call/commission."
Then the "NERD" gets to wonder what the F to do with their life come their mid 30s, while said sales person is set.
But yeah, let's keep calling ourselves nerds and allowing this to happen.
The current culture is, "hey thanks NERD for geeking out on that distributed systems BS, really helped me out on my last sales call/commission."
Then the "NERD" gets to wonder what the F to do with their life come their mid 30s, while said sales person is set.
But yeah, let's keep calling ourselves nerds and allowing this to happen.
We also call ourselves "geeks," which is also quite bad, and we've done little to stop the media and industry rebranding us as "coders," a job title with little prestige or connotations of authority and professionalism, over our previous titles like "software engineer/developer/architect" or even just "programmer." Collectively, we're really pretty socially inept, aren't we?
"Look how diverse our company is"
Nerds seems to be a well used term there [0,1]. I agree that I'd prefer something else. Maybe they got the same guys that did the new logo to come up with it?
[0] http://nerds.airbnb.com/
[1] https://twitter.com/AirbnbNerds
[0] http://nerds.airbnb.com/
[1] https://twitter.com/AirbnbNerds
[deleted]