Stack Exchange Engineering: How We Built Our Blog(blog.stackexchange.com)
blog.stackexchange.com
Stack Exchange Engineering: How We Built Our Blog
http://blog.stackexchange.com/2015/07/how-we-built-our-blog/
60 comments
They've also lost COMPLEXITY, and this always pays off, even if in roundabout ways that you cannot measure, and even if it's only for the positive effects on the mental-health of the engineers working on this!
Really... modern WordPress is a very complicated beast and keeping all sorts of details about how you need to optimize it, server, caching, db in you head is just no worth it. Yeah, "changing two lines in the nginx configuration" could have worked, but imho simply the cost of keeping in your head the information about what two lines to change is just too large. As it grows, it simply drives you insane by how many technical trivialities you need to keep in your head to keep it working smoothly, when you could instead use that brainpower so solve the really interesting problems.
An the fact that "their marketing team has to use the Github interface and learn Markdown" is also a hidden benefit, if you think about all the subtle frontend bugs that happen when somebody uses the WYSIWYG editor of WP and they end up with a combination of styles that breaks the layout. For any project above a certain size, the "dumb WIM" (What I Mean) interface of Markdown always trumps the WYSIWYG.
Also, the benefit of mostly NOT HAVING TO THINK ABOUT SECURITY AT ALL is freaking HUGE! Developers can simply say "it's the server guys' problem" now. Forget about having to manually audit the code of 3rd party plugins if you really care about security. And forget about updating and realizing something breaks and you have to fix it.
For any fast growing company, keep this in mind: REDUCE COMPLEXITY. ALWAYS. NO MATTER HOW LARGE THE SHORT TERM COST.
Really... modern WordPress is a very complicated beast and keeping all sorts of details about how you need to optimize it, server, caching, db in you head is just no worth it. Yeah, "changing two lines in the nginx configuration" could have worked, but imho simply the cost of keeping in your head the information about what two lines to change is just too large. As it grows, it simply drives you insane by how many technical trivialities you need to keep in your head to keep it working smoothly, when you could instead use that brainpower so solve the really interesting problems.
An the fact that "their marketing team has to use the Github interface and learn Markdown" is also a hidden benefit, if you think about all the subtle frontend bugs that happen when somebody uses the WYSIWYG editor of WP and they end up with a combination of styles that breaks the layout. For any project above a certain size, the "dumb WIM" (What I Mean) interface of Markdown always trumps the WYSIWYG.
Also, the benefit of mostly NOT HAVING TO THINK ABOUT SECURITY AT ALL is freaking HUGE! Developers can simply say "it's the server guys' problem" now. Forget about having to manually audit the code of 3rd party plugins if you really care about security. And forget about updating and realizing something breaks and you have to fix it.
For any fast growing company, keep this in mind: REDUCE COMPLEXITY. ALWAYS. NO MATTER HOW LARGE THE SHORT TERM COST.
Just as an aside to WordPress optimization, so there are two ways to do that. One of them is to analyze content flow (how often are things updated) and traffic flow and then come up with a reasonable strategy. Things like use memcaching for sidebars, separate those out depending whether you need to, then work on caching plugins. The caching plugins need work because they tend to be programmed to allow robots to spider uncached content, which makes no sense to me. Robot traffic on high volume sites is just insane. With a hundred blogs or so on a multisite installation you could easily spend a month doing postmortems and hunting down the least common denominator (oh hello switch_to_blog, why do you use 36 MB of RAM on each call?)
Or, you can slap some ram sticks in the machine, install varnish-cache, write a few rules for it and go play minecraft the other 29 days.
Or, you can slap some ram sticks in the machine, install varnish-cache, write a few rules for it and go play minecraft the other 29 days.
> As it grows, it simply drives you insane by how many technical trivialities you need to keep in your head to keep it working smoothly
For everything else there's Ansible.
We've taken the approach to not type a single command into a live server's console and use CM tools to document and automate. This, along with well written operating documentation can be invaluable.
For everything else there's Ansible.
We've taken the approach to not type a single command into a live server's console and use CM tools to document and automate. This, along with well written operating documentation can be invaluable.
Hi! I'm the Director of Product Marketing at Stack Exchange. It might sound like more work for us to use Github and markdown, but I see it as a huge plus. For non technical employees, this is a great opportunity to learn a new skill and collaborate with the engineering organization. Jon trained a few of us already... so far, so good!
Thanks for writing this...too many of us engineers treat non-technical employees as morons/infants, as if you didn't spend 12+ years of your life learning math and English and aren't able to cope with new text/symbolic systems...Learning Markdown is as easy/difficult as learning a rich-text editor...why do people assume that pressing a button with a cute icon on it is inherently easier than a Markdown convention? At least by learning the few Markdown conventions there are, you get some hands on experience with understanding how literal computer languages have to be...rather than forever treating rich text editors as a magical machine.
God, the hours I've seen co-workers waste, trying to eliminate some errant space in a rich-text-editor-generated hyperlink, by pressing a random sequence of rich-text buttons. A lot of times, they just go straight to the raw HTML to fix things...and if you're already doing that, you're capable of learning Markdown (let's skip the discussion of why editing raw HTML can be even more disastrous).
Kudos to you for being eager to learn new skills...I'm an engineer and a self-taught web dev I wish someone had forced me to learn Markdown much earlier than I had.
God, the hours I've seen co-workers waste, trying to eliminate some errant space in a rich-text-editor-generated hyperlink, by pressing a random sequence of rich-text buttons. A lot of times, they just go straight to the raw HTML to fix things...and if you're already doing that, you're capable of learning Markdown (let's skip the discussion of why editing raw HTML can be even more disastrous).
Kudos to you for being eager to learn new skills...I'm an engineer and a self-taught web dev I wish someone had forced me to learn Markdown much earlier than I had.
Hi! While the change surely was not a huge investment I just do not believe that the benefits justify the costs. Since you wrote a blog post about it, I felt free to question your decisions.
Nevertheless, now that it is done, I hope the new system works well for you. Maybe your team gets used to git for version control in other areas of work, too, which would be a huge win and ultimately prove me wrong ;-)
Nevertheless, now that it is done, I hope the new system works well for you. Maybe your team gets used to git for version control in other areas of work, too, which would be a huge win and ultimately prove me wrong ;-)
I hope it works well too! Maybe I'll write a follow-up blog post in a few months and report back on our progress. In general, I think that marketers today should acquire some basic front-end knowledge and that knowing git and markdown is a good step towards getting there. Coding things like email templates and landing pages without the help of a designer or developer will be huge for us.
mattmanser(2)
I'm the author of the post and the lead on this project. Many of the costs you mention aren't so bad. As mentioned earlier, we actually consider learning GH and Markdown a plus for non-devs. As for comments and writing Python, some pain here, but were ultimately solvable.
Performance also wasn't the only plus here. Closing major security holes, making more of our content and technology more open, and moving to a platform that our devs liked working in are just some of the other wins.
It's too early to say definitively now, but we think the change is probably a good one.
Performance also wasn't the only plus here. Closing major security holes, making more of our content and technology more open, and moving to a platform that our devs liked working in are just some of the other wins.
It's too early to say definitively now, but we think the change is probably a good one.
You haven't broken the podcast RSS feed, thank you. Another large podcast network recently did a redesign and broke all of their feeds, apparently on purpose. Crazy.
However, the podcast feed is impossible to find now if you didn't have it before. Previously I think it was on every podcast post as an RSS link. Now the individual tag RSS links all point to the main feed, rather than to a per-tag feed.
e.g. http://blog.stackoverflow.com/category/podcasts/feed/ still works if you know it, but going to https://blog.stackexchange.com/tags/podcasts/ gives you an RSS link to just "/feed/".
This is probably not a good place for bug reports :-S
However, the podcast feed is impossible to find now if you didn't have it before. Previously I think it was on every podcast post as an RSS link. Now the individual tag RSS links all point to the main feed, rather than to a per-tag feed.
e.g. http://blog.stackoverflow.com/category/podcasts/feed/ still works if you know it, but going to https://blog.stackexchange.com/tags/podcasts/ gives you an RSS link to just "/feed/".
This is probably not a good place for bug reports :-S
Did anyone evaluate ghost? If so, I'm curious what the verdict was.
https://ghost.org/
https://ghost.org/
We did consider Ghost, I love it and it would've given us uploading images to S3 for free, but static files and being able to use any markdown editor you want won over.
Last year, I took two days to build a mostly static blog to replace WordPress. It pays off everytime there is a WordPress vulnerability I don't have to worry about, and everytime we post an interesting post and I don't have to worry about if WordPress is going to fall over (yes, maybe there's some caching thing we could have used). It pays off when I have multiple data centers running, because WordPress doesn't have the ability to read from a local MySQL slave, so perf on the datacenter far from the master is terrible (the i18n plugin we were using didn't help).
I imagine the payoff is similar for the stack overflow team as well.
EDIT: I don't understand why they are using disqus for comments, when they have a self hosted comment platform, in my case we don't accept comments.
I imagine the payoff is similar for the stack overflow team as well.
EDIT: I don't understand why they are using disqus for comments, when they have a self hosted comment platform, in my case we don't accept comments.
Simple caching would have solved all of your issues. Classic case of not profiling before blaming the whole system.
Stack Exchange uses Markdown for everything on their Q&A sites already. If they expect their users to be able to use it, I think they can expect the same from their employees. It's likely not a wasted effort to learn it as they might need it somewhere else as well.
The Stack Exchange blog was hacked a few years ago, I suspect that was part of the motivation to switch to something they know better.
The Stack Exchange blog was hacked a few years ago, I suspect that was part of the motivation to switch to something they know better.
My employer originally had a Wordpress-based blogging product.
They spent more than $2M writing their own blogging system, and it still was missing many features that come with Wordpress.
They ditched it and switched back to Wordpress.
They spent more than $2M writing their own blogging system, and it still was missing many features that come with Wordpress.
They ditched it and switched back to Wordpress.
TWO MILLION DOLLARS on a blogging system.
I'm speechless.
I'm speechless.
It might surprise you even more to know that the creators of the most popular blogging system have raise $317M.
10 silicon Valley software engineers on something goes through 2 million in under a year when you consider total cost of employment.
If it takes 10 engineers a year to build a blog then something is very wrong.
I would like to see a study where one would give the same project with a solid deadline to three different teams of 3, 15, 30 engineers and see how long it takes each them to finish it.
The marketing team can use http://prose.io without learning Markdown.
Using using an external comment provider promotes some popular users to comment such as Grimlock whom otherwise wouldn't.
Using using an external comment provider promotes some popular users to comment such as Grimlock whom otherwise wouldn't.
Someone else also mentioned prose.io - something I didn't come across when building this. Going to explore.
Asana did something similar: https://eng.asana.com/2014/02/scaling-asana-com/
There is an argument to be made that a ubiquitous, developer-first system translates into more instant changes. Seeing changes quickly makes it easier to iterate, which is better over time than the initial investment to learn Markdown.
There is an argument to be made that a ubiquitous, developer-first system translates into more instant changes. Seeing changes quickly makes it easier to iterate, which is better over time than the initial investment to learn Markdown.
i dislike wordpress quite a bit, but i also agree that this was a head-scratching investment.
Spoiler alert: it takes awhile to get there, but the answer is: Jekyll. Which is neat...even as active as their blogging efforts are, as I was reading the intro grafs, I kept thinking, "Yeah, could be done just fine as Jekyll"...I was mostly expecting them to announce new in-house built blogging software, but was thrilled to see that the rest of the post is about how to migrate from WordPress to Jekyll.
At SunSed we are trying to solve this exact problem: A blogging platform for professional whith batteries included.
Kind of excited to see WordPress does not work for everyone.
Thanks for sharing!
I'm all for spending engineering time on non-project related tasks occasionally, but this seems like an excessive use of valuable time to reinvent the wheel. I'm sure they could have found a blogging platform that didn't require custom markdown interfaces, Python scripting, and re-writing everything from scratch. It's just a blog.
Our problems actually weren't with Jekyll. Like I mention in the post, much of the functionality came right out of the box. The major pain here was exporting and combing through the data afterwards, and that required some Python to parse comments and convert to markdown. Once we did, that meant less security holes, opening up our content and technology more, among other pluses.
An old unix greybeard of my acquaintance told me circa 2004 that "writing a blog engine is this generation's version of writing a shell".
Creation is more fun than selection. And in the same position, I'd ditch Wordpress like a stolen sports car.
Creation is more fun than selection. And in the same position, I'd ditch Wordpress like a stolen sports car.
I disagree. It seems like the biggest time-wasters were exporting all their data from wordpress without breaking anything.
Even if they found "a blogging platform that iddn't require custom markdown interfaces" that would not have solved the problems they faced getting data out of wordpress.
Even if they found "a blogging platform that iddn't require custom markdown interfaces" that would not have solved the problems they faced getting data out of wordpress.
So why didn't they just fix WordPress. If export was expensive and they were willing to burn engineering time on it, the solution seems obvious.
However, WordPress is not cool so they went to the tired old "reinvent blogging". No open source contributions, just building castles in their own sandbox.
However, WordPress is not cool so they went to the tired old "reinvent blogging". No open source contributions, just building castles in their own sandbox.
Wordpress is filled with third rails.
I don't want to make this sound like an appeal to authority...but have you created/maintained both WordPress and static-generated blogs (either Jekyll, or something similar like Middleman)? Without having done both, I'd tend to agree with you: a blog is a blog.
The opportunity cost from system migration (and re-invention) is non-trivial. But so are the opportunities gained with a new infrastructure. The speed gains from serving static files is nice, but to me, they are no where near the kind of gains I make by being able to work with plain text files.
When I want to write a file, I pop open my (probably already open) text editor, bang it out, save, switch to Terminal, push to S3. Did I make a typo? OK, fix the typo, save, re-run the Terminal command. Maybe I want to push a few posts, but I want to make sure they all look right...OK, run the jekyll server and look at localhost. Then run the push command. Oh wait, maybe one of those posts should be a draft. OK, set it to draft in the metadata, re-push, etc. etc.
I have a particularly bad experience with WordPress because I've been running my blog on it for five years on the cheapest Dreamhost shared server. However, I've spent a good amount of time researching and installing WordPress plugins to do caching...but the problem is that every goddamnned database action, including setting a post from draft to publish, or vice versa, is a 5 to 50 second process.
Obviously, Stack Exchange doesn't suffer under that kind of hosting issue. And they could probably set up their WordPress on a local instance so that it can be locally browsed (in my experience, this can be profoundly difficult, and quite the weekend project). But no matter what their hosting infrastructure is, it still doesn't negate the fact that every posting action requires a database -- everything from creating users, setting user roles, setting post status, and of course, editing the post.
Sure, you have to learn those things in Jekyll too...but the conventions are easy...and most importantly, you can do them within the confines of your text editor. That means everyone who contributes to the blog, engineers and marketers, gets better at their text editor (and operating system). This is a much more worthwhile skill than getting better at WordPress...which is a non-trivial skill...WordPress, to its credit, is fast moving in its development...for occasional publishers like me, this means I spent a good amount of cognitive energy re-learning the interface. With Jekyll, even as the software changes, I just have to learn my text editor. And that's just wonderful, considering how much of my work involves a text editor.
For non-techies, I imagine it's a little daunting at first...but they're fully-functioning adults. They can handle plain text. And working with Jekyll not only makes it easier for the engineers to do the occasional hacking of templates, it allows the non-techies to actually _see_ it without having to jump through the many plugin submenus in the WordPress admin. It is literally a Cmd-T (in Sublime Text) to look up a file. Or, a Find All across a project...which is another huge benefit...blog contributors often need to look up past material...the text editor search is far superior to what can be done in the WP Admin or other web search interfaces.
So it's not that "WordPress is not cool"...and the fact that you think that one solution is to "just fix WordPress" makes me think you don't understand the problem. WordPress doesn't need to be "fixed" to be more like Jekyll, it is a fundamentally different product, with certain tradeoffs and benefits worth pursing. To say that the OP should've "just fixed WordPress" is like saying, "Why use Facebook? Just get better at using the CC: field in email"
The opportunity cost from system migration (and re-invention) is non-trivial. But so are the opportunities gained with a new infrastructure. The speed gains from serving static files is nice, but to me, they are no where near the kind of gains I make by being able to work with plain text files.
When I want to write a file, I pop open my (probably already open) text editor, bang it out, save, switch to Terminal, push to S3. Did I make a typo? OK, fix the typo, save, re-run the Terminal command. Maybe I want to push a few posts, but I want to make sure they all look right...OK, run the jekyll server and look at localhost. Then run the push command. Oh wait, maybe one of those posts should be a draft. OK, set it to draft in the metadata, re-push, etc. etc.
I have a particularly bad experience with WordPress because I've been running my blog on it for five years on the cheapest Dreamhost shared server. However, I've spent a good amount of time researching and installing WordPress plugins to do caching...but the problem is that every goddamnned database action, including setting a post from draft to publish, or vice versa, is a 5 to 50 second process.
Obviously, Stack Exchange doesn't suffer under that kind of hosting issue. And they could probably set up their WordPress on a local instance so that it can be locally browsed (in my experience, this can be profoundly difficult, and quite the weekend project). But no matter what their hosting infrastructure is, it still doesn't negate the fact that every posting action requires a database -- everything from creating users, setting user roles, setting post status, and of course, editing the post.
Sure, you have to learn those things in Jekyll too...but the conventions are easy...and most importantly, you can do them within the confines of your text editor. That means everyone who contributes to the blog, engineers and marketers, gets better at their text editor (and operating system). This is a much more worthwhile skill than getting better at WordPress...which is a non-trivial skill...WordPress, to its credit, is fast moving in its development...for occasional publishers like me, this means I spent a good amount of cognitive energy re-learning the interface. With Jekyll, even as the software changes, I just have to learn my text editor. And that's just wonderful, considering how much of my work involves a text editor.
For non-techies, I imagine it's a little daunting at first...but they're fully-functioning adults. They can handle plain text. And working with Jekyll not only makes it easier for the engineers to do the occasional hacking of templates, it allows the non-techies to actually _see_ it without having to jump through the many plugin submenus in the WordPress admin. It is literally a Cmd-T (in Sublime Text) to look up a file. Or, a Find All across a project...which is another huge benefit...blog contributors often need to look up past material...the text editor search is far superior to what can be done in the WP Admin or other web search interfaces.
So it's not that "WordPress is not cool"...and the fact that you think that one solution is to "just fix WordPress" makes me think you don't understand the problem. WordPress doesn't need to be "fixed" to be more like Jekyll, it is a fundamentally different product, with certain tradeoffs and benefits worth pursing. To say that the OP should've "just fixed WordPress" is like saying, "Why use Facebook? Just get better at using the CC: field in email"
"Why use Facebook? Just get better at using the CC: field in email"
:)
:)
We tend to reinvent the wheel often not because we need more wheels but because we need more inventors.
That’s exactly what I just did for my daily video / podcasting site Rate My App.com.
- Everything in markdown
- Generate the podcasts as an RSS feed
- Generate the list of videos as JSON, filtered inside the AngularJS app
Such an approach is not great for SEO in the short-term, so I won't be surprised if they lose ranking because their posts and comments are missing from the the html until their jQuery code fetches and renders them.
For me, I don't have 700 posts so that's easy to address later once I have content worth indexing, keeping the urls the same of course. :)
- Everything in markdown
- Generate the podcasts as an RSS feed
- Generate the list of videos as JSON, filtered inside the AngularJS app
Such an approach is not great for SEO in the short-term, so I won't be surprised if they lose ranking because their posts and comments are missing from the the html until their jQuery code fetches and renders them.
For me, I don't have 700 posts so that's easy to address later once I have content worth indexing, keeping the urls the same of course. :)
Check out [prerender](https://prerender.io/) to get around the SEO issue.
It is not possible to subscribe to feeds individually ? For instance I would like to subscribe only for the engineering RSS feed. At the moment all I could find was a general feed for the entire blog.
Static blogs are doing pretty well, but setting up a sepparate interface or flow for writers - not admins and developers - is still a big of a chore, unless you don’t mind giving people full access to the site repo.
I’m thinking about setting up a separate draft and post repo and importing them into the root with git subtree.
I’m thinking about setting up a separate draft and post repo and importing them into the root with git subtree.
Damn. Was hoping for a self hosted (and full featured) disqus alternative or something to appear one day.
Discourse was really, really close for us. The breaker was the lack of a real WordPress import support: https://meta.discourse.org/t/is-it-possibile-to-import-exist...
That being said, that evaluation was done before we found out how difficult it would be with Disqus.
That being said, that evaluation was done before we found out how difficult it would be with Disqus.
Are there any good, corporate solutions for organizations that lack in-house engineering resources?
Jekyll is pretty low impact as far as engineering goes...all the engineering work here went into migration of the existing blog. I would think getting Jekyll set up would be even easier than a self-hosted Wordpress installation.
It seems like this was a big waste of time reinventing Octopress.
Octopress is very difficult to use, and I would not recommend it. Using jekyll directly is a much better choice.
Octopress is built on top of jekyll and what is a real nuisance is how long it takes to re-generate a site with more than a few posts.
This isn't true anymore. Octopress uses an older version of Jekyll. If you're on 3.0 (which we're using) has incremental generation. Here's the log of a recent incremental build:
[Step 1/3] Jekyll build (11s)
[20:52:40][Jekyll build] Jekyll build --config '_blog.config.yml' --no-watch
[20:52:43][Jekyll build] Configuration file: C:/BuildAgent/work/4e27a39603eb2bf1/blog/_config.yml
[20:52:43][Jekyll build] Source: C:/BuildAgent/work/4e27a39603eb2bf1/blog
[20:52:43][Jekyll build] Destination: C:/BuildAgent/work/4e27a39603eb2bf1/blog/_site
[20:52:43][Jekyll build] Incremental build: disabled
[20:52:43][Jekyll build] Generating...
[20:52:52][Jekyll build] done in 9.082 seconds.Cool, time to upgrade then!
They had me at "Not Wordpress or PHP"
They lost the comment system and had to move the data to an external provider, which instantly bit them.
They had to write Python scripts to munge the data.
But now they have the performance of static html. I guess just changing two lines in the nginx configuration did not came to mind.
I think this was a very poor investment and will never pay off.