Static Site Post Scheduler: Using AWS Lambda and Serverless to Schedule Blog Posts(serverless.com)
serverless.com
Static Site Post Scheduler: Using AWS Lambda and Serverless to Schedule Blog Posts
https://serverless.com/blog/static-site-post-scheduler/
47 comments
Founder of https://buttercms.com here – we should talk!
A possible goal is to have the serving stack be as simple as possible.
After all, you can argue that the publishing-side stack is anyway very complex: it includes e.g. the Github UI, if one makes edits using the Github UI.
After all, you can argue that the publishing-side stack is anyway very complex: it includes e.g. the Github UI, if one makes edits using the Github UI.
I use jekyll NOT because it's simple (Actually it's 100 times more complex to use than a simple wordpress or medium) but because:
1. I can keep track changes with git 2. I can host them on Github pages for FREE 3. I own the content 4. More flexible
In fact I've been really annoyed with all the boilerplate code and extensions I need to deal with in order to have a "usable" workflow set up, I almost thought about moving back to free blog services like medium, tumblr, wordpress, etc. but the only reasons I can't are the ones I mentioned above.
1. I can keep track changes with git 2. I can host them on Github pages for FREE 3. I own the content 4. More flexible
In fact I've been really annoyed with all the boilerplate code and extensions I need to deal with in order to have a "usable" workflow set up, I almost thought about moving back to free blog services like medium, tumblr, wordpress, etc. but the only reasons I can't are the ones I mentioned above.
Jekyll is way more simple from a "stack" perspective. It has no database, has less moving parts.
Wordpress is simple from an end user perspective.
Services like medium and tumblr hide the complexity from its users. Its a trade off.
Wordpress is simple from an end user perspective.
Services like medium and tumblr hide the complexity from its users. Its a trade off.
What you said is all correct.
But my point was it's not just about "simplifying the stack". You said "the allure of a static site or a static blog is to simplify the stack;", but the thing is, not everyone thinks that way.
But my point was it's not just about "simplifying the stack". You said "the allure of a static site or a static blog is to simplify the stack;", but the thing is, not everyone thinks that way.
Simple != easy.
Wordpress and co are easy, but they're fundamentally much more complex than static sites.
Jekyll is quite simple, but it can be hard to use in comparison.
Wordpress and co are easy, but they're fundamentally much more complex than static sites.
Jekyll is quite simple, but it can be hard to use in comparison.
Thanks for the comment foxhop! We try to keep things as simple as possible where we can.
This project came out of the need from our content team needing to publish content late at night and when people are out of office =)
Check out the code https://github.com/serverless/post-scheduler/blob/master/han... this serverless service is just 2 functions that get deployed. It's a set it and forget it service =)
This project came out of the need from our content team needing to publish content late at night and when people are out of office =)
Check out the code https://github.com/serverless/post-scheduler/blob/master/han... this serverless service is just 2 functions that get deployed. It's a set it and forget it service =)
I think if I was doing operations for a company using a static site I would end up with something similar.
I suppose my original perspective was coming from a one-person operation.
I suppose my original perspective was coming from a one-person operation.
If you want the flexibility and performance of static sites, but don't want to deal with workflow setups, you should check Pragma - http://www.laktek.com/2016/11/29/introducing-pragma/
With Jekyll you can just future-date your blog posts then trigger a republish on a daily cron. By default future-dated posts won't show up. So a daily build will publish them on the right day.
If you host using Netlify you can create a build hook to just curl. You could even use Cronitor for a free cron.
If you host using Netlify you can create a build hook to just curl. You could even use Cronitor for a free cron.
This is what I do too. It works well.
Although in my case, I host my Jekyll based site on my own server and just run a daily build via local cron.
The workflow to post anything (present or scheduled) is nothing more than a single git push to get the new content on the server.
It's as simple as can be. #serversforlife
Although in my case, I host my Jekyll based site on my own server and just run a daily build via local cron.
The workflow to post anything (present or scheduled) is nothing more than a single git push to get the new content on the server.
It's as simple as can be. #serversforlife
I used Serverless back in the v0.5 days, picked it up again yesterday for another project. I have to say, the current version has improved vastly, from project architecture to ease of use (remember all those `json` files?!) and also documentations! Not to mention the provider support it now has. I like the idea of Serverless, and now I love it even more!
I do this, though with Teamcity. More predictable, less AWS heavy, but cool solution especially if you already have the tools to make it easy.
http://explainshell.com/explain?cmd=at
Or use a CI cron running daily?
Or use a CI cron running daily?
Yeah, they said its "serverless" but it requires an AWS lambda function. I use Jekyll for my blog but I simply commit, push, and run a script that copies the files to S3. https://www.tberra.com/aws/amazon/meta/2016/11/12/the-birth-... its not scheduled but I could easily tell linux to run that command once at whatever time I wanted.
or add a cron job to `git mv` a file from `pending` to `current` and run the commit/publish cycle.
AWS lambda functions __are__ serverless. There's no notion of uptime, provisioning or managing servers when using lambda.
> There's no notion of uptime, provisioning or managing servers when using lambda.
There's no notion of uptime, provisioning or managing Web sites when using Web sites; but the operators of the Web site certainly have to tackle such things.
Likewise, I'm sure the operators of AWS lambda are very much concerned about such things, even if the users aren't.
In that sense, wordpress.com is "serverless".
There's no notion of uptime, provisioning or managing Web sites when using Web sites; but the operators of the Web site certainly have to tackle such things.
Likewise, I'm sure the operators of AWS lambda are very much concerned about such things, even if the users aren't.
In that sense, wordpress.com is "serverless".
This is really just a semantic argument. You're free to call it Function as a Service or FaaS if you want.
Hosted WordPress is SaaS but not FaaS.
Hosted WordPress is SaaS but not FaaS.
This is true, but what triggers the lambda function to run? Where does the lambda function run when its triggered?
"Serverless" really means "your code is run as a container with a finite lifetime on an arbitrary machine in a standardized, non-configurable PaaS machine cluster."
In essence, it's a return to the "cgi-bin" model of scripting, where you write a script that only runs for the lifetime of one request, and then a web server will spin it up to respond to a relevant request and it'll terminate at the end of that request.
The only differences in the modern instantiation of the concept, are that:
1. the "cgi-bin" directory is a mounted object store, common to an entire cluster; and
2. there's some fancy preloading logic that lets Lambda functions stay "hot" between calls to them, so that they can run very quickly. (Sort of like Phusion Passenger does for Ruby code, except with enough isolation to make it safe for multitenant environments. Heroku's old Alpine stack might be a better comparison.)
In essence, it's a return to the "cgi-bin" model of scripting, where you write a script that only runs for the lifetime of one request, and then a web server will spin it up to respond to a relevant request and it'll terminate at the end of that request.
The only differences in the modern instantiation of the concept, are that:
1. the "cgi-bin" directory is a mounted object store, common to an entire cluster; and
2. there's some fancy preloading logic that lets Lambda functions stay "hot" between calls to them, so that they can run very quickly. (Sort of like Phusion Passenger does for Ruby code, except with enough isolation to make it safe for multitenant environments. Heroku's old Alpine stack might be a better comparison.)
One lambda runs when github webhooks are fired: https://github.com/serverless/post-scheduler#github-webhook-...
The other lambda fires on a cron job https://github.com/serverless/post-scheduler#cron-job-archit...
The other lambda fires on a cron job https://github.com/serverless/post-scheduler#cron-job-archit...
Clearly you need to learn more about lambda https://aws.amazon.com/lambda/
[deleted]
Since serverless uses AWS primarily. I'm assuming if I put all my eggs in the serverless basket and Amazon has some amazing outage. I'm screwed?
To which I don't understand the appeal of serverless besides having no servers lol.
To which I don't understand the appeal of serverless besides having no servers lol.
Hello there! The serverless framework supports a number of different FAAS (function as a service) providers like IBM openwhisk, AWS lambda, & Azure functions from microsoft. See https://serverless.com/framework/docs/providers/ for more info.
So it's up to you where your code runs =)
So it's up to you where your code runs =)
And it's still a static web site, so you're only "screwed" in terms of updates. And if you can't wait for Lambda to be restored, either run the script locally or manually hot-fix those static HTML files.
Wow. Talk about building a rocket ship just to cross the street.
Edit: My backyard is filled with old rocket ships, so I am an expert on this subject :-)
Edit: My backyard is filled with old rocket ships, so I am an expert on this subject :-)
I built the rocket ship so you don't have to ™
https://www.youtube.com/watch?v=qz72V84uWWA
https://www.youtube.com/watch?v=qz72V84uWWA
Instead of hourly polling, which I find quite inelegant, couldn't you trigger creation of new schedule rule with the correct time that then triggers the publishing?
Hey zokier. Thanks for the comment.
In theory yes. I could provision a new function with an exact cron schedule to run once and then tear itself down.
In practice, this is a lot of extra logic to write to achieve this and would also potentially make own AWS account rather messy.
The hourly cron runs about 720 times a month which is far below the 1,000,000 free invocations that lambda provides under their free tier.
In theory yes. I could provision a new function with an exact cron schedule to run once and then tear itself down.
In practice, this is a lot of extra logic to write to achieve this and would also potentially make own AWS account rather messy.
The hourly cron runs about 720 times a month which is far below the 1,000,000 free invocations that lambda provides under their free tier.
How about using the new AWS Step Functions instead?
This would allow a full push model while still keeping a nice and tight workflow:
1. fire off AWS Step function on new post to S3 bucket
2. read delay / schedule time in metadata from (assumable) markdown post
3. trigger AWS Step Function using the Wait State defined based on read value from 2.
4. As part of Step Function: After Wait State -> Execute Lambda to publish post.
Note: 3) requires setting variable time in Wait State based on environment. Haven't read up so don't know if that's possible, but guess it should be.
1. fire off AWS Step function on new post to S3 bucket
2. read delay / schedule time in metadata from (assumable) markdown post
3. trigger AWS Step Function using the Wait State defined based on read value from 2.
4. As part of Step Function: After Wait State -> Execute Lambda to publish post.
Note: 3) requires setting variable time in Wait State based on environment. Haven't read up so don't know if that's possible, but guess it should be.
Hey gbrits. Thanks for the comment.
This looks like a great alternative solution!
Haven't messed around with step functions yet but was looking into it today after I found this https://www.youtube.com/watch?v=9MKL5Jr2zZ4&t=226s
Are you using step functions for other flows?
This looks like a great alternative solution!
Haven't messed around with step functions yet but was looking into it today after I found this https://www.youtube.com/watch?v=9MKL5Jr2zZ4&t=226s
Are you using step functions for other flows?
This is the correct answer.
AWS lambda and 'serverless' to replace a cronjob
And paying Amazon for the privilege of doing so!
Well done... golf clap
And paying Amazon for the privilege of doing so!
Well done... golf clap
Thanks tom! I love golf!
> The hourly cron runs about 720 times a month which is far below the 1,000,000 free invocations that lambda provides under their free tier.
This entire service (the 2 functions & 1 dynamo table) should run well under the AWS free tier. see https://aws.amazon.com/free/
> The hourly cron runs about 720 times a month which is far below the 1,000,000 free invocations that lambda provides under their free tier.
This entire service (the 2 functions & 1 dynamo table) should run well under the AWS free tier. see https://aws.amazon.com/free/
Does this not seem excessive? Doesn't anyone care about simplicity, or elegance, or -- at minimum -- not relying on others?
Good point. I care greatly about these things.
This project greatly simplifies how we publish content in an elegant out of sight out of mind way.
Before, if someone was out of town, we would need to rely on others to push the merge button on posts for us. Now, the robots do it for us!
This project greatly simplifies how we publish content in an elegant out of sight out of mind way.
Before, if someone was out of town, we would need to rely on others to push the merge button on posts for us. Now, the robots do it for us!
While I don't know your situation your use case is understandable. It's just hard to watch such big tools serving up simple HTML with what could be a shell script or two running in cron.
>not relying on others
Avoiding AWS based solutions to "not rely on others" is like using a kerosene lamp in the center of Manhattan. You may be more self sufficient, but.... why?
Avoiding AWS based solutions to "not rely on others" is like using a kerosene lamp in the center of Manhattan. You may be more self sufficient, but.... why?
Recent AWS outages come to mind...
... as for self-sufficiency, why not? You're trading freedom and independence for serfdom in Amazon's little kingdom. They'll seduce you with shit exactly like this -- stringing together pointless, useless services to achieve an end that in actuality requires 999999999% less machinery than what they sold you -- to make sure your next project is with them and all their comfy little tools at something 2x to 3x the price. The collective knowledge of running servers represented by a federated, disparate group of specialists slowly drains away from public access, only to re-coalesce behind private walls to enrich a bunch of greedy, already-rich fucks instead of the general public. (You can already see this in the declining quality of google results for technical queries)
Meanwhile lots of good admin guys lose their jobs and the world of software development continues its transformation into a cesspool of spoiled brat devs begging, neigh CRYING for Bezos/Nadella/etc to wipe their ass for them.
Welcome to the future! I'll take my self-sufficiency...
... as for self-sufficiency, why not? You're trading freedom and independence for serfdom in Amazon's little kingdom. They'll seduce you with shit exactly like this -- stringing together pointless, useless services to achieve an end that in actuality requires 999999999% less machinery than what they sold you -- to make sure your next project is with them and all their comfy little tools at something 2x to 3x the price. The collective knowledge of running servers represented by a federated, disparate group of specialists slowly drains away from public access, only to re-coalesce behind private walls to enrich a bunch of greedy, already-rich fucks instead of the general public. (You can already see this in the declining quality of google results for technical queries)
Meanwhile lots of good admin guys lose their jobs and the world of software development continues its transformation into a cesspool of spoiled brat devs begging, neigh CRYING for Bezos/Nadella/etc to wipe their ass for them.
Welcome to the future! I'll take my self-sufficiency...
Pointing at a service with 99.99% uptime and claim the outages as a reason to go elsewhere? Host it yourself and hit that same uptime target. I'll be impressed.
AWS isn't built for little static sites. It is built to host Amazon. And others likewise as big. The ability to do neat toys like this at a low cost, utilizing far more machinery than you would ever need to for such small project, is a happy byproduct.
> Meanwhile lots of good admin guys lose their jobs
Don't go all union-y on me! Frictional Unemployment is unavoidable. The reality is that services like AWS free up all those clearly intelligent (by virtue of being "good") admins to utilize their intellect elsewhere. We live in the age of the STEM shortage. If you can work a computer and can be described as "good", I won't worry for you.
FWIW, it's "nay" not "neigh"
AWS isn't built for little static sites. It is built to host Amazon. And others likewise as big. The ability to do neat toys like this at a low cost, utilizing far more machinery than you would ever need to for such small project, is a happy byproduct.
> Meanwhile lots of good admin guys lose their jobs
Don't go all union-y on me! Frictional Unemployment is unavoidable. The reality is that services like AWS free up all those clearly intelligent (by virtue of being "good") admins to utilize their intellect elsewhere. We live in the age of the STEM shortage. If you can work a computer and can be described as "good", I won't worry for you.
FWIW, it's "nay" not "neigh"
> Pointing at a service with 99.99% uptime and claim the outages as a reason to go elsewhere? Host it yourself and hit that same uptime target. I'll be impressed.
Well, according to pingdom I hit 99.98% uptime with the dozen-or-so cheap little DO boxes I run for my firm's infrastructure. That's playing the dual roles of sole developer and sole support on those systems. My data pipeline collects records from hundreds of deployments worldwide, through all manner of diverse corporate IT environments. It really isn't that hard.
I'll agree with your points on frictional unemployment when the thing displacing them represents actual forward progress. AWS is a land grab, not progress.
Well, according to pingdom I hit 99.98% uptime with the dozen-or-so cheap little DO boxes I run for my firm's infrastructure. That's playing the dual roles of sole developer and sole support on those systems. My data pipeline collects records from hundreds of deployments worldwide, through all manner of diverse corporate IT environments. It really isn't that hard.
I'll agree with your points on frictional unemployment when the thing displacing them represents actual forward progress. AWS is a land grab, not progress.
I think this is the future of web publishing. I expect that we'll see several such frameworks for all the major "serverless" platforms.
Adding the cognitive burden of "serverless" and/or lambda just to schedule seems to complicate something that isn't very complicated. Its almost like an invented problem.
That said, long live this static site generator movement.
I'm betting this movement will continue to grow which is why I'm bootstrapping https://www.remarkbox.com (comments-as-a-service)