I Built My Own Shitty Static Site Generator(erikwinter.nl)
erikwinter.nl
I Built My Own Shitty Static Site Generator
https://erikwinter.nl/articles/2020/why-i-built-my-own-shitty-static-site-generator/
178 comments
For this exact purpose, I want a CLI-based template processor. Just a standalone template processor. They all seem to be buried in inside other frameworks or libraries: Smarty inside PHP (don't want to write a php-based CLI tool), Jinja2 is built for use by Python (I don't care to write python) ... I just want to call an engine with a template and an environment and get the final document out.
I have found when I’m looking at the top static site generators they always lack a good programmatic API in one way or another. Either you can’t change defaults if needed, or hook into the build itself in some meaningful way, or it’s a shell they requires deep customization in areas you may not be prepared to take on.
Doesn’t mean Hugo, 11eventy, Jekyll etc aren’t good it’s just an area that could still have mass improvement I think
Doesn’t mean Hugo, 11eventy, Jekyll etc aren’t good it’s just an area that could still have mass improvement I think
I've found Lektor's API decent. It's easier when the generator is written in an interpreter language, as compiled languages generally have a harder time exposing their APIs.
It’s such a simple thing to make. It’s easier than trying to figure out and customize someone else’s.
Writing a SSG for oneself is just like writing a single Makefile, for one small project, with one contributor, without cross platform concerns.
Using a generic SSG is like learning CMake.
Using a generic SSG is like learning CMake.
This is quite well put.
Althought most of the times using a generic SSG feels more like learning the autotools than CMake.
Althought most of the times using a generic SSG feels more like learning the autotools than CMake.
If helpful, we open sourced our static site at Ockam.io
https://github.com/ockam-network/website
https://github.com/ockam-network/website
I love these posts on static site generators. They are a constantly reoccurring topic. Everyone wants something that's "easy" and "simple" but this is at odds with having flexibility to handle more than one person's favorite workflow. There's endless discussion because there are endless solutions.
Personally, I've used a number of static site generators and I ended up in the opposite direction. I needed a website where I could write posts and not have to worry about more than just writing and posting, no deployment, no plain text intermediate languages between me and the end result.
I ended up using Wordpress. I was already familiar with it, knew how to self-host it, and was comfortable modifying a theme to suit my purposes.
Writing and posting new articles is so much easier using rich text. I generally type up my post in Apple Notes first. Copying and posting to Wordpress saves most of the formatting. Inserting images is drag and drop.
Large numbers of good plugins meant it was easy to find something to implement features like having a click-through warning on 18+ posts. Install a plugin, configure it, and I'm done.
Wordpress works well for me, but a lot of programmers are rightfully horrified. Wordpress is popular and complex, which results in a long history of security issues requiring mitigation. It's inefficient and needs caching. It needs to be backed up because everything is in a database, not plain text. As a result, it requires more systems knowledge than a static website.
Static site generators and applications like Wordpress are tools.
Everyone ultimately looks for a tool that works for them. My goal is to write posts, not fiddle endlessly. Other people love the fiddling.
So, if you’re looking to build and maintain a website, think about what’s important to you. What’s your skill set, interests, preferred workflow, and goals?
Personally, I've used a number of static site generators and I ended up in the opposite direction. I needed a website where I could write posts and not have to worry about more than just writing and posting, no deployment, no plain text intermediate languages between me and the end result.
I ended up using Wordpress. I was already familiar with it, knew how to self-host it, and was comfortable modifying a theme to suit my purposes.
Writing and posting new articles is so much easier using rich text. I generally type up my post in Apple Notes first. Copying and posting to Wordpress saves most of the formatting. Inserting images is drag and drop.
Large numbers of good plugins meant it was easy to find something to implement features like having a click-through warning on 18+ posts. Install a plugin, configure it, and I'm done.
Wordpress works well for me, but a lot of programmers are rightfully horrified. Wordpress is popular and complex, which results in a long history of security issues requiring mitigation. It's inefficient and needs caching. It needs to be backed up because everything is in a database, not plain text. As a result, it requires more systems knowledge than a static website.
Static site generators and applications like Wordpress are tools.
Everyone ultimately looks for a tool that works for them. My goal is to write posts, not fiddle endlessly. Other people love the fiddling.
So, if you’re looking to build and maintain a website, think about what’s important to you. What’s your skill set, interests, preferred workflow, and goals?
Sometimes I wish I could just use WordPress as a static site generator off of a docker machine that turns on on login and off at log off. Nobody wants to deal with slow and insecure always-on WordPress sites.
I know that there are some websites that take care of this… for money. Unfortunately $20/month is not in the budget for one-off sites.
I know that there are some websites that take care of this… for money. Unfortunately $20/month is not in the budget for one-off sites.
You can use WP2Static and either run a container locally with something like https://lokl.dev, or spin up a VM that's firewalled off and access it through an SSH tunnel. The auto-on/-off part really does little for security, it's the firewalling that does the trick. Turning it off is good to save costs though.
I'm actually working on basically exactly what you described (using WP2Static), and we're planning for a competitive $10-15/mo price, but we're not quite ready for launch yet.
I'm actually working on basically exactly what you described (using WP2Static), and we're planning for a competitive $10-15/mo price, but we're not quite ready for launch yet.
Still searching a easy way to use digital ocean, Docker to pushing a simple web site so I can do my mumble jumbo there daily.
It should be possible. You can use wget to crawl your Wordpress site and save it all to html files, which you can then copy/ftp to your static site. I've done this with Drupal. Drupal 6 was at EOL and not getting security fixes. The site was fairly large and had been built over a period of years, but was no longer being updated. We wanted to keep it online, but did not want to take on a project to migrate it to the next major release of Drupal and then get on that update treadmill.
I forget the exact set of wget options but you can find blog posts about how to do this.
I forget the exact set of wget options but you can find blog posts about how to do this.
I’ve done it too, but wget is pretty “dumb” and will try to save files as main.js?v=1.2.3. Also wget doesn’t support everything so you have to carry over some files manually.
> Sometimes I wish I could just use WordPress as a static site generator off of a docker machine that turns on on login and off at log off.
But you can do exactly that right now. For example:
- use wordpress as your data backend in next.js
- not exactly a static site generator, but you can use w3tc plugin to perform full page caching with very long expiry time, which essentially turns your wordpress site into a static site. You can also do this using cloudflare "cache everything" page rule, but it must be paired with another page rule to skip caching if wp cookie found. Similarly, you can add web app firewall rules on cloudflare to disallow login access except for you.
But you can do exactly that right now. For example:
- use wordpress as your data backend in next.js
- not exactly a static site generator, but you can use w3tc plugin to perform full page caching with very long expiry time, which essentially turns your wordpress site into a static site. You can also do this using cloudflare "cache everything" page rule, but it must be paired with another page rule to skip caching if wp cookie found. Similarly, you can add web app firewall rules on cloudflare to disallow login access except for you.
Neither case is really static and you still need the server running.
Also for whatever reason even with the right headers I could never force CloudFlare to actually cache everything. I think it will just keep a copy of a visited page in some servers, but not in all, and definitely it won’t cache the whole site at once.
Also for whatever reason even with the right headers I could never force CloudFlare to actually cache everything. I think it will just keep a copy of a visited page in some servers, but not in all, and definitely it won’t cache the whole site at once.
The next.js method is completely static. You can run your wordpress instance locally in your laptop/pc, run next.js static site generator, then upload the resulting static site to aws or github pages. No need to run the wordpress instance 24/7 as it's only required as a data source during static site generation and can be turned off when you're done with it.
There is also an option in cloudflare to pre-cache everything on their edge servers so visitors will never hit the origin server, but it's not available in free plan (I think it's enterprise only plan).
There is also an option in cloudflare to pre-cache everything on their edge servers so visitors will never hit the origin server, but it's not available in free plan (I think it's enterprise only plan).
I did exactly this several times.
Hand off WordPress / Django CMS to untechy people (either hosted or installed on their pc locally), setup a daily scraping job that updates a static website. By using free static hosts (eg netlify) and free scheduled processes somewhere (eg heroku) you can do it for free.
Hand off WordPress / Django CMS to untechy people (either hosted or installed on their pc locally), setup a daily scraping job that updates a static website. By using free static hosts (eg netlify) and free scheduled processes somewhere (eg heroku) you can do it for free.
Wordpress is pretty great but for me the main reason to go with a static site generator is the simplicity of hosting. Wordpress is a complex beast which requires a server and application that must be updated regularly and backed up.
With a SSG I just have gitlab host it for me which is everything sorted and git does the backups.
With a SSG I just have gitlab host it for me which is everything sorted and git does the backups.
Yup. As I mentioned, it's a tool.
But the biggest reason I use Wordpress is I can't handle the friction of treating my website like code. My author hat hates programming and needs a faster feedback loop than flipping back and forth between different windows to see what's happening.
Being able to click Edit, fix a typo, and hit Publish, without any other steps, is what I need.
That's before I get into all of the features Wordpress supports. I installed the JetPack plugin and now I have mailing lists, analytics, monitoring, and a bunch of other things with No Extra Work.
For any website, the complexity has to go somewhere. If you're a programmer with simple needs, an SSG is easier than wrangling an angry kraken like Wordpress.
But the biggest reason I use Wordpress is I can't handle the friction of treating my website like code. My author hat hates programming and needs a faster feedback loop than flipping back and forth between different windows to see what's happening.
Being able to click Edit, fix a typo, and hit Publish, without any other steps, is what I need.
That's before I get into all of the features Wordpress supports. I installed the JetPack plugin and now I have mailing lists, analytics, monitoring, and a bunch of other things with No Extra Work.
For any website, the complexity has to go somewhere. If you're a programmer with simple needs, an SSG is easier than wrangling an angry kraken like Wordpress.
It depends on your "Just". With Wordpress I "Just" use Dreamhost and do a one-click install.
The trick with Wordpress is to accept that somethings wont be as slick as if you spent $1-10k of your own time building perfect static site generator and customising it beautifully. You have to do what most non-IT people do and make do. But I feel like I rarely have to make do with Wordpress - I can get speed, I can customise a bit with PHP sometimes (although I avoid that) and there are a tonne of plugins.
You can build many "no-code" level solutions in Wordpress for free.
The trick with Wordpress is to accept that somethings wont be as slick as if you spent $1-10k of your own time building perfect static site generator and customising it beautifully. You have to do what most non-IT people do and make do. But I feel like I rarely have to make do with Wordpress - I can get speed, I can customise a bit with PHP sometimes (although I avoid that) and there are a tonne of plugins.
You can build many "no-code" level solutions in Wordpress for free.
But my problem is similar with static sites: The various templates I’ve bought used Jekyll (Python), Gulp and NPM, it is not feasible to explain newbies how to properly install the dependencies for 3 systems (especially Python, with the 2.7 vs 3.7 difficulty).
Wordpress is inherently unstable on the long term because plugins must be upgraded (anyone who’s had a vulnerability issue knows it). But there is no other decently simple technology.
Wordpress is inherently unstable on the long term because plugins must be upgraded (anyone who’s had a vulnerability issue knows it). But there is no other decently simple technology.
Hugo is pretty easy to use. Its in the repos for most distros and you just grab the premade gitlab CI file for it and it all just works.
Writing posts is slightly more of a pain than with wordpress but its nothing compared to the pain of either updating wordpress or having it hacked.
Writing posts is slightly more of a pain than with wordpress but its nothing compared to the pain of either updating wordpress or having it hacked.
git does not do the backups, it provides file history, which is not the same thing.
There are many horror stories of people using git as their backup strategy and finding out a corrupted object or a goofed up rebase not noticed right away meant a mass panic looking for an old clone.
If you want to backup your site, do it in addition to git.
There are many horror stories of people using git as their backup strategy and finding out a corrupted object or a goofed up rebase not noticed right away meant a mass panic looking for an old clone.
If you want to backup your site, do it in addition to git.
Any recommendations for a CMS that is as pleasant to write in as WordPress but designed from the start for outputting static files? I feel like that would have some advantages over trying to hammer WordPress (and more importantly its plugin ecosystem) to act as an SSG.
Check out headless CMSes: https://jamstack.org/headless-cms/
> to hammer WordPress (and more importantly its plugin ecosystem) to act as an SSG
I don't have recent experience here but, last time I used WP (10+ years ago), they had caching plugins. Isn't that kind of the same? If not, couldn't you just yank out the editor of WP if you think it's more pleasant and put it into anything else you wanted?
I don't have recent experience here but, last time I used WP (10+ years ago), they had caching plugins. Isn't that kind of the same? If not, couldn't you just yank out the editor of WP if you think it's more pleasant and put it into anything else you wanted?
But with self-hosted Wordpress, especially with large number of plugins, you will have to constantly update your Wordpress installation or get hacked and spammed, no?
Yes, but wordpress has implemented its own form of a "cron" which allows it to do self-updates. Basically every request spawns a "cron" request (literally a HTTP request to itself over localhost) to be processed asynchronously if it's been long enough since the last. And since a "proper" wordpress install can self-modify, it can thus self-update.
My knowledge is about 5 years old at this point, but the idea still scares me silly.
My knowledge is about 5 years old at this point, but the idea still scares me silly.
There's no shortage of static site generators but I've yet to find one I didn't find frustrating.
Only like, 3 seem to have any significant number of themes and half of them don't seem to work with the latest version of the generator.
Only like, 3 seem to have any significant number of themes and half of them don't seem to work with the latest version of the generator.
Static HTML served through a bad / misconfigured httpd is worse than some integrated solutions. Some years ago everybody was using Apache, a terrible implementation.
The problem with Apache was people using the default config on small servers. Forking per request without proper limits on a small server? Zero to Swap at a hundred requests per minute. Less if it was using mod_php.
The whole architecture was bad... Try serving static pages from an Apache configured with (and using, for some other user) mod_perl, for example. My point was that the whole setup counts, not just the fact that the pages are static. I'd rather serve content from some small Go executable than Apache + static pages. But apparently it's undesirable to point such things out. Best if I don't post anymore.
Using mod_perl wasn't the best example, perhaps, as it works as one would expect and only starts a few Perl interpreters in the process which runs multiple threads.
It is famously mod_php that isn't good in this regard and will naively start one interpreter per request. Since PHP 5.4 (released in 2011) everyone should just use PHP in FPM mode instead. The fact that mod_php only supported preforking caused a lot of spite and angst against PHP the language, which was understandable but perhaps a bit misguided.
None of this has anything to do with Apache, of course. No web server should have any trouble serving static pages at any speed and you could do much worse than Apache.
It is famously mod_php that isn't good in this regard and will naively start one interpreter per request. Since PHP 5.4 (released in 2011) everyone should just use PHP in FPM mode instead. The fact that mod_php only supported preforking caused a lot of spite and angst against PHP the language, which was understandable but perhaps a bit misguided.
None of this has anything to do with Apache, of course. No web server should have any trouble serving static pages at any speed and you could do much worse than Apache.
Your problem with your first post was that you said:
Some years ago everybody was using Apache, a terrible implementation
Without backing that statement up and explaining why. Your post came across as simple flame-bait. Nevertheless, please keep posting. Contrary opinions count. Without them, the world becomes the ultimate echo chamber.
Some years ago everybody was using Apache, a terrible implementation
Without backing that statement up and explaining why. Your post came across as simple flame-bait. Nevertheless, please keep posting. Contrary opinions count. Without them, the world becomes the ultimate echo chamber.
I don't think the author is wrong for wanting to use something different or write their own tool, but the YAML front matter format is much bigger than just Hugo. There are tons of libraries out there, and even Github has a special viewer for Markdown files with front matter.
e.g. Ruby: https://rubygems.org/gems/front-matter
Python: https://pypi.org/project/python-frontmatter/
Javascript: https://www.npmjs.com/package/front-matter
PHP: https://packagist.org/packages/webuni/front-matter
Go: https://github.com/gernest/front
Elixir: https://github.com/sebastiandedeyne/yaml_front_matter
Clojure: https://github.com/liquidz/frontmatter
And many of those languages have more than one library.
Like it or not front matter in Markdown files is a bit of a standard these days.
e.g. Ruby: https://rubygems.org/gems/front-matter
Python: https://pypi.org/project/python-frontmatter/
Javascript: https://www.npmjs.com/package/front-matter
PHP: https://packagist.org/packages/webuni/front-matter
Go: https://github.com/gernest/front
Elixir: https://github.com/sebastiandedeyne/yaml_front_matter
Clojure: https://github.com/liquidz/frontmatter
And many of those languages have more than one library.
Like it or not front matter in Markdown files is a bit of a standard these days.
And it's not difficult to deal with it manually. Grab the first line as the delimiter and split the document on the next occurrence.
Personally, I find it a bit silly to optimize for outputting things in many different formats when you'll have to deal with tweaking the output for each format to get it right. And it's not a one and done deal. You'll still find things going wrong on occasion. :)
Personally, I find it a bit silly to optimize for outputting things in many different formats when you'll have to deal with tweaking the output for each format to get it right. And it's not a one and done deal. You'll still find things going wrong on occasion. :)
Pandoc: https://pandoc.org/MANUAL.html#metadata-blocks (scroll down a bit to "Extension: yaml_metadata_block")
Yeah my gatsby blog uses it too. Not sure why the author claimed only Hugo supports it.
I respond very positively to the notion of "shitty" just being another way to express that it scratches the author's itch but probably only the author's.
We made a static site generator that goes over ANY dynamic and renders a bunch of URLs to static HTML in the “same folder” for purposes of referencing relative paths for resources.
We had two main issues... one is that links to dynamic pages would need to either be rewritten to static ones, or the dev ops would have to our 301 redirects for the list of dynamic URLs which we rendered, to the corresponding static html pages.
The other issue is that various aspects of the page would be related to authenticated sessions, and should NOT appear in the static site generated resources.
Any ideas about these issues?
We had two main issues... one is that links to dynamic pages would need to either be rewritten to static ones, or the dev ops would have to our 301 redirects for the list of dynamic URLs which we rendered, to the corresponding static html pages.
The other issue is that various aspects of the page would be related to authenticated sessions, and should NOT appear in the static site generated resources.
Any ideas about these issues?
Why not serve the generated static content from a different domain? Assuming all links to the dynamic pages are relative, you don't need to rewrite the urls that way. Alternatively just update the crawler to rewrite any url that matches certain rules.
As for hiding certain element, maybe appending some simple css to the generated pages to hide them is enough? something like `.login-form {display: none !important;}`
As for hiding certain element, maybe appending some simple css to the generated pages to hide them is enough? something like `.login-form {display: none !important;}`
You could solve some of this by having a small script as a fallback. PHP would be my goto; it's pretty fast for simple things.
Another alternative is to have your generator create an nginx config with the redirects. (W3 Total Cache for Wordpress does something like this)
Another alternative is to have your generator create an nginx config with the redirects. (W3 Total Cache for Wordpress does something like this)
I wrote my static site generator, too. It is a lot of fun, and much more powerful for my specific use case than any other around (and probably would fail miserably for any other use case). It took 2-3 days and I regret nothing.
There are some novel things though, like, I use markdown files, but separate sections with ``` so I can use different inputs like ```MD for markdown, or ```JSON for pure JSON data, or ```PHP for PHP code. The VS Code highlights the portion of the code correctly, so I can work on many formats/languages on the same page. The .md file looks like this:
There are some novel things though, like, I use markdown files, but separate sections with ``` so I can use different inputs like ```MD for markdown, or ```JSON for pure JSON data, or ```PHP for PHP code. The VS Code highlights the portion of the code correctly, so I can work on many formats/languages on the same page. The .md file looks like this:
```INI META
includes = docs/index.html.md menu menuTop
title = Theme
base = docs/docs.html
slug = theme
```
```JSON menu
[
{ "label": "Demo", "url": "/demo/features" },
{ "label": "Try it!", "url": "/try.html?d=/data/try.json" },
{ "label": "Docs", "url": "/docs" }
]
```
```MD BODY
# Customize the story
Below are the parameters to customize the style of the story.
```
Cool but why not just use regular MD with YAML front matter? I don't find that JSON readable at all, comparatively, and that INI isn't any better than the YAML version.
---
includes: docs/index.html.md menu menuTop
title: Theme
base: docs/docs.html
slug: theme
menu:
- label: Demo
url: /demo/features
- label: Try it!
url: /try.html?d=/data/try.json
- label: Docs
url: /doc
---
# Customize the story
Below are the parameters to customize the style of the story.
The MD file is just for information, and I want to consume this information in different forms. The MD file is just to highlight the code correctly on VS Code on a single file. The PHP script interprets each section and processes it accordingly. This way I am not limited to markdown.
For example, for documentation, I use markdown, but for the website, that I want to control the layout better, I use JSON only, and when I need to create complex operation (as a consult to an external database), I use PHP. The markdown is very limiting, this was the motivation to create my own.
For example, for documentation, I use markdown, but for the website, that I want to control the layout better, I use JSON only, and when I need to create complex operation (as a consult to an external database), I use PHP. The markdown is very limiting, this was the motivation to create my own.
[deleted]
I completely understand the desire to build something practical that's your own.
I just built a small web application framework that I can build some basic personal apps on top of. Just like the author's, it's "shitty". It lacks features, doesn't handle edge cases, isn't particularly secure or performant.
None of that matters. It's locked up behind HTTP authentication and it's for my use only. It has exactly the features I desire, and none that I don't. I understand it top to bottom. It was fun to write and extend. It serves my needs perfectly. I didn't have to fight with, or conform to someone else's notion of how software should be built.
I just built a small web application framework that I can build some basic personal apps on top of. Just like the author's, it's "shitty". It lacks features, doesn't handle edge cases, isn't particularly secure or performant.
None of that matters. It's locked up behind HTTP authentication and it's for my use only. It has exactly the features I desire, and none that I don't. I understand it top to bottom. It was fun to write and extend. It serves my needs perfectly. I didn't have to fight with, or conform to someone else's notion of how software should be built.
I wish I could upvote this multiple times.
You go into any experienced woodworker’s shop, you can divide it into maybe 20% of the tools that see the vast majority of the action—table saws, band saws, milling machines, proper hammers. But then there is a halo of 60% or more of the things there are little helpful things—doodads, mallets, jigs, often even shelving and work surfaces—such that she uses a bunch of these on any build she undertakes, and she made them herself. “Why would I stumble around with someone else's mallet when it’s about the same cost and about half a day’s work to machine my own mallet down in a router that has exactly the weight and shape that I like it to have? Pays for itself.”
We are so far from this ideal. All of our software breaks because we imported left-pad and somebody unpublished left-pad. Wasn't even a world changing perfectly V8-on-x64-tuned performant version of left-pad, but we gotta “not reinvent the wheel.”
I don't think we do it for no reason, there is something about software itself right now that is amateurish. We’re still writing subroutines working on data structures; our brains are FORTRAN even if we call it Java, and whatever that is makes it really hard to decouple the tools that we are using to build software from the final outcome that has been built. The woodworker has a sense of the “shape” of the artifact, we routinely build the equivalent of gorgeous wooden tables that start spewing garbage onto their shiny surfaces after three weeks unattended, “oh, I didn’t realize that the built-in trash chutes could get clogged.” Wait—what? “Yeah, we allocate more wood at the bottom of the table to make up for the wood that we remove at the top of the table every night so that the surface is always clean.” Your user doesn't clean the table? “Hahaha you’re silly, users could not possibly have the technical knowledge to keep anything clean, they are not trained in sponges, and I especially could not trust the with something as beautiful and delicate as this table.” The problem has a beautiful sort of fractal quality to it, every zoom-in is more WTF.
You go into any experienced woodworker’s shop, you can divide it into maybe 20% of the tools that see the vast majority of the action—table saws, band saws, milling machines, proper hammers. But then there is a halo of 60% or more of the things there are little helpful things—doodads, mallets, jigs, often even shelving and work surfaces—such that she uses a bunch of these on any build she undertakes, and she made them herself. “Why would I stumble around with someone else's mallet when it’s about the same cost and about half a day’s work to machine my own mallet down in a router that has exactly the weight and shape that I like it to have? Pays for itself.”
We are so far from this ideal. All of our software breaks because we imported left-pad and somebody unpublished left-pad. Wasn't even a world changing perfectly V8-on-x64-tuned performant version of left-pad, but we gotta “not reinvent the wheel.”
I don't think we do it for no reason, there is something about software itself right now that is amateurish. We’re still writing subroutines working on data structures; our brains are FORTRAN even if we call it Java, and whatever that is makes it really hard to decouple the tools that we are using to build software from the final outcome that has been built. The woodworker has a sense of the “shape” of the artifact, we routinely build the equivalent of gorgeous wooden tables that start spewing garbage onto their shiny surfaces after three weeks unattended, “oh, I didn’t realize that the built-in trash chutes could get clogged.” Wait—what? “Yeah, we allocate more wood at the bottom of the table to make up for the wood that we remove at the top of the table every night so that the surface is always clean.” Your user doesn't clean the table? “Hahaha you’re silly, users could not possibly have the technical knowledge to keep anything clean, they are not trained in sponges, and I especially could not trust the with something as beautiful and delicate as this table.” The problem has a beautiful sort of fractal quality to it, every zoom-in is more WTF.
All analogies fall down at some point, but the woodworker analogy you make is just not correct, in my opinion.
The problem is that many things a developer considers tools (as in: "the best tool for the job") are actually components.
The equivalent of a woodworker's tool for a software developer would be an editor. Only you or your team might care about which particular brand of sawing machine or editor you use. And they are not an integral part of the end result.
A component for a woodworker would be a hinge or drawer rails. You would probably not appreciate it if your woodworker uses hinges that they made themselves. They are not worth the time to build. They're probably not very reliable. And if they fail they might not be easily replaced or repaired except by the person that made them. So you would only use custom components if the part you need is otherwise unavailable or does not meet specific needs.
Are you a woodworker and making a cabinet for yourself? Go ahead and make your own hinges, but for paid projects you probably should rely on standard components as much as possible; only doing custom woodwork or software engineering where it adds value.
The problem is that many things a developer considers tools (as in: "the best tool for the job") are actually components.
The equivalent of a woodworker's tool for a software developer would be an editor. Only you or your team might care about which particular brand of sawing machine or editor you use. And they are not an integral part of the end result.
A component for a woodworker would be a hinge or drawer rails. You would probably not appreciate it if your woodworker uses hinges that they made themselves. They are not worth the time to build. They're probably not very reliable. And if they fail they might not be easily replaced or repaired except by the person that made them. So you would only use custom components if the part you need is otherwise unavailable or does not meet specific needs.
Are you a woodworker and making a cabinet for yourself? Go ahead and make your own hinges, but for paid projects you probably should rely on standard components as much as possible; only doing custom woodwork or software engineering where it adds value.
For software you're more likely to get a hinge from CarpentryHub.
Hinges - like everything else on CarpentryHub - are either in a state of permanent development or abandoned. So you design your project for Hinge 13.2, but then you come back a month later - just before you open your Etsy store - and discover that Hinge x.n has now become HingeOS, which is breaking change.
It also has its own HingeAPI, which is poorly documented, somewhere.
You were used to Hinge's ScrewHole 5.3. But that's now buried inside HingeAPI, which calculates your project budget and lifetime carbon footprint for you, but no longer accepts external screwhole positions, because it works its own values from the complex project specification you have to supply - defined in DirkLang, which is a hot new carpentry specification language only ten people know, and replaces HammerLang, which was fifty five years old and very popular but had a lot of frankly questionable design choices.
And the values HingeAPI returns aren't mutable, because mutability is bad practice.
Hinges - like everything else on CarpentryHub - are either in a state of permanent development or abandoned. So you design your project for Hinge 13.2, but then you come back a month later - just before you open your Etsy store - and discover that Hinge x.n has now become HingeOS, which is breaking change.
It also has its own HingeAPI, which is poorly documented, somewhere.
You were used to Hinge's ScrewHole 5.3. But that's now buried inside HingeAPI, which calculates your project budget and lifetime carbon footprint for you, but no longer accepts external screwhole positions, because it works its own values from the complex project specification you have to supply - defined in DirkLang, which is a hot new carpentry specification language only ten people know, and replaces HammerLang, which was fifty five years old and very popular but had a lot of frankly questionable design choices.
And the values HingeAPI returns aren't mutable, because mutability is bad practice.
> CarpentryHub
I too look forward to the 2021 YC batch
I too look forward to the 2021 YC batch
As a wood worker I often build things that make my current set of tools work better. Router that I own Let me make a router table that fits in my workspace and has the features I need, now the router can be used in other ways. Want to cut circles, make a circle cutter for my router. Dovetails, jig to make that with my router. All of the tools are add on to a pretty basic tool, and they are set up for me.
To go back to the static website, I really like the Wiki format (Current favorite is http://pmwiki.org ). I picked it for three reasons 1) It's a very powerful wiki 2) I can convert the site into static pages. This lets me build in the wiki format, then publish out. 3) There is a cookbook (a tool another user wrote) to make PDF. You can select the pages in page order and produce a PDF. To that base I've added my custom items.
Pmwiki in it's own right is simple (a router), but allows for cookbooks (custom extensions) that others can add. One of the interesting things is the current lead developer doesn't really add features to the core, they also create cookbooks to add on. And the core is pretty soft, if you don't like the markup, you can change it.
A long way to go to say that I can see writing software like the OP article. But if you should look at a core tool and see if you can build around that. @Molf uses hinges as an example. You can do a lot around a stock hinge (paint, bevel the edges, add "hand crafted tool marks", a process that is easier to do than starting from ground zero.
To go back to the static website, I really like the Wiki format (Current favorite is http://pmwiki.org ). I picked it for three reasons 1) It's a very powerful wiki 2) I can convert the site into static pages. This lets me build in the wiki format, then publish out. 3) There is a cookbook (a tool another user wrote) to make PDF. You can select the pages in page order and produce a PDF. To that base I've added my custom items.
Pmwiki in it's own right is simple (a router), but allows for cookbooks (custom extensions) that others can add. One of the interesting things is the current lead developer doesn't really add features to the core, they also create cookbooks to add on. And the core is pretty soft, if you don't like the markup, you can change it.
A long way to go to say that I can see writing software like the OP article. But if you should look at a core tool and see if you can build around that. @Molf uses hinges as an example. You can do a lot around a stock hinge (paint, bevel the edges, add "hand crafted tool marks", a process that is easier to do than starting from ground zero.
I think the analogy is tortuous, but parent has a fair point -- it's just that left pad is maybe a bad example. If we were to go with components over tools, I'd say very simple building blocks are very useful as components. It's more complex dependencies where the issue lies.
To absolutely torture the analogy: say you have a general hinge that you can attach to any door. And that's fantastic, but it's a lot more complex than it needs to be (it being generalised necessitates this) and it isn't actually completely optimal in most cases (it just works ok, ish). You can use Super Hinge on all your paid projects, and it will work, it's just that it's often better to make a hinge that exactly fits the required specifications.
To absolutely torture the analogy: say you have a general hinge that you can attach to any door. And that's fantastic, but it's a lot more complex than it needs to be (it being generalised necessitates this) and it isn't actually completely optimal in most cases (it just works ok, ish). You can use Super Hinge on all your paid projects, and it will work, it's just that it's often better to make a hinge that exactly fits the required specifications.
The other thing this makes me think -- custom built carpentry is currently very expensive, for real. If you also want all the components to be custom-designed and bespoke, it's only the wealthiest who will be able afford it.
There are for sure economic factors behind choices in how software is built, for sure.
Back to the analogy and actual tools -- what employer is going to pay you to write your own editor to use to write the software the employer wants? It's not just that they are mean and choose not to pay you for this, it is not a cost they could sustain (for all but the biggest employers I guess. Google, sure, can have an editor team. Still can't let every individual programmer build their own editor)
There are for sure economic factors behind choices in how software is built, for sure.
Back to the analogy and actual tools -- what employer is going to pay you to write your own editor to use to write the software the employer wants? It's not just that they are mean and choose not to pay you for this, it is not a cost they could sustain (for all but the biggest employers I guess. Google, sure, can have an editor team. Still can't let every individual programmer build their own editor)
On the other-hand, most woodworkers I know (unlike, say construction workers) try to use joinery when possible. The joinery is usually custom built by necessity, often assisted with a mix of home-built or purchased jigs. The one exception are dowels, which are standard and very generalized. I think that the software analogy might be design patterns.
You can also think of lower-level components, like nails and glue, which even more general and standardized than something like a hinge, maybe these are like standard-library or builtins for a language?
You can also think of lower-level components, like nails and glue, which even more general and standardized than something like a hinge, maybe these are like standard-library or builtins for a language?
Edit: dropped connection, double post
"don't reinvent the wheel" is specifically for projects which other people will be forced to maintain.
You can reinvent the wheel as many times you want. If you however do that at work, your colleagues and your replacement after you left the job likely won't think too kindly of you.
You can reinvent the wheel as many times you want. If you however do that at work, your colleagues and your replacement after you left the job likely won't think too kindly of you.
I think that developers, albeit knowing about "don't reinvent the wheel", will do it nonetheless. I think psychological factors play a role here, like the "IKEA effect":
> The IKEA effect is a cognitive bias in which consumers place a disproportionately high value on products they partially created. The name refers to Swedish manufacturer and furniture retailer IKEA, which sells many items of furniture that require assembly. A 2011 study found that subjects were willing to pay 63% more for furniture they had assembled themselves, than for equivalent pre-assembled items.
https://en.wikipedia.org/wiki/IKEA_effect
I think this effect also applies to developers. They tend to value self-written applications subconsciously higher than "pre-assembled" applications.
> The IKEA effect is a cognitive bias in which consumers place a disproportionately high value on products they partially created. The name refers to Swedish manufacturer and furniture retailer IKEA, which sells many items of furniture that require assembly. A 2011 study found that subjects were willing to pay 63% more for furniture they had assembled themselves, than for equivalent pre-assembled items.
https://en.wikipedia.org/wiki/IKEA_effect
I think this effect also applies to developers. They tend to value self-written applications subconsciously higher than "pre-assembled" applications.
I value self-written applications consciously. Because I know everything about them, I can quickly change everything, adapt it as I need, fix any issues. That's not true for other applications and might be impossible for proprietary ones.
I don't think that this is the same about Ikea, though. If I bought Ikea table, I can't really adapt it, unless I'm advanced woodsman with proper tools. If I'm just ordinary guy with screwdriver, I can just follow the script to build that table.
I think that Ikea analogue is someone building his vim environment from other's plugins following guides. He does not know C, so he can't hack vim, he does not know vim configuration language, so he can't hack plugins.
I don't think that this is the same about Ikea, though. If I bought Ikea table, I can't really adapt it, unless I'm advanced woodsman with proper tools. If I'm just ordinary guy with screwdriver, I can just follow the script to build that table.
I think that Ikea analogue is someone building his vim environment from other's plugins following guides. He does not know C, so he can't hack vim, he does not know vim configuration language, so he can't hack plugins.
As a developer you will always use tools and rely on applications others have written. If you write your program in C, then you use a C compiler like gcc and most developers have not read the whole source code of gcc. Same is true for the C library you use, including standard library, you rely on others developers code you have never seen.
Therefore, I think the IKEA effect holds true in software development, because you rely on "pre-assembled" items, but I agree that you have much larger degree or freedom on how you build your application in e.g. C compared to a IKEA build. Maybe, this higher degree of freedom even strengthens the effect.
Therefore, I think the IKEA effect holds true in software development, because you rely on "pre-assembled" items, but I agree that you have much larger degree or freedom on how you build your application in e.g. C compared to a IKEA build. Maybe, this higher degree of freedom even strengthens the effect.
While I don't disagree, there might be many other issues:
- Frameworks and tools are almost always ill-fitting. You have many features you don't need, you lack many features you need. You have dependencies you don't want, code that you don't know, models that might not match your own (and might not be explained well enough, or might not even be consistent with themselves anyway).
- Frameworks and tools are often leaky. They don't solve problems perfectly anyway. They substitute the problem of having to develop a custom solution to the problem with having to figure out how to use the tool effectively, plus any problems derived of the shortcomings of the tool itself, which are often harder to solve on a tool you didn't develop yourself. The better you want to do something, the more disappointed you become with leaky solutions. And other people leaks are far more uncomfortable than your own leaks.
- Some people just want to have control and know what they are doing, even if they need to put much more work on it. Not productive, but more consistent.
- Writing your own tools deepens your understanding of the landscape. This is particularly significant in a world where we have hundreds of frameworks and tools, but very few coherent explanations of the context they work on and the problems they really try to solve. We have reached the point where if someone wants to make a non-trivial website, we tell them to learn React instead of sending them to learn how the internet and websites work so they can choose a tool for their needs afterwards. And this piles up and giving coherent explanations of any landscape becomes harder because no landscape is coherent anymore.
- Related to the previous, there's a certain trend that emerges where the more dependent we become on frameworks and tools, the more the problems in a given scenario are ignored or solved by adding more abstration layers and fancier features on tools, instead of working on the root underlying issues, and the problems become fuzzier. Working on the actual problems by yourself helps you see how much cruft and dirt there is behind even well established technologies... and some of us are masochists and want to know about all that sh*t, because if it remains hidden there's little chance to fix it.
- Frameworks and tools are almost always ill-fitting. You have many features you don't need, you lack many features you need. You have dependencies you don't want, code that you don't know, models that might not match your own (and might not be explained well enough, or might not even be consistent with themselves anyway).
- Frameworks and tools are often leaky. They don't solve problems perfectly anyway. They substitute the problem of having to develop a custom solution to the problem with having to figure out how to use the tool effectively, plus any problems derived of the shortcomings of the tool itself, which are often harder to solve on a tool you didn't develop yourself. The better you want to do something, the more disappointed you become with leaky solutions. And other people leaks are far more uncomfortable than your own leaks.
- Some people just want to have control and know what they are doing, even if they need to put much more work on it. Not productive, but more consistent.
- Writing your own tools deepens your understanding of the landscape. This is particularly significant in a world where we have hundreds of frameworks and tools, but very few coherent explanations of the context they work on and the problems they really try to solve. We have reached the point where if someone wants to make a non-trivial website, we tell them to learn React instead of sending them to learn how the internet and websites work so they can choose a tool for their needs afterwards. And this piles up and giving coherent explanations of any landscape becomes harder because no landscape is coherent anymore.
- Related to the previous, there's a certain trend that emerges where the more dependent we become on frameworks and tools, the more the problems in a given scenario are ignored or solved by adding more abstration layers and fancier features on tools, instead of working on the root underlying issues, and the problems become fuzzier. Working on the actual problems by yourself helps you see how much cruft and dirt there is behind even well established technologies... and some of us are masochists and want to know about all that sh*t, because if it remains hidden there's little chance to fix it.
> Frameworks and tools are often leaky. They don't solve problems perfectly anyway.
How about compilers?
How about compilers?
You seem very focused on languages and compilers. Honestly, those are the basic tools we use as programmers, and they are designed to be turing complete, comfortable to use and very flexible. Yet, we still discuss a lot about them. They are a "tool" that we use a lot, so we are very concerned and particular about them. But yeah, you can't compare frameworks to compilers or programming languages. They are on very different categories.
> But yeah, you can't compare frameworks to compilers or programming languages. They are on very different categories.
That's not what I'm trying to do (to compare them), but people who like to write things from scratch often blame frameworks e.g. they think they are leaky or don't solve problems perfectly anyway.
My question is why they think they can rely on interpreters or compilers, because they also have limitations, bugs, CVEs, leaky abstractions and other known problems. They also form an unknown variable, because most haven't read the source of the compiler or interpreter they use.
That's not what I'm trying to do (to compare them), but people who like to write things from scratch often blame frameworks e.g. they think they are leaky or don't solve problems perfectly anyway.
My question is why they think they can rely on interpreters or compilers, because they also have limitations, bugs, CVEs, leaky abstractions and other known problems. They also form an unknown variable, because most haven't read the source of the compiler or interpreter they use.
What is the likelihood of finding a compiler bug vs a bug in a framework (or a missing method, or undesired side-effect, etc ...) I suspect that for most types of software work many orders of magnitude more for the former. Software developers can go their whole career without encountering a compiler bug and can basically assume it is flawless even they they read blog posts and change-logs informing them that it is not. But they likely will encounter an issue on every single project they have their framework that irks them. Maybe this is just my niche of software though (web-based, enterprise, saas type software). I could imagine those working on system software in lower-level languages fighting the compiler much more.
There has been thousands of issues/bugs over the years in gcc, glibc and the kernel and there are thousands to come, same like in every big software. I don't think there is any evidence that compilers and interpreters are an exception and generally have less bugs than any other piece of software. It is also very much possible to use a non-compiler/interpreter software for years and never encounter a bug.
true, but it is a balance. If you import packages for things that can be done in one small function, you force them a lot of dependencies, which can be worse.
I think this depends on the specifics. It's not always just about maintainability.
You shouldn't reinvent Django if other members of your team already know Django, as you're imposing a learning cost in doing so.
You rather obviously shouldn't reinvent the Linux kernel, as doing so is an enormous undertaking. It's probably a huge waste of resources, your solution will probably be far inferior, and you're likely to fail anyway.
You shouldn't reimplement TLS, as you will almost certainly do so incorrectly, and it would take a great deal of effort. It's a waste of resources and will worsen your cybersecurity risk profile.
You shouldn't reinvent Django if other members of your team already know Django, as you're imposing a learning cost in doing so.
You rather obviously shouldn't reinvent the Linux kernel, as doing so is an enormous undertaking. It's probably a huge waste of resources, your solution will probably be far inferior, and you're likely to fail anyway.
You shouldn't reimplement TLS, as you will almost certainly do so incorrectly, and it would take a great deal of effort. It's a waste of resources and will worsen your cybersecurity risk profile.
I disagree. It's not recommended that you should reimplement TLS. "Shouldn't" is the wrong word in use.
Why should you not? You only learn by developing. Precautions follow with such, as likes not running in production. But how else do you learn if you do not?
If your interested in learning about secure sockets and the rest, you should reimplement TLS. Because that will teach you the inner workings of TLS.
The vibe "you shouldn't" crushes the development side of things.
Why should you not? You only learn by developing. Precautions follow with such, as likes not running in production. But how else do you learn if you do not?
If your interested in learning about secure sockets and the rest, you should reimplement TLS. Because that will teach you the inner workings of TLS.
The vibe "you shouldn't" crushes the development side of things.
Sure, reimplementing TLS is a great way to learn the protocol. We were discussing reimplementing things with an eye to then using that implementation for serious work, but yes, reimplementing as a learning exercise is an exception, provided you never use your toy implementation in production.
> "Shouldn't" is the wrong word to use. Why should you not?
I'll assume that here you're not referring to reimplementing purely as a learning exercise.
I already said why: you will almost certainly do so incorrectly, and this will worsen your cybersecurity risk profile. For serious work, you should use a mature battle-tested implementation like everyone else. The slightest error in a cryptography codebase can lead to severe security vulnerabilities. This is well understood and much has been written on the topic of don't roll your own crypto, e.g. https://security.stackexchange.com/a/18198/
> The vibe "you shouldn't" crushes the development side of things.
No, it's solid cybersecurity advice. It's irresponsible to make use of either an amateur cryptographic scheme, or an amateur implementation of a cryptographic scheme, in a serious codebase. Reimplementing TLS purely as a learning exercise is of course still to be encouraged.
More generally, there is no culture of you shouldn't posing a problem in the software development world. Nuclear engineers and bridge engineers are taught you shouldn't... but software developers are taught just go for it.
> "Shouldn't" is the wrong word to use. Why should you not?
I'll assume that here you're not referring to reimplementing purely as a learning exercise.
I already said why: you will almost certainly do so incorrectly, and this will worsen your cybersecurity risk profile. For serious work, you should use a mature battle-tested implementation like everyone else. The slightest error in a cryptography codebase can lead to severe security vulnerabilities. This is well understood and much has been written on the topic of don't roll your own crypto, e.g. https://security.stackexchange.com/a/18198/
> The vibe "you shouldn't" crushes the development side of things.
No, it's solid cybersecurity advice. It's irresponsible to make use of either an amateur cryptographic scheme, or an amateur implementation of a cryptographic scheme, in a serious codebase. Reimplementing TLS purely as a learning exercise is of course still to be encouraged.
More generally, there is no culture of you shouldn't posing a problem in the software development world. Nuclear engineers and bridge engineers are taught you shouldn't... but software developers are taught just go for it.
Agreeable remarks. I was heading with the educational view point but is still thrown to the educational side just the same. I ask how else do you move to production if you don't take your educational work to the next level?
If I create a mature, well thought piece of encryption. There is no irresponsibly in wanting to use it in production. Yes, I should have it validated and if that validation comes back as something with no thought, buggy and disastrous then that would be irresponsible of me. And one could argue it's irresponsible to run it without validation too. To which I would agree upon.
I still disbelieve that those who create something being told: "you should not be allowed in production ever" and only run "battle-proven" is the wrong approach. Create something, validate it and then run it.
Besides battle-proven systems had to be ran in production in the first place and that those battle-tested systems still have vulnerabilities themselves, ie: HeartBleed. Again which was immune with Mbed TLS/PolarSSL that someone had reimplemented TLS. Validation is the key that's required.
"You should create your own implementation of TLS but you should should not run it in production as that would be irresponsible until you have validation". Not just to blanket "you should t" end of. Above is what should be passed to developers. If you feel your work is good enough, then pay the price for validation.
Besides what makes an expert if you don't make it for yourself?
If I create a mature, well thought piece of encryption. There is no irresponsibly in wanting to use it in production. Yes, I should have it validated and if that validation comes back as something with no thought, buggy and disastrous then that would be irresponsible of me. And one could argue it's irresponsible to run it without validation too. To which I would agree upon.
I still disbelieve that those who create something being told: "you should not be allowed in production ever" and only run "battle-proven" is the wrong approach. Create something, validate it and then run it.
Besides battle-proven systems had to be ran in production in the first place and that those battle-tested systems still have vulnerabilities themselves, ie: HeartBleed. Again which was immune with Mbed TLS/PolarSSL that someone had reimplemented TLS. Validation is the key that's required.
"You should create your own implementation of TLS but you should should not run it in production as that would be irresponsible until you have validation". Not just to blanket "you should t" end of. Above is what should be passed to developers. If you feel your work is good enough, then pay the price for validation.
Besides what makes an expert if you don't make it for yourself?
> I ask how else do you move to production if you don't take your educational work to the next level?
> What makes an expert if you don't make it for yourself?
You study the field, like any other field. This doesn't collide with what I've said.
In this context there needs to be a fairly bright line between learning, and producing real-world cryptographic systems. It might be instructive to have engineering students build an airbag system, but you don't then put it in your car.
> If I create a mature, well thought piece of encryption
Unless you're a professional cryptographer, someone like Bruce Schneier, Tanja Lange, or Filippo Valsorda, it's best to assume that you haven't created a well thought out cryptographic solution. See Schneier's Law. [0] If you have a PhD related to cryptography, and/or a history of employment as a cryptography specialist at a major technology company, then you may have a solid enough grasp of the field to be taken seriously, but short of that, you should leave cryptography to the experts.
It's really hard to get the theory just right, and it's also really hard to get the implementation just right. Fortunately there are existing out-of-the-box solutions that do all the things we want: secure channels, secure file encryption, authentication, etc.
> Yes, I should have it validated
We have a validation process: standards bodies. For instance, in the TLS 1.3 standard, they introduced the requirement for supporting the x25519 algorithm. That algorithm was developed by a team of professional cryptographers, not by a well-meaning dabbler, and it has been subject to careful scrutiny by the cryptographic community.
After standardisation, we see the algorithm implemented in the few trustworthy TLS libraries (e.g. OpenSSL and Google's Tink), which we then adopt for use in the real world.
Serious organisations do not play around with this stuff, they only use trusted standard algorithms. Microsoft/Apple/Amazon/Google have crypto teams who are qualified to write their own implementations of the standard algorithms. The rest of us then use those implementations. Microsoft's Active Directory is backed by the standard Kerberos crypto protocol, for instance.
> I still disbelieve that those who create something being told: "you should not be allowed in production ever" and only run "battle-proven" is the wrong approach. Create something, validate it and then run it.
We agree in a sense, it's just that the bar for considering it battle proven is set very, very high, and for good reason. Developing crypto isn't like styling a webpage with CSS. It's technically challenging to do correctly, it's difficult to know if you've done it correctly, and the consequences of getting it wrong are severe.
> Besides battle-proven systems had to be ran in production in the first place and that those battle-tested systems still have vulnerabilities themselves, ie: HeartBleed. Again which was immune with Mbed TLS/PolarSSL that someone had reimplemented TLS. Validation is the key that's required.
I don't know what 'validation' is meant to mean here. If we had a way to easily detect such issues, we would use it. Again, it's extremely difficult to get this stuff just right. The smallest defect can have terrible consequences.
This applies even when we're doing everything correctly. As you say, we see issues even in major implementations. That doesn't mean that using amateur crypto code is a good idea. It isn't. Every cryptographer agrees that it's a terrible idea to do that. Sometimes aeronautical engineers build dangerous aircraft, but that doesn't mean we let amateurs have a go.
> If you feel your work is good enough, then pay the price for validation.
That isn't how it works. Cryptography is an academic discipline, it makes advances through slow-moving academic publishing and standards bodies, not by paying for a code-review. If you really want to make a contribution to the field, you'll need to make a career of it.
Again though, in a sense there's little need. Much of the best crypto software in the world is Free and Open Source.
If you want to implement TLS as an exercise, or make a neat cryptographic 'toy' program of some sort, then great, but don't gamble anything of value on it (user data, say).
[0] https://www.schneier.com/blog/archives/2011/04/schneiers_law...
> What makes an expert if you don't make it for yourself?
You study the field, like any other field. This doesn't collide with what I've said.
In this context there needs to be a fairly bright line between learning, and producing real-world cryptographic systems. It might be instructive to have engineering students build an airbag system, but you don't then put it in your car.
> If I create a mature, well thought piece of encryption
Unless you're a professional cryptographer, someone like Bruce Schneier, Tanja Lange, or Filippo Valsorda, it's best to assume that you haven't created a well thought out cryptographic solution. See Schneier's Law. [0] If you have a PhD related to cryptography, and/or a history of employment as a cryptography specialist at a major technology company, then you may have a solid enough grasp of the field to be taken seriously, but short of that, you should leave cryptography to the experts.
It's really hard to get the theory just right, and it's also really hard to get the implementation just right. Fortunately there are existing out-of-the-box solutions that do all the things we want: secure channels, secure file encryption, authentication, etc.
> Yes, I should have it validated
We have a validation process: standards bodies. For instance, in the TLS 1.3 standard, they introduced the requirement for supporting the x25519 algorithm. That algorithm was developed by a team of professional cryptographers, not by a well-meaning dabbler, and it has been subject to careful scrutiny by the cryptographic community.
After standardisation, we see the algorithm implemented in the few trustworthy TLS libraries (e.g. OpenSSL and Google's Tink), which we then adopt for use in the real world.
Serious organisations do not play around with this stuff, they only use trusted standard algorithms. Microsoft/Apple/Amazon/Google have crypto teams who are qualified to write their own implementations of the standard algorithms. The rest of us then use those implementations. Microsoft's Active Directory is backed by the standard Kerberos crypto protocol, for instance.
> I still disbelieve that those who create something being told: "you should not be allowed in production ever" and only run "battle-proven" is the wrong approach. Create something, validate it and then run it.
We agree in a sense, it's just that the bar for considering it battle proven is set very, very high, and for good reason. Developing crypto isn't like styling a webpage with CSS. It's technically challenging to do correctly, it's difficult to know if you've done it correctly, and the consequences of getting it wrong are severe.
> Besides battle-proven systems had to be ran in production in the first place and that those battle-tested systems still have vulnerabilities themselves, ie: HeartBleed. Again which was immune with Mbed TLS/PolarSSL that someone had reimplemented TLS. Validation is the key that's required.
I don't know what 'validation' is meant to mean here. If we had a way to easily detect such issues, we would use it. Again, it's extremely difficult to get this stuff just right. The smallest defect can have terrible consequences.
This applies even when we're doing everything correctly. As you say, we see issues even in major implementations. That doesn't mean that using amateur crypto code is a good idea. It isn't. Every cryptographer agrees that it's a terrible idea to do that. Sometimes aeronautical engineers build dangerous aircraft, but that doesn't mean we let amateurs have a go.
> If you feel your work is good enough, then pay the price for validation.
That isn't how it works. Cryptography is an academic discipline, it makes advances through slow-moving academic publishing and standards bodies, not by paying for a code-review. If you really want to make a contribution to the field, you'll need to make a career of it.
Again though, in a sense there's little need. Much of the best crypto software in the world is Free and Open Source.
If you want to implement TLS as an exercise, or make a neat cryptographic 'toy' program of some sort, then great, but don't gamble anything of value on it (user data, say).
[0] https://www.schneier.com/blog/archives/2011/04/schneiers_law...
It is most interesting, especially with regards to other teams.
My mind is primarily focused on the practical viewpoint of a experienced Systems Operator/Admin rather then a Cryptologist.
Lets build our own, make it work and push it live and see what crumbles. I amware that TLS and the likes are all very specialist and to the level as you wouldn't give a Ferrari owner keys to a Jet Fighter. What did spark the flame for me is that be "Free and Opensource" until it comes to crypto which then becomes a touch area of "leave it for the experts" but with full reason makes sense in a way. Maybe because I know little on the matter, that is what makes a large problem.
Learn something new everyday. Thank you for your time on the matter.
My mind is primarily focused on the practical viewpoint of a experienced Systems Operator/Admin rather then a Cryptologist.
Lets build our own, make it work and push it live and see what crumbles. I amware that TLS and the likes are all very specialist and to the level as you wouldn't give a Ferrari owner keys to a Jet Fighter. What did spark the flame for me is that be "Free and Opensource" until it comes to crypto which then becomes a touch area of "leave it for the experts" but with full reason makes sense in a way. Maybe because I know little on the matter, that is what makes a large problem.
Learn something new everyday. Thank you for your time on the matter.
> Lets build our own, make it work and push it live and see what crumbles.
A good attitude for a weekend project, but not a good attitude for cybersecurity.
> "Free and Opensource" until it comes to crypto which then becomes a touch area of "leave it for the experts" but with full reason makes sense in a way.
It's still very beneficial for it to be Free and Open Source. It can be studied by hobbyists and students, it can be audited and analyzed by anyone at all, or even by automated systems.
Glad it's been helpful.
A good attitude for a weekend project, but not a good attitude for cybersecurity.
> "Free and Opensource" until it comes to crypto which then becomes a touch area of "leave it for the experts" but with full reason makes sense in a way.
It's still very beneficial for it to be Free and Open Source. It can be studied by hobbyists and students, it can be audited and analyzed by anyone at all, or even by automated systems.
Glad it's been helpful.
That analogy only stretches so far.
I'd much rather inherit a project that contains 'sprintf("%010d", i)', or equivalent, than one with yet another dependency. Especially when repeated a thousand times over the course on an entire project.
Every dependency is one that I must understand all the possible return types, failure modes, and upgrade paths. Every function is one that I must read. The needle must be quite clearly on the benefits side in order to be worth it.
I'd much rather inherit a project that contains 'sprintf("%010d", i)', or equivalent, than one with yet another dependency. Especially when repeated a thousand times over the course on an entire project.
Every dependency is one that I must understand all the possible return types, failure modes, and upgrade paths. Every function is one that I must read. The needle must be quite clearly on the benefits side in order to be worth it.
If your colleagues and replacements can't maintain the code you wrote, the fact that you didn't use 3rd-party code instead is not the problem.
Addendum: just to clarify, installing 3rd-party library dependencies is a good idea. There are many good reasons to do so. This just isn't one of them.
I've encountered ~100% custom codebases that are unmaintainable, and I've encountered ~100% custom codebases that are extremely maintainable, even for a new person joining a team. And we're not talking about leftPad-esque libs; entire frameworks made in-house.
It's rare, and there are reasons to avoid it (locking yourself into your own tech, not benefiting from community innovation, not being "on trend" with common dev conventions). But ease of maintenance is not one of those reasons. If it's hard to maintain, that's on you.
I've encountered ~100% custom codebases that are unmaintainable, and I've encountered ~100% custom codebases that are extremely maintainable, even for a new person joining a team. And we're not talking about leftPad-esque libs; entire frameworks made in-house.
It's rare, and there are reasons to avoid it (locking yourself into your own tech, not benefiting from community innovation, not being "on trend" with common dev conventions). But ease of maintenance is not one of those reasons. If it's hard to maintain, that's on you.
Well, in that case you should document your code extensively. Document the code in the source code (but please do not have your files in a 90% comment 10% code ratio), and document it elsewhere, too, so your code will not be filled with documentation. I truly hate going through code that has more documentation than code. I often just use a tool that moves all the comments to a separate file.
Maybe it shouldn't be "don't reinvent the wheel", but rather "reinvent the wheel well"
A woodworker would have more off the shelf tools if they were equivalently cheap/good/fast/efficient as a custom tool. Physical objects can’t be so freely copied as a software library.
I do agree fully that programmers should strive to understand every layer, and building your own tools is a great way to get experience that helps with that. Yes, left-pad is unnecessarily granular abstraction. However, in each domain, we inevitably depend on other software, and that breadth of that graph is part of the reason for why you can accomplish so much with so few people writing software.
I do agree fully that programmers should strive to understand every layer, and building your own tools is a great way to get experience that helps with that. Yes, left-pad is unnecessarily granular abstraction. However, in each domain, we inevitably depend on other software, and that breadth of that graph is part of the reason for why you can accomplish so much with so few people writing software.
To be honest, whenever I put up a new table in our house to get some fresh cleared working surface, within a matter of months, said table has accumulated a lot of clutter, spurring me to soon put up another table to get some cleared working space. If only there were some way to return tables to that uncluttered state.
Obligatory: https://xkcd.com/2347/
I am using Hugo for my personal site... and I suffer. If I just disregard strange formats etc. there are just a things that Hugo doesnt do, as a design decision, like start a binary when you are rebuilding the site. Which is to me extremely strange "security" decison that is just making updating site (like putting `software --help` on to a webpage) more annoying and workarounded by bunch of scripts instead relying on hugo only.
I think that over the time all the software starts creeping with bunch of features that most people dont want or need and strange decisions from the past hunts it.
I think that over the time all the software starts creeping with bunch of features that most people dont want or need and strange decisions from the past hunts it.
Few people accept it but most people love reinventing the wheel.
I really hate that expression. Nobody EVER reinvents the wheel. We all know about wheels, and how they work and why they are useful. But we invent new types of wheels all the time. Maybe we put teeth on it and invent the cog. Maybe we invent the tire to improve traction and reduce bumps. Maybe we put spokes between the hub and the rim and to reduce rotating weight. These are all new and useful inventions despite still being wheels.
Sometimes an existing wheel design is perfect for our use case, or at least good enough that creating a new one isn't worth the effort. Sometimes a tweaked wheel is what we need. Sometimes a radical departure from established wheel design is required to make a new machine possible.
Build vs buy is an eternal question, and it's reasonable to examine and reexamine it for any project. But "reinvent the wheel" is just hyperbole.
Sometimes an existing wheel design is perfect for our use case, or at least good enough that creating a new one isn't worth the effort. Sometimes a tweaked wheel is what we need. Sometimes a radical departure from established wheel design is required to make a new machine possible.
Build vs buy is an eternal question, and it's reasonable to examine and reexamine it for any project. But "reinvent the wheel" is just hyperbole.
People absolutely build solutions to problems that are already solved, often worse than the existing solutions.
Your complaint about the phrase seems to be that it doesn't well describe the process of adjusting another design to work better for a specific use-case. But that's not what the phrase is trying to describe, it's literally describing the problem where people reinvent that which absolutely does not need to be reinvented.
Your complaint about the phrase seems to be that it doesn't well describe the process of adjusting another design to work better for a specific use-case. But that's not what the phrase is trying to describe, it's literally describing the problem where people reinvent that which absolutely does not need to be reinvented.
> Nobody EVER reinvents the wheel. We all know about wheels, and how they work and why they are useful.
You greatly overestimate people's knowledge of wheels.
You greatly overestimate people's knowledge of wheels.
An average engineer in any field probably has developed enough systems thinking to reason a great deal about the variation of wheels across applications.
It's the axle that was truly revolutionary.
I've seen many applications re-created feature for feature just to do it in a different language. If it doesn't improve the wheel or solve a new problem, that's reinventing the wheel.
My point was not that nobody ever does unnecessary work. That does happen, obviously, and we can certainly object to it.
In the case of rewriting an application feature for feature in another language, that may or may not be a good idea. Maybe we're switching from an interpreted language to a compiled language because we need better performance. Maybe we're replacing a huge pile of impenetrable bash with a well organized Python app. Maybe we're taking advantage of a library that's only available in another language. Maybe there's a good reason for the rewrite but it's so much work that it's still not worth it. Maybe it's just a engineers wanting to use the latest toys because it'll look good on their resumes.
It's totally legit to question the need for a given project. But the phrase "reinvent the wheel" is just scornful insult. It adds nothing to the discussion, and in fact makes it harder to discuss trade-offs.
In the case of rewriting an application feature for feature in another language, that may or may not be a good idea. Maybe we're switching from an interpreted language to a compiled language because we need better performance. Maybe we're replacing a huge pile of impenetrable bash with a well organized Python app. Maybe we're taking advantage of a library that's only available in another language. Maybe there's a good reason for the rewrite but it's so much work that it's still not worth it. Maybe it's just a engineers wanting to use the latest toys because it'll look good on their resumes.
It's totally legit to question the need for a given project. But the phrase "reinvent the wheel" is just scornful insult. It adds nothing to the discussion, and in fact makes it harder to discuss trade-offs.
Is there a better word than “shitty” - slightly more positive, more positive, something not totally crap, more fun and whole lot of learning.
The thing is, I really really like the word "shitty" here.
Because, let's face it, most software out there (especially most software built by individual hobbyists) is ... let's say not great.
I realize that this is a cultural thing. In the US for example they tend to favor soft language more. They would call it "beta", "early access" or "personal project".
As an European I tend to see it more as... well, "shitty".
And that is fine. It is in fact refreshing not to have to filter out the marketing buzzwords like "innovative" or "minimalistic". The title immediately put me in a good disposition towards the author. It feels honest.
Because, let's face it, most software out there (especially most software built by individual hobbyists) is ... let's say not great.
I realize that this is a cultural thing. In the US for example they tend to favor soft language more. They would call it "beta", "early access" or "personal project".
As an European I tend to see it more as... well, "shitty".
And that is fine. It is in fact refreshing not to have to filter out the marketing buzzwords like "innovative" or "minimalistic". The title immediately put me in a good disposition towards the author. It feels honest.
As a European [1], I have no idea what you refer when you say "especially most software built by individual hobbyists is [...] not great". Have you seen the websites built professionally by consulting agencies? Public administrations? Sure a hobbyist just starting out doesn't write great code, but hobbyists care a lot more about quality and will improve a lot faster than those just slinging sausages or those making the bucks.
When I peek into your average open source project (made, mostly, by hobbyists) I see good to great quality. Whenever any private code gets leaked, or on your average company, or just inspecting websites, you see the horrors. Just my experience here :)
[1] Spaniard, I do believe our websites might be on the bottom of quality of Europe though, so my experience might be different from other "European"s :)
When I peek into your average open source project (made, mostly, by hobbyists) I see good to great quality. Whenever any private code gets leaked, or on your average company, or just inspecting websites, you see the horrors. Just my experience here :)
[1] Spaniard, I do believe our websites might be on the bottom of quality of Europe though, so my experience might be different from other "European"s :)
When I said _individual_ I meant "projects developed by one person only".
In general I agree with you, open source tends to have better quality than proprietary software.
I will point out however that Open Source sees a lot of contribution from non-hobbyists too. That's my case, in fact: my salary for the last 10 years came from working on open source projects.
In general I agree with you, open source tends to have better quality than proprietary software.
I will point out however that Open Source sees a lot of contribution from non-hobbyists too. That's my case, in fact: my salary for the last 10 years came from working on open source projects.
I have a "shitty" static site generator. It's "shitty" because almost no thought or design went into it. It started as 20-30 lines of code and grew into ~1000. It's got hacks, bad decisions, it redoes work like reading files more than ones or reading the files it already wrote.
But, it works, it has rare features I need, features that AFAIK most of the famous ones do not and could not be easily added (well, at least when I checked)
That said, while it started 100% hand written I did add stuff over the years. For example I had my own templating system but switched to 'handlebars' at some point. I don't actually remember if there was a good reason or not. A long time ago I wanted to add RSS/Atom support. Sure that should be easy to do on my own (i've done it in the past) but whatever library I looked into was easy without too many dependencies. The latest, I added jsdom to process a few more automated transformations.
The word "shitty" really does describe the code and I'm embarrassed to show it really but it works and I have better things to do than refactor to make it clean.
But, it works, it has rare features I need, features that AFAIK most of the famous ones do not and could not be easily added (well, at least when I checked)
That said, while it started 100% hand written I did add stuff over the years. For example I had my own templating system but switched to 'handlebars' at some point. I don't actually remember if there was a good reason or not. A long time ago I wanted to add RSS/Atom support. Sure that should be easy to do on my own (i've done it in the past) but whatever library I looked into was easy without too many dependencies. The latest, I added jsdom to process a few more automated transformations.
The word "shitty" really does describe the code and I'm embarrassed to show it really but it works and I have better things to do than refactor to make it clean.
"bespoke" is probably the one you'd want to use.
"Bespoke" implies made to order for someone else.
I would just use "personal".
I would just use "personal".
"Shitty software" was an old saying even back in 1995:
http://scripting.com/davenet/1995/09/03/wemakeshittysoftware...
http://scripting.com/davenet/1995/09/03/wemakeshittysoftware...
"We Make Shitty Software... With Bugs!"
Ha Ha!
Ha Ha!
Hacky
"minimal"?
Purpose-driven? Custom? Personal(ized)? Custom-made?
"mvp"
positively shitty.
err...
err...
Craft?
[deleted]
I spent about a day migrating my personal site from grav to hugo, and the main takeaway I got from that experience is that both of these "simple" static site generators are ridiculously over-engineered.
I could slap something together with a few lines of Make or Python that does exactly what I need, in an hour or two. I could even hook it up to AWS and have it update automatically.
Doing all of that is less time consuming and requires less effort than reading through the grav or hugo documentation, installing and setting everything up, and troubleshooting the inevitable problems.
The only issue I have is that doing front-end web development/design is not something I enjoy, so grav and hugo's existing library of themes is very convenient. Convenient enough to put up with the other annoyances.
I could slap something together with a few lines of Make or Python that does exactly what I need, in an hour or two. I could even hook it up to AWS and have it update automatically.
Doing all of that is less time consuming and requires less effort than reading through the grav or hugo documentation, installing and setting everything up, and troubleshooting the inevitable problems.
The only issue I have is that doing front-end web development/design is not something I enjoy, so grav and hugo's existing library of themes is very convenient. Convenient enough to put up with the other annoyances.
I really dislike this kind of comment. The typical "I could do this myself in a weekend in <a low number> lines of code".
They tend to oversimplify complex problems. They fail to recognize that perhaps they haven't thought of every single twist and turn in the path. These comments believe building something is a straight line, and that because they can think it up and do some napkin math, everyone else's work is invalid.
It's easy to say "I can build x in x time" until the hard work starts and you realize reality isn't that simple.
They tend to oversimplify complex problems. They fail to recognize that perhaps they haven't thought of every single twist and turn in the path. These comments believe building something is a straight line, and that because they can think it up and do some napkin math, everyone else's work is invalid.
It's easy to say "I can build x in x time" until the hard work starts and you realize reality isn't that simple.
yes and no....
they're saying "does exactly what I need". this poster probably doesn't need all the twists and turns that these tools address. many of us could build our own "tool for X" that does just enough of what we need, but it's almost never remotely suitable for use by anyone else, let alone a growing community.
they're saying "does exactly what I need". this poster probably doesn't need all the twists and turns that these tools address. many of us could build our own "tool for X" that does just enough of what we need, but it's almost never remotely suitable for use by anyone else, let alone a growing community.
Yeah I think everyone is right here, it’s as if it’s okay for there being more than one valid solution to a problem
It’s worth pointing out though that hugo has chosen to be a bag of tools to solve everyone’s problem rather than a simple one solution for all approach.
Omakase is sometimes desired when you don’t have the will to put up with another new ecosystem to learn.
Omakase is sometimes desired when you don’t have the will to put up with another new ecosystem to learn.
> I could slap something together with a few lines of Make or Python that does exactly what I need
Here is a static page generator implemented in a 200 LOC Makefile: https://github.com/patrickbr/hagel
Here is a static page generator implemented in a 200 LOC Makefile: https://github.com/patrickbr/hagel
o.O
Edit: Now I see in the readme:
> Error messages are for the most part cryptic errors from sed. Most of them are entirely useless to you.
A quite honest readme & project :- )
@cat $(dir $@).$(notdir $@).$(notdir $<).newlesc.tmp | tr -d '\n' | sed -e 's|\([$(ALLWD_VAR_CHRS)]*\)={{|\n\1={{|g' -e 's|\\\([^n]\)|\\\\\1|g' -e 's|/|\\/|g' -e 's|\&|\\&|g' | sed 's|\([$(ALLWD_VAR_CHRS)]*\)={{\(.*\)}}| s/$$$(strip $(subst .,,$(suffix $(patsubst %.info,%.category,$<)))){\1}/\2/|' > [email protected]
Edit: Now I see in the readme:
> Error messages are for the most part cryptic errors from sed. Most of them are entirely useless to you.
A quite honest readme & project :- )
Static site generator in 130 lines of Python (excluding comments, docstrings, and blank lines): https://github.com/sunainapai/makesite
Disclosure: My wife developed this project.
Disclosure: My wife developed this project.
I thought the same and actually did it. The first version was more or less usable and took just a couple of evenings to put together... I then wanted more and more small features to do basic things like add a ToC, loop through arrays/other pages, add code highlighting to markdown code blocks, then the nitty gritty of fixing bugs when including html in markdown or vice-versa etc... it's been a couple of years now and I use the project in several websites I made, including my personal one... but it's still not in a state I would consider useful for others :D.
“I could totally make a site generator in a couple hours except for the features that are providing me a lot of value” is a really dumb take.
Also ignoring the features that provide other people a lot of value. I used hugo a bit for my site and it seemed like the perfect balance of power and complexity. I actually found hugo to not be quite powerful enough so I added some external pre processing in to my CI config before hugo runs so I could do stuff like have image magic crop images to create thumbnails for the index.
> main takeaway I got from that experience is that both of these "simple" static site generators are ridiculously over-engineered.
Amen - Hugo is a total pain to work with in my experience. Many hours pulling hair out trying to do very simple things (like a page that appears if the user just enters a directory (e.g. "mysite.com/stuff/" and not "mysite.com/stuff/page"))
I found Jekyll to be much, much, much, much easier to work with than Hugo - I ported a site from dokuwiki and with Jekyll I was done within a day (complete with writing a custom theme), but with Hugo I spent days and days just wrestling trying to get basic things to work before I had even started theming before giving up and just using Jekyll.
I am sure there are people who need compilation in 20ms vs 1000ms, but I can wait a couple of seconds if it means I don't have to spend days messing with Hugo just to get it to do basic things.
Amen - Hugo is a total pain to work with in my experience. Many hours pulling hair out trying to do very simple things (like a page that appears if the user just enters a directory (e.g. "mysite.com/stuff/" and not "mysite.com/stuff/page"))
I found Jekyll to be much, much, much, much easier to work with than Hugo - I ported a site from dokuwiki and with Jekyll I was done within a day (complete with writing a custom theme), but with Hugo I spent days and days just wrestling trying to get basic things to work before I had even started theming before giving up and just using Jekyll.
I am sure there are people who need compilation in 20ms vs 1000ms, but I can wait a couple of seconds if it means I don't have to spend days messing with Hugo just to get it to do basic things.
Agreed - I tried to migrate from Jekyll to Hugo just to try to get Ruby off my computer, and I could not believe that I had to use a theme or else everything broke. My Jekyll site just uses bootstrap and a 50-line CSS file.
But if you don't do everything the specific Hugo way - boom everything renders blank, and there are no warnings or errors.
But if you don't do everything the specific Hugo way - boom everything renders blank, and there are no warnings or errors.
Some markdown parsers allow you to take a look at the AST.
I used it to replace a code section containing dot code with a png.
or using comments like <!--more--> to generate site Teasers.
https://github.com/enter-haken/book/blob/master/lib/generato...
You can use pandoc to convert markdown in almost every format you like.
I use it to write letters. https://hake.one/blog/letter.html
If you have access to the AST, you can integrate as much metadata as you like.
I used it to replace a code section containing dot code with a png.
or using comments like <!--more--> to generate site Teasers.
https://github.com/enter-haken/book/blob/master/lib/generato...
You can use pandoc to convert markdown in almost every format you like.
I use it to write letters. https://hake.one/blog/letter.html
If you have access to the AST, you can integrate as much metadata as you like.
Nothing wrong with writing your own SSG (I recently rolled my own solution for my blog as well), but the most important part of the new solution is glossed over by the author: documentation. What's critical, regardless of the solution you use, is to get a runbook of your software.
By way of example, my hand-rolled blog has a README file which lists these things in it. Whenever you pick up a new SSG or create your own you should make sure that you have a good plan for at least these things:
1. How to start writing a new post. For me, that's the script to fill out the template, plus the instructions to run the dev server.
2. Frontmatter fields supported. If there's too many to list all of them, at least list every one you've used before, since you'll probably use those again.
3. Tools for writing. For me, that's a few scripts to do things like properly import media.
4. Publishing. This is a full checklist including "verify mobile", "verify excerpt on posts page", "run publish/deploy", "submit to these sites".
One of the reasons your personal SSG feels better than anything is because you know every nook and cranny in it. That feeling will go away after about 6 months, and you'll thank yourself for having written good documentation.
By way of example, my hand-rolled blog has a README file which lists these things in it. Whenever you pick up a new SSG or create your own you should make sure that you have a good plan for at least these things:
1. How to start writing a new post. For me, that's the script to fill out the template, plus the instructions to run the dev server.
2. Frontmatter fields supported. If there's too many to list all of them, at least list every one you've used before, since you'll probably use those again.
3. Tools for writing. For me, that's a few scripts to do things like properly import media.
4. Publishing. This is a full checklist including "verify mobile", "verify excerpt on posts page", "run publish/deploy", "submit to these sites".
One of the reasons your personal SSG feels better than anything is because you know every nook and cranny in it. That feeling will go away after about 6 months, and you'll thank yourself for having written good documentation.
Agreed. Additionally, this should go for every piece of software you write for yourself, not just your SSG.
I've never heard the term "runbook" before, but it sounds very useful. Thanks!
https://en.wikipedia.org/wiki/Runbook
https://en.wikipedia.org/wiki/Runbook
Like many other in this thread, I too built my own static site. It implements only parts of Jekyll that I use and nothing more: theme and markdown-based content with front matter. It's implemented in Flask using Frozen-Flask and didn't took long to write. I think I spent a weekend to write, tweaked the theme and migrate my contents to it.
I don't have the same need as the author though. If I want to post something that is not a blog post (e.g. a page that showcase a particular personal project), I'll just post it as a standalone page and add whatever custom stuff the standalone page needs in the theme.
I don't have the same need as the author though. If I want to post something that is not a blog post (e.g. a page that showcase a particular personal project), I'll just post it as a standalone page and add whatever custom stuff the standalone page needs in the theme.
I totally agree on this matter. There's something
really empowering about writing your blogging stack
from scratch. I did this a couple years ago
(https://github.com/smasher164/akhil.cc), and I might
even do it again to scale it out.
I wanted to extend markdown with latex and graphviz, and wrote an extensible markdown that grabs output from code execution (https://github.com/smasher164/mexdown).
I wanted to extend markdown with latex and graphviz, and wrote an extensible markdown that grabs output from code execution (https://github.com/smasher164/mexdown).
It's good author wrote his own static site generator. If he implemented using restructured text specification it will be future proof and maintainable along with using multiple tools like pandoc, sphinx, pelican, nikola and obviously python docutils itself for generating different format outputs.
Eventhough markdown is popular, its not a standard speicification. If author wanted to work with something which works for generating output for a website, pdf file or an ebook, he should have looked into a proper standard like RestructuredText (RST).
I think rst is much cleaner and better specification than markdown. Given its a specification there are multiple implementations available. So may be community should adopt it with more open arms. Still wonder like xsd vs relaxng in this case also markdown being inferior implementation without specification is winning over rst (restructuredText) format and specification.
Eventhough markdown is popular, its not a standard speicification. If author wanted to work with something which works for generating output for a website, pdf file or an ebook, he should have looked into a proper standard like RestructuredText (RST).
I think rst is much cleaner and better specification than markdown. Given its a specification there are multiple implementations available. So may be community should adopt it with more open arms. Still wonder like xsd vs relaxng in this case also markdown being inferior implementation without specification is winning over rst (restructuredText) format and specification.
I find RST simply miserable to write and even worse to read. All of your points are valid regarding it being an actual spec, but reading and writing it in the raw vexes me to no end.
I still use it for my module documentation, but I still use Jekyll for my personal sites instead of something like Sphinx (which is another element of hell for me - the errors when your RST is wrong are worse than useless;
> "You have an error somewhere between _here_ and _here_"
"You just pointed to everything"
> "Yes I did"
I still use it for my module documentation, but I still use Jekyll for my personal sites instead of something like Sphinx (which is another element of hell for me - the errors when your RST is wrong are worse than useless;
> "You have an error somewhere between _here_ and _here_"
"You just pointed to everything"
> "Yes I did"
AsciiDoc(tor) is worth a look, it's better specified than Markdown (and has much features), but doesn't look as ugly as restructuredText.
It's quite niche, though.
And if you don't want the Ruby dependency, AsciiDoctorJ is very usable.
It's quite niche, though.
And if you don't want the Ruby dependency, AsciiDoctorJ is very usable.
Thanks for writing this. Though the post was a bit too long for me, I skimmed through it and got many relevant points. I’ve been thinking about using an SSG, but have found it difficult to decide on which one would be easy to use and also less restrictive for my (simple) needs.
The trade offs between any SSG and platforms like WordPress or Twitter are clearer. They sit on a spectrum of predefined needs, and figuring out what matches your needs is the key (and also the cause for analysis paralysis).
The trade offs between any SSG and platforms like WordPress or Twitter are clearer. They sit on a spectrum of predefined needs, and figuring out what matches your needs is the key (and also the cause for analysis paralysis).
Just try Hugo. Jekyll is okay but extremely slow with hundreds of Ruby plug-in dependencies and nonsensical breaking changes between versions.
The author is pedantic and seems to just not want to use other tools. That’s fine. But this is a dumb road for others to follow.
Remember the entire thing started because they don’t like front matter. They could literally have written a script to parse front matter, which are literally a few plain text lines.
Then they wanted asciidoc but without the dependency. Except their own code will likely have an asciidoc dependency unless they wrote their own entire asciidoc parser. In which case they could also have done this and plugged it into the same script.
“You don’t want to get stuck into the thinking of others,” they scream into the wires, followed by hobbling themselves into the most minimal viable product straight jacket imaginable.
Fool programmers and their time are soon parted.
The author is pedantic and seems to just not want to use other tools. That’s fine. But this is a dumb road for others to follow.
Remember the entire thing started because they don’t like front matter. They could literally have written a script to parse front matter, which are literally a few plain text lines.
Then they wanted asciidoc but without the dependency. Except their own code will likely have an asciidoc dependency unless they wrote their own entire asciidoc parser. In which case they could also have done this and plugged it into the same script.
“You don’t want to get stuck into the thinking of others,” they scream into the wires, followed by hobbling themselves into the most minimal viable product straight jacket imaginable.
Fool programmers and their time are soon parted.
This post kinda reflects my thoughts as well. Picked it as a hobby to explore writing a static site generator. I wrote mine in nodejs. https://github.com/rammygit/homegrown , it again a shitty SSG but again i enjoyed the process of exploring
[deleted]
Nice. That gave me inspiration to finish my shitty SSG. It already powers my personal blog and my wife’s as well but it need some polish for a full launch
For me, the reason was simpler: it took less effort to write an SSG than it did to learn any existing one, and I got the fun of doing it.
I have yet to start the blog I've been thinking about for 10 years. But I fully intend to build my own SSG for it. I gave a lot of thought to my motivation for doing so, and reached a simple conclusion: programmers gotta program. A blog is all about self expression, and the purest form of that is the code that drives the blog.
[deleted]
I've been looking for static site generator myself and, as often with OSS, found myself overwhelmed with the offer of seemingly identical yet subtly different generators. There's the old xkcd joke about solving the standards problem by creating an Nth one, but really that might be better than decision paralysis.
I agree 100%, but more than anything for me was control: I had been using nanoc (https://nanoc.ws) for years and don't get me wrong, that's an excellent piece of software... but then I "left it" for a few months also because I stopped coding in Ruby for a while (and stopped blogging) .When I came back to it, I discovered that a new major version was out and I had to upgrade, potentially breaking quite a bit of things because I didn't keep the pace with the new changes.
While this is true for nearly all software, I missed being in control of when to upgrade, and I also missed a few little features here and there. The result? I shopped around for it a bit and then decided to roll out my own: HastySite (https://hastysite.h3rald.com) which now powers a bunch of sites of mine like https://h3rald.com and https://min-lang.org
Actually I ended up writing my own markdown processor and my own programming language before doing that (which also needed a Readline/line noise replacement, and wrappers for a regexp and compression library) but... hey, that's part of the fun isn't it? That's the very reason why I love being a programmer: it's not about the final result, it's about how you get there and what you learn along the way.
So kudos to you sir, I wish you all the best and that you build more things yourself.
While this is true for nearly all software, I missed being in control of when to upgrade, and I also missed a few little features here and there. The result? I shopped around for it a bit and then decided to roll out my own: HastySite (https://hastysite.h3rald.com) which now powers a bunch of sites of mine like https://h3rald.com and https://min-lang.org
Actually I ended up writing my own markdown processor and my own programming language before doing that (which also needed a Readline/line noise replacement, and wrappers for a regexp and compression library) but... hey, that's part of the fun isn't it? That's the very reason why I love being a programmer: it's not about the final result, it's about how you get there and what you learn along the way.
So kudos to you sir, I wish you all the best and that you build more things yourself.
Why did you have to upgrade? You can pin any version you like in your Gemfile and use it in perpetuity.
[deleted]
I too have a shitty static site generator.
My issue was mostly long term backup. When you think about keeping your writing around decades--ie, longer than you might keep a site--there's no solution I loved.
I also love Scrivener for actually writing. So I built a SSG that takes exports from Scrivener and turns that into webpages.
I get to use my writing tool of choice + I have full backups and the ability to recreate the site on demand.
Also, Scrivener has metadata as part of the UI, so no awkward front matter. Just some custom fields (as the software already expects).
My issue was mostly long term backup. When you think about keeping your writing around decades--ie, longer than you might keep a site--there's no solution I loved.
I also love Scrivener for actually writing. So I built a SSG that takes exports from Scrivener and turns that into webpages.
I get to use my writing tool of choice + I have full backups and the ability to recreate the site on demand.
Also, Scrivener has metadata as part of the UI, so no awkward front matter. Just some custom fields (as the software already expects).
I'm surprised there aren't more SSCs (shitty Scrivener clones). Scrivener is by far my favorite writing tool, but it's also the tool that most frequently makes me wish I had the time to write my own version from scratch.
[deleted]
I built a shitty SSG in QuickBasic in 1996 to run what is now http://onlineslangdictionary.com/ . Pre-XML! It used a “bespoke” (read: shitty) markup language.
Definitely a fun exercise. I bet I still have the code around somewhere.
(The QB version was succeeded by a series of shitty SSGs: one written in Visual Basic, one written in Java, and one written in C#, before the current iteration which is backed by a dynamic website.)
Definitely a fun exercise. I bet I still have the code around somewhere.
(The QB version was succeeded by a series of shitty SSGs: one written in Visual Basic, one written in Java, and one written in C#, before the current iteration which is backed by a dynamic website.)
[deleted]
Not applicable to the author’s reasons for building, but I wonder if “I built my own SSG” is the equivalent of “I built my own CMS” for this generation.
It's funny, I also ended up building a custom static site generator and it was actually really simple to do so!
When I initially started googling around for custom static site generators, I could not find much content so I just wrote a quick post on it: https://smalldata.tech/blog/2018/08/16/building-a-simple-sta...
When I initially started googling around for custom static site generators, I could not find much content so I just wrote a quick post on it: https://smalldata.tech/blog/2018/08/16/building-a-simple-sta...
Like many other here I looked at a lot of options and ended up writing my own, because spending time finding the right solution (if it existed) was more effort than just doing it myself.
Inspired by https://github.com/giuseppeg/xm I ended up writing a absolutely minimal no-frills PHP snippet to turn the PHP file itself in to an html page.
https://github.com/gnyman/piss / https://blog.nyman.re/2020/10/11/introducing-piss-a.html
Doing it yourself is of course not an option for most people. But as the author and many other commenters here, programming is a hobby so the journey is as important as the destination.
Inspired by https://github.com/giuseppeg/xm I ended up writing a absolutely minimal no-frills PHP snippet to turn the PHP file itself in to an html page.
https://github.com/gnyman/piss / https://blog.nyman.re/2020/10/11/introducing-piss-a.html
Doing it yourself is of course not an option for most people. But as the author and many other commenters here, programming is a hobby so the journey is as important as the destination.
Have you looked at Lektor? https://www.getlektor.com/ . We use it for in-house needs and we have also built an extension, Lektorium https://github.com/sphericalpm/lektorium that allows us to create hosted instances of Lektor for content managers who aren't technical enough to run Lektor locally.
TiddlyWiki (https://tiddlywiki.com), which I think is amazing both in simplicity and flexibility, also offers export to static sites. There are two versions: either the wiki as a single html+js-file or on node.js, but export to static html exists in both. You can test it from the home page (https://tiddlywiki.com), just go to Tools in the sidebar -> Export all -> Static HTML. Amazingly simple.
Not mine, but a (nice) demo: https://www.mentalnodes.com/ and more advanced guide: https://nesslabs.com/tiddlywiki-static-website-generator
Not mine, but a (nice) demo: https://www.mentalnodes.com/ and more advanced guide: https://nesslabs.com/tiddlywiki-static-website-generator
> I want to have one folder for storing all my writing. I want to organize that folder the way I think fits best with the content. Then I want to point the site generator to that folder and it should figure out the rest by itself.
Same. I think lots of people would appreciate this. Writing is naturally private (in draft) before it may become public (published).
I am tempted to follow the author's path. But I don't think I need to go the whole way. I just need a preprocessor of sorts. I have a folder of markdown files I edit with Zettlr. I could flip a `published: true` flag in the frontmatter. Then iterate over each file and conditionally copy them somewhere to be dealt with by an off-the-shelf SSG (11ty is the current favourite).
Same. I think lots of people would appreciate this. Writing is naturally private (in draft) before it may become public (published).
I am tempted to follow the author's path. But I don't think I need to go the whole way. I just need a preprocessor of sorts. I have a folder of markdown files I edit with Zettlr. I could flip a `published: true` flag in the frontmatter. Then iterate over each file and conditionally copy them somewhere to be dealt with by an off-the-shelf SSG (11ty is the current favourite).
I also have my own SSG that is basically a little bash script that turns Markdown files to HTML files via 'multimarkdown'. Plus a bunch of PHP and JS scripts:
* Extract meta data like categories
* TOC generator
* Building a search index for Algolia
I used 11ty for a project recently and liked it a lot. It's very simple and the nice thing is that it understands several templating languages out of the box. You can basically mix them as well if you want or simply pick your favorite. Would use again: https://www.11ty.dev/
* Extract meta data like categories
* TOC generator
* Building a search index for Algolia
I used 11ty for a project recently and liked it a lot. It's very simple and the nice thing is that it understands several templating languages out of the box. You can basically mix them as well if you want or simply pick your favorite. Would use again: https://www.11ty.dev/
I may consider eleventy in the future as it also supports EJS. It is the template language that I am most familiar, no context switching for my brain. My blogging platform is simply some EJS files glued together with Express. super simple.
I built my own one too. Then I realised I reinvented server-side includes again. So I just went and used those. Again.
For more than 25 years, I am simply writing HTML with only a little CSS and some JavaScript. During that time, I have developed a link checking program, which helps me fixing links and updating the tags. It is a simple C program and rather specific for my needs: https://www.iwriteiam.nl/chkhtml_c.txt
It also moves all files that need to be updated to an ftp folder for easy uploading to my hoster.
I also built (and abandoned!) a small static site generator, and it was as well a very surprisingly pleasant experience. Unless you are doing something very different on purpose, all of the underlying blocks already exist somewhere. You just need to find where and exactly how to combine them, so you are basically writing glue code. It feels even more like a superpower compared to normal programming, you write a few lines and suddenly you have this huge feature working quite well.
I wrote a ton of SSG myself. I'm going to talk about three of them.
The first is fugitive [1]. It came to life because I just learned about git hooks and thought of using them for generating a blog with all of its meta data in Git itself. I wanted to have only Git as a dependency so I wrote the whole thing in pure sh (not even using some convenient Bash features). It was for fun and for just myself initially but it had maybe like two dozen users in the end, some of them even contributed to the project or maintained their own forks.
The second one is not released. It's what I use for my own web page [2]. It's a make file + a few Bash scripts that make use of xml2 parse an HTML template and content files, then uses the GNU coreutils (mostly sed) to insert the content in the template and adjust a few things, and finally use 2xml to build the final HTML document. It also has two additional scripts: one that can take an XML file and build either an RSS feed or an HTML output to be inserted into a page, and one that can build an HTML representation of my list of publications.
The third and most recent one is called cigala [3]. It was written for people around me with no web development experience but who needed a simple web page (typically people in academia outside CS). It's a single PHP file that uses SQLite to store the CMS content and is able to build static HTML files whe you want to publish your website. It was originally run from my own server where I created accounts manually each time someone asked me to. But it now is available for larger use and is easy to install as well.
[1] https://code.up8.edu/pablo/fugitive
[2] https://pablo.rauzy.name/
[3] https://code.up8.edu/pablo/cigala
The first is fugitive [1]. It came to life because I just learned about git hooks and thought of using them for generating a blog with all of its meta data in Git itself. I wanted to have only Git as a dependency so I wrote the whole thing in pure sh (not even using some convenient Bash features). It was for fun and for just myself initially but it had maybe like two dozen users in the end, some of them even contributed to the project or maintained their own forks.
The second one is not released. It's what I use for my own web page [2]. It's a make file + a few Bash scripts that make use of xml2 parse an HTML template and content files, then uses the GNU coreutils (mostly sed) to insert the content in the template and adjust a few things, and finally use 2xml to build the final HTML document. It also has two additional scripts: one that can take an XML file and build either an RSS feed or an HTML output to be inserted into a page, and one that can build an HTML representation of my list of publications.
The third and most recent one is called cigala [3]. It was written for people around me with no web development experience but who needed a simple web page (typically people in academia outside CS). It's a single PHP file that uses SQLite to store the CMS content and is able to build static HTML files whe you want to publish your website. It was originally run from my own server where I created accounts manually each time someone asked me to. But it now is available for larger use and is easy to install as well.
[1] https://code.up8.edu/pablo/fugitive
[2] https://pablo.rauzy.name/
[3] https://code.up8.edu/pablo/cigala
I built a shitty static site generator, too!
I had no idea what I was doing. Didn't look at how other SSG's work... I just made some template HTML files, stuffed some content in YAML/Markdown frankenstein files, and then used JavaScript string.split().join() calls to stitch together the output files
It worked! It is very satisfying to have one build command that generates the entire site and pushes it to gh-pages for hosting.
I had no idea what I was doing. Didn't look at how other SSG's work... I just made some template HTML files, stuffed some content in YAML/Markdown frankenstein files, and then used JavaScript string.split().join() calls to stitch together the output files
It worked! It is very satisfying to have one build command that generates the entire site and pushes it to gh-pages for hosting.
Hey, what a coincidence. I've recently did some updates on my own shitty static site generator: https://santiagodevelopment.com
In my case, I stuck with markdown and still used frontmatter to add metadata to posts. Since I'm using asp.net core, I can use the markdown library Markdig which supports frontmatter, but I can also just fallback to the name of the markdown file (regex out the date and title), since I don't need much information from metadata. Adding middleware to asp.net core is fairly easy as well, so I can simply create some simple middleware to create static files as requests come through.
The only annoying case I had was how to manage syncing between git or gitlab (I created post retrieval clients for both) and already generated static files. I have a fairly forced method at the moment, where almost every generated file is wiped when a new git commit shows up, but I get to find those solutions myself since it's mine (!).
Definitely agree with the sentiments here. Although, I feel that the fact there's already working products out there makes it less incentivized to do projects like this. The new programmers I know seem to be more interested in doing something new or creating software you can't just get from some SaaS. I guess when I created my first website I didn't have that worry, there was no squarespace or whatever else, so there was the double benefit of creating something both simple and new.
In my case, I stuck with markdown and still used frontmatter to add metadata to posts. Since I'm using asp.net core, I can use the markdown library Markdig which supports frontmatter, but I can also just fallback to the name of the markdown file (regex out the date and title), since I don't need much information from metadata. Adding middleware to asp.net core is fairly easy as well, so I can simply create some simple middleware to create static files as requests come through.
The only annoying case I had was how to manage syncing between git or gitlab (I created post retrieval clients for both) and already generated static files. I have a fairly forced method at the moment, where almost every generated file is wiped when a new git commit shows up, but I get to find those solutions myself since it's mine (!).
Definitely agree with the sentiments here. Although, I feel that the fact there's already working products out there makes it less incentivized to do projects like this. The new programmers I know seem to be more interested in doing something new or creating software you can't just get from some SaaS. I guess when I created my first website I didn't have that worry, there was no squarespace or whatever else, so there was the double benefit of creating something both simple and new.
plot of "Number of blog posts" vs "Number of blog posts about elaborate blog setup":
https://twitter.com/jxxf/status/1332839627170967554
https://twitter.com/jxxf/status/1332839627170967554
I used to have my own and it was very... hacky.
Basically it was a PHP website where there would be snippets of PHP at the top of the page followed by html with the occasional `<?php echo $var; ?>`.
Then I wrote a simple bash script that curled the website from localhost and saved everything to a local folder (kind of like how Jekyll has _site). Then the script used the neocities gem [1] to automatically upload.
I now use Jekyll, but I've thought about making my own because only I will use it and I'll understand it from top to bottom. With Jekyll whenever I need to make a change I have to re-read the docs to figure out what I need to do.
[1]: https://github.com/neocities/neocities-ruby
Basically it was a PHP website where there would be snippets of PHP at the top of the page followed by html with the occasional `<?php echo $var; ?>`.
Then I wrote a simple bash script that curled the website from localhost and saved everything to a local folder (kind of like how Jekyll has _site). Then the script used the neocities gem [1] to automatically upload.
I now use Jekyll, but I've thought about making my own because only I will use it and I'll understand it from top to bottom. With Jekyll whenever I need to make a change I have to re-read the docs to figure out what I need to do.
[1]: https://github.com/neocities/neocities-ruby
Well I certainly never expected my project's name to conflict with another "shitty static site generator" haha. I ended up swapping "shitty" for "stupid" however: https://github.com/esell/sssg
I started down the Go template route and quickly realized that pandoc + some other standard tools got me everything _I_ needed. It was a great learning experience for sure!
I started down the Go template route and quickly realized that pandoc + some other standard tools got me everything _I_ needed. It was a great learning experience for sure!
I wrote three
The first one was some kind of macro processor to insert text in files. I do not remember the syntax, but in each file it was like: %headermacro% ...the content... %footer%. Then the macros were replaced by layout data, turning each file into a generated static page.
Then I realized that is the wrong way around. All the content files had the same macros. It is very redundant. So I wrote a new one, that did it the other way around. Now there was a layout file with placeholders {{$WRITEX content/data}} and it would load the data from an XML content file and insert it in the layout file to generate the output file. And it had loops and conditions. Afair I could even write my "Makefile" in that language.
Then I wrote a webscraper and implemented XPath 2, because nothing else supported that. Then XPath 2 is very close to XQuery, so I implemented that as well. Now XQuery is very similar to the second generator, but with a w3 standarized syntax. The above insert expression simply becomes {content/data}. So now I use the webscraper as site generator.
The first one was some kind of macro processor to insert text in files. I do not remember the syntax, but in each file it was like: %headermacro% ...the content... %footer%. Then the macros were replaced by layout data, turning each file into a generated static page.
Then I realized that is the wrong way around. All the content files had the same macros. It is very redundant. So I wrote a new one, that did it the other way around. Now there was a layout file with placeholders {{$WRITEX content/data}} and it would load the data from an XML content file and insert it in the layout file to generate the output file. And it had loops and conditions. Afair I could even write my "Makefile" in that language.
Then I wrote a webscraper and implemented XPath 2, because nothing else supported that. Then XPath 2 is very close to XQuery, so I implemented that as well. Now XQuery is very similar to the second generator, but with a w3 standarized syntax. The above insert expression simply becomes {content/data}. So now I use the webscraper as site generator.
I use a few different languages to various levels of competence, and I always do a static site generator in each one as my sample project that lets me get a feel for their differences and strengths.
So I currently have my own generators written in .Net Core, Go, Node, Python, Ruby, and PHP.
Of them all, Go is by far the fastest, Node the simplest, .Net Core the cleanest, PHP the hackiest, Python the nicest, but Ruby the most balanced in all those areas.
The Ruby one uses Markdown with YAML frontmatter, all in one script file that can be dropped directly into any folder and committed as part of my sites.
If you're curious (and remember the Ruby one is not the best) you can see it in github at https://github.com/kcartlidge/ruthless and running my site at https://kcartlidge.com (using Netlify).
So I currently have my own generators written in .Net Core, Go, Node, Python, Ruby, and PHP.
Of them all, Go is by far the fastest, Node the simplest, .Net Core the cleanest, PHP the hackiest, Python the nicest, but Ruby the most balanced in all those areas.
The Ruby one uses Markdown with YAML frontmatter, all in one script file that can be dropped directly into any folder and committed as part of my sites.
If you're curious (and remember the Ruby one is not the best) you can see it in github at https://github.com/kcartlidge/ruthless and running my site at https://kcartlidge.com (using Netlify).