Brian Kernighan on the Typesetting of “The Go Programming Language” Book(rkrishnan.org)
rkrishnan.org
Brian Kernighan on the Typesetting of “The Go Programming Language” Book
https://rkrishnan.org/posts/2016-03-07-how-is-gopl-typeset.html
75 comments
As a typesetting aficionado, I have tried most of the suggestions here (Prince, Troff, Lout, Docbook XML, Asciidoc, LaTeX, ConTeXt, .. except GUI typesetting tools.)
TeX appears bad from a distance, mostly from hearsay, until you have to write an actual technical book of certain expected high production standards and you realize TeX gives you the best shot at doing so.
There is a learning curve. It is an investment of your time, but it pays off.
Also, it is not like LaTeX et al are standing still in time. There are companies and teams working on improving the ecosystem everyday. I personally like ConTeXt unless I'm doing a predefined template exists (for conferences etc.,) where it makes sense to just use their LaTeX guidelines.
TeX appears bad from a distance, mostly from hearsay, until you have to write an actual technical book of certain expected high production standards and you realize TeX gives you the best shot at doing so.
There is a learning curve. It is an investment of your time, but it pays off.
Also, it is not like LaTeX et al are standing still in time. There are companies and teams working on improving the ecosystem everyday. I personally like ConTeXt unless I'm doing a predefined template exists (for conferences etc.,) where it makes sense to just use their LaTeX guidelines.
Of recent attempts at dethroning LaTeX, there's also SILE, which actually reuses some LaTeX internals. For some examples of the output (hard to find, the author seems not great at marketing), see e.g.:
https://github.com/simoncozens/sile/blob/master/examples/ara...
https://github.com/simoncozens/sile/blob/master/examples/art...
https://github.com/simoncozens/sile/blob/master/examples/fra...
https://github.com/simoncozens/sile/blob/master/examples/par...
https://github.com/simoncozens/sile/blob/master/examples/jap...
https://github.com/simoncozens/sile/blob/master/examples/tes...
For a deeper dive, see the author's presentation at FOSDEM 2015 - I found it really amusing - from the starting line of: "I wrote a typesetting system - by mistake."
https://www.youtube.com/watch?v=5BIP_N9qQm4
https://github.com/simoncozens/sile/blob/master/examples/ara...
https://github.com/simoncozens/sile/blob/master/examples/art...
https://github.com/simoncozens/sile/blob/master/examples/fra...
https://github.com/simoncozens/sile/blob/master/examples/par...
https://github.com/simoncozens/sile/blob/master/examples/jap...
https://github.com/simoncozens/sile/blob/master/examples/tes...
For a deeper dive, see the author's presentation at FOSDEM 2015 - I found it really amusing - from the starting line of: "I wrote a typesetting system - by mistake."
https://www.youtube.com/watch?v=5BIP_N9qQm4
It doesn't do math though.
yep, that's a significant and notable TODO. Though I have an idea that maybe trying automatic translation of MathJax from JS to Lua using one of the available tools could have a chance of being a win as a relatively quick hack. But I couldn't find time yet to compile SILE itself first, as I don't have a Linux box handy, unfortunately :(
Care to comment on the reasons for using ConTeXt versus LaTeX? I've been curious about ConTeXt for a while, but I fall back on LaTeX out of familiarity (and the need to use provided document classes.)
LaTeX's selling point is, it allows someone(publisher, conference committee, school, professor) to set what is acceptable typesetting and rest of us can focus on the content.
ConTeXt is excellent if you care about design -- visual, layout, use of non-traditional typeface families, non-traditional print medium, create magazines, glossies, etc.,.
Writing a new style, class etc., for LaTeX feels like it's for "experts only". On the other hand, extending and creating new styles etc., for ConTeXt is much more natural part of creating the document.
ConTeXt is also highly programmable and extensible using Lua, XML(think machine generated catalogue).
With Metapost (graphics language), you can create highly sophistical vector diagrams. I have found pgf/tikz to be highly opinionated and it take much more upfront learning compared to metapost.
ConTeXt is also more intuitive, IMHO.
ConTeXt is excellent if you care about design -- visual, layout, use of non-traditional typeface families, non-traditional print medium, create magazines, glossies, etc.,.
Writing a new style, class etc., for LaTeX feels like it's for "experts only". On the other hand, extending and creating new styles etc., for ConTeXt is much more natural part of creating the document.
ConTeXt is also highly programmable and extensible using Lua, XML(think machine generated catalogue).
With Metapost (graphics language), you can create highly sophistical vector diagrams. I have found pgf/tikz to be highly opinionated and it take much more upfront learning compared to metapost.
ConTeXt is also more intuitive, IMHO.
I like LaTeX for formatting small works. But would a publisher be willing to take LaTeX input when producing a book?
many scientific and tech publishing houses have their own LaTeX styles (eg: MIT, nostarch).
They've been a pain in the ass about it IME, but you can convert it to markdown for editing, then PDF for pre-print.
Sadly I have to work in Windows at times. Context seems to not be the most supported Windows platform. (The link to the Windows simplified package is 404) There is a New Installer http://wiki.contextgarden.net/Windows_Installation_new which does not contain a working binary. The only way to find anything for windows leads to a tar file.
I still think there is a better way to do typesetting in 2016 but all I have seen are a bunch of smaller projects without much support.
Personally I would love a markup language redefined with built in PDF-X support and a simple infrastructure. Context is Ruby, Tex, and ConText. I find LaTex to work well (for the most part) but it is not very flexible but I found the MikText Package Manager as a big step forward.
I still think there is a better way to do typesetting in 2016 but all I have seen are a bunch of smaller projects without much support.
Personally I would love a markup language redefined with built in PDF-X support and a simple infrastructure. Context is Ruby, Tex, and ConText. I find LaTex to work well (for the most part) but it is not very flexible but I found the MikText Package Manager as a big step forward.
Both the TeX Live and MiKTeX distributions provide binaries for Context on Windows.
I wonder if it works under the new Linux subsystem.
I thought ConText was using Lua for scripting things.
This suggests that:
http://wiki.contextgarden.net/Mark_IV
This suggests that:
http://wiki.contextgarden.net/Mark_IV
I wish I could answer this.. but I'm a Mac and Linux user.
with the new "Bash for windows", it will hopefully get easier in future.
with the new "Bash for windows", it will hopefully get easier in future.
Well the BIG issue is you can't create new files on Window files only save as. I use virtual machines but my Window computers at work are long in the tooth machines.
> Also, it is not like LaTeX et al are standing still in time.
Indeed -- XeTeX (and XeLaTeX) support Unicode and OpenType fonts natively (including system installed fonts). It's a pleasure to use.
Indeed -- XeTeX (and XeLaTeX) support Unicode and OpenType fonts natively (including system installed fonts). It's a pleasure to use.
I liked TeX for typesetting my thesis, but I feel like, looking back, it would be nearly impossible to convert my document to a reflowable format. Not sure what can be done about that if you want nice typesetting in a fixed size environment, though. I'm curious to what extent the dual-format problem has been addressed.
Ironically this page typesets Prof. Kernighan's email in a <pre> tag so mobile users could enjoy scrolling left and right 100 times to read it.
Same here, to help future mobile readers:
Hi, Ramakrishnan --
Thanks for the kind words on the typography. The core formatting is just troff (groff, really) with the -ms macro package. We tried Latex briefly but neither Alan or I care for its output, and I personally find it impossible to control. Troff is the devil I know -- it's ugly and irregular in many ways but it will put characters where you want them.
The input was in XML, with a tag set of about 25 items for headings, paragraphs, index terms, program insertion, simple tables, and the like. A Go program converted this either into HTML for rapid viewing on the screen and potentially for an e-book version, or into troff for printing. Using XML was a mild nuisance when writing but the error checking was very helpful.
We wrote a variety of small Go programs and scripts to fix things up, including one that rewrites troff output to do vertical justification so all pages are the same height. There is also some fiddling with the generated postscript to put on printer's marks.
The fonts are Alan's choice, the result of a lot of work on his part to find them and get the right sizes and appearances. The handful of Asian characters were tough to get right; troff doesn't do wide Unicode characters properly, and there are a few places where we rewrote text to hide that fact.
The drawings are all Alan's work, using Google's drawing program. We toyed briefly with using pic but it wasn't really up to the job, and it would not have worked with HTML without a lot of work. We avoided mathematics beyond superscripts, and the tables are pretty limited; eqn and tbl would have been ok but again would not have dealt with HTML.
We should probably write a more organized description of what we did and make the tools available, though I think most readers are less interested in the process than you (and I) are. The other thing is that although one starts with grand ideas of being clean and orderly, by the end the process is somewhat of a mess, with a complicated makefile to keep it running.
Thanks again for writing.
Brian
Hi, Ramakrishnan --
Thanks for the kind words on the typography. The core formatting is just troff (groff, really) with the -ms macro package. We tried Latex briefly but neither Alan or I care for its output, and I personally find it impossible to control. Troff is the devil I know -- it's ugly and irregular in many ways but it will put characters where you want them.
The input was in XML, with a tag set of about 25 items for headings, paragraphs, index terms, program insertion, simple tables, and the like. A Go program converted this either into HTML for rapid viewing on the screen and potentially for an e-book version, or into troff for printing. Using XML was a mild nuisance when writing but the error checking was very helpful.
We wrote a variety of small Go programs and scripts to fix things up, including one that rewrites troff output to do vertical justification so all pages are the same height. There is also some fiddling with the generated postscript to put on printer's marks.
The fonts are Alan's choice, the result of a lot of work on his part to find them and get the right sizes and appearances. The handful of Asian characters were tough to get right; troff doesn't do wide Unicode characters properly, and there are a few places where we rewrote text to hide that fact.
The drawings are all Alan's work, using Google's drawing program. We toyed briefly with using pic but it wasn't really up to the job, and it would not have worked with HTML without a lot of work. We avoided mathematics beyond superscripts, and the tables are pretty limited; eqn and tbl would have been ok but again would not have dealt with HTML.
We should probably write a more organized description of what we did and make the tools available, though I think most readers are less interested in the process than you (and I) are. The other thing is that although one starts with grand ideas of being clean and orderly, by the end the process is somewhat of a mess, with a complicated makefile to keep it running.
Thanks again for writing.
Brian
Thank you!
Recently I've been playing with Prince [1] and I've found it to be very powerful with excellent support for paged media. Given that HTML, CSS, Javascript, SVG is used to build the document, it's also familiar to most people already. There doesn't seem to be any open source competitor to Prince though.
[1]: http://www.princexml.com/
[1]: http://www.princexml.com/
I want to echo the support for Prince here, even though it's not open-source it is a very useful piece of software. I create web-based art books at a museum[1], and Prince has allowed us to generate high-quality PDFs[2] and even print-on-demand editions of the same book for very little additional effort. Full-bleed images, roman numeral page numbers, footnotes, multi-column text pages, all are possible through CSS.
In a past life I was a print designer relying on Adobe software for everything, and it is tremendously liberating to be able to leave that world of vendor-lock-in behind (InDesign has a nasty habit of making each new version's files incompatible with previous ones). So compared to where I'm coming from, Prince (even as proprietary software) is a big step forward because it allows me to work in open formats.
[1]: http://www.getty.edu/publications/romanmosaics
[2][PDF, 8MB]: http://www.getty.edu/publications/romanmosaics/assets/downlo...
In a past life I was a print designer relying on Adobe software for everything, and it is tremendously liberating to be able to leave that world of vendor-lock-in behind (InDesign has a nasty habit of making each new version's files incompatible with previous ones). So compared to where I'm coming from, Prince (even as proprietary software) is a big step forward because it allows me to work in open formats.
[1]: http://www.getty.edu/publications/romanmosaics
[2][PDF, 8MB]: http://www.getty.edu/publications/romanmosaics/assets/downlo...
PrinceXML is excellent, we used it for many years for typesetting programs for conferences, etc. After a while, though, I picked up enough LaTeX to be competent at generating LaTeX from our applications (just as we would generate HTML or any other markup) and that's what I use for report generation now. While there are more templates to maintain, the quality of the results is absolutely worth the effort.
On a tangent, a very nice "win" with the (pdf)LaTeX approach is that it's fairly easy to collate other PDF documents into a larger one, with a custom table of contents, consistent running headers and footers, index of authors, etc., and intersperse them with other content from our database. (This was done by splitting the child documents into pages with pdftk, then inserting them as "PDF graphics" into the generated LaTeX document -- and as long as the original PDFs were searchable, they remained searchable in the final output.) In one of our paper-intensive processes (a proposal approval workflow), this approach cut down tremendously on manual labour that was being done to prepare proposal dossiers, and the resulting collation was much easier for the reviewers to work with (e.g., everyone could turn to page "K-34" in the dossier, and be certain they were looking at the same page of the same document).
On a tangent, a very nice "win" with the (pdf)LaTeX approach is that it's fairly easy to collate other PDF documents into a larger one, with a custom table of contents, consistent running headers and footers, index of authors, etc., and intersperse them with other content from our database. (This was done by splitting the child documents into pages with pdftk, then inserting them as "PDF graphics" into the generated LaTeX document -- and as long as the original PDFs were searchable, they remained searchable in the final output.) In one of our paper-intensive processes (a proposal approval workflow), this approach cut down tremendously on manual labour that was being done to prepare proposal dossiers, and the resulting collation was much easier for the reviewers to work with (e.g., everyone could turn to page "K-34" in the dossier, and be certain they were looking at the same page of the same document).
>There doesn't seem to be any open source competitor to Prince though.
This would be my issue with using it. This is partly why I stay with LaTeX for our book as I can convert it to other formats if I really need to. I find LaTeX gives me more control over presentation & type-setting than web-based stuff as well.
This would be my issue with using it. This is partly why I stay with LaTeX for our book as I can convert it to other formats if I really need to. I find LaTeX gives me more control over presentation & type-setting than web-based stuff as well.
It also means your documents are locked to a proprietary standard that isn't that widely used (compared to say MS Word) and as such, may not be totally recoverable 10,20 or 30 years from now. You want to keep that in mind when you choose which typesetting system to use - for the record, I keep a digital diary that I intend should still exist 50 years from now. I use only markdown, and the occasional entry that mention really private stuff is ascii encrypted with GPG.
>It also means your documents are locked to a proprietary standard that isn't that widely used (compared to say MS Word)
HTML and CSS?
HTML and CSS?
I assume there is some additional special sauce that enables you to do things that are not possible in a browser, which means you still won't get the exact formatting, index, etc correct.
To extent of my knowledge, Prince only implements web standards set by w3. The reason browsers can't do what Prince does is because they lack support for those standards.
Wikipedia lists some of what Prince does that others don't:
https://en.wikipedia.org/wiki/Comparison_of_layout_engines_(...
Wikipedia lists some of what Prince does that others don't:
https://en.wikipedia.org/wiki/Comparison_of_layout_engines_(...
I'm happy to know about Prince, but US$3800 for a server license is beyond extravagant for any but large corporate users.
Life:
The other thing is that although one starts with grand ideas of being clean and orderly, by the end the process is somewhat of a mess, with a complicated makefile to keep it running.Yes! I loved that even Brian Kernighan can make a mess when trying to get something done :-) There is hope for us all!
Another way of looking at it is that, if even Kernighan can make a mess when trying to get something done, there's no hope for anyone.
…
Your take on it is more uplifting
…
Your take on it is more uplifting
It's a lesson that things don't have to be perfect to be able to ship.
It's sad that a complete structure writing/publishing workflow is still so hard. XSL-FO is awkward and limited, and will probably remain so. TeX is not design-friendly. On the other side, InDesign is mostly visual only. Framemaker seems to be only serious tool but it's very expensive.
I was a devoted FrameMaker user until Adobe killed the Mac product. I used to have to do fairly complicated legal documents, and it was worth every penny. Plus, it was a very good word processor--you never had to leave the program to edit text. So I used it for everything I had to write.
A book like this could have been handled fairly easily in FrameMaker, but sadly Adobe has decided to position in it to be only for large enterprise shops running Windows. They're probably (rightfully) afraid it would cannabalize InDesign sales if they made it more accessible.
A book like this could have been handled fairly easily in FrameMaker, but sadly Adobe has decided to position in it to be only for large enterprise shops running Windows. They're probably (rightfully) afraid it would cannabalize InDesign sales if they made it more accessible.
I have used it in 1992 on Sun stations. It was a delight. I am pretty sure it would gain a lot of attraction if they open sourced it.
Hm. I never heard of FrameMaker before, but their 2015 release is available at $29.99/month, which is hardly "positioned only for large enterprises".
Yeah, just out of curiosity I looked at what the current deal was, and that might almost be reasonable if a) I were still going to be using it every day and b) there were still a Mac/Linux version. I wouldn't do a subscription deal it just for personal use, but that's a different issue. But 'positioning' is partly a marketing thing. Not having a Mac version clearly separates the product from Indesign for a lot of users.
I keep an eye on the Scribus project from time to time. Who knows, maybe someday it can compete.
https://www.scribus.net/
https://www.scribus.net/
What about Apple Pages? I find that very good and easy to deal with typesetting. But I am no expert in the field so it might have sever weaknesses I am not aware of. Anyway I like that it seems to have been designed with layout in mind from the start unlike MS Word e.g. where doing layout has never felt natural.
Typesetting isn't the same as word processing, you need a lot more power than something like Pages gives you.
For example, our book is 308,000 words and 32 chapters. By the time it's done it'll probably be ~310-325k plus index. LaTeX means I can change one line in the header and have the build completely re-typeset the book according to those parameters programmatically, and it does a _very_ good job.
Re: index - how do you generate an index with Apple Pages? And I don't mean concordance, I mean a proper index. With LaTeX it's...programmatic. We provide the metadata in the form of `\index{blah}`, it generates the index for us.
It's extremely hard to find anything that's competitive on all these fronts and which is also open source. LaTeX is pretty much it, everything else is focused only on basic formatting & presentation and hasn't even gotten to a proper typesetting algorithm.
For example, our book is 308,000 words and 32 chapters. By the time it's done it'll probably be ~310-325k plus index. LaTeX means I can change one line in the header and have the build completely re-typeset the book according to those parameters programmatically, and it does a _very_ good job.
Re: index - how do you generate an index with Apple Pages? And I don't mean concordance, I mean a proper index. With LaTeX it's...programmatic. We provide the metadata in the form of `\index{blah}`, it generates the index for us.
It's extremely hard to find anything that's competitive on all these fronts and which is also open source. LaTeX is pretty much it, everything else is focused only on basic formatting & presentation and hasn't even gotten to a proper typesetting algorithm.
A word processor is a totally different beast from type setting.
https://programmingforresearch.wordpress.com/2012/03/23/word...
Though people have been trying to solve this in Word and templates but those still fail at technical books still IMHO.
https://programmingforresearch.wordpress.com/2012/03/23/word...
Though people have been trying to solve this in Word and templates but those still fail at technical books still IMHO.
Heirloom troff supports OpenType and OpenType features. I still haven't found time to look into it, but it sounds awesome.
Written in XML?! That's the first I've heard. Though also a fine proof that what you're writing is much more important than how you're writing it.
(Still, though -- XML?)
(Still, though -- XML?)
Why not XML? XML was made for documents, they were writing a document.
XML may have been made for documents but it is certainly one of the worst language families ever for actually typing documents by hand.
You don't have to write the XML tags while you type. The hardest part about writing long-form technical prose is writing long-form technical prose. You can concentrate on getting that right, in the editor of your choosing, using placeholders for diagrams and layout elements. Then, in a second pass, mark the text up.
Indeed. And it's odd no one's mentioned Pandoc yet.
Millions of websites are made with writing HTML by hand and it's certainly not that bad.
What would you use instead that would allow you to define your own markup? LaTeX? \em{Hello} isn't much better than <em>Hello</em>.
Also, XML + XSLT gives you the capability to easily transform your original to other formats.
For an actual typing-on-the-keyboard format, it seems like almost anything would have been better, particularly given the number of markup languages that can translate trivially into XML.
I may have misunderstood -- he was actually typing in XML tags as he wrote the book, right?
I may have misunderstood -- he was actually typing in XML tags as he wrote the book, right?
Yes, they did. I saw the XML and it didn't look much different from well-written HTML. The amount of markup you need for a book is fairly minimal: h1, h2, p, b, i, c (code font), ix (index entry). Think of it as a customizable HTML.
For anyone missing the lede here: the book is typeset in troff. This is astonishing because troff is 40+ years old and AFAIK hasn't gotten significant development in ~10 years. If it ain't broke don't fix it. http://www.troff.org/history.html
Kernighan has always used troff, so it is not a surprise that he is still using it. He can do things with troff that most people would have a hard time to do with LaTeX. On the other hand, this doesn't mean that troff is better, in fact as you said it has not been updated for a long time. For someone starting now, it is so much easier to learn and be productive with LaTeX instead of troff.
I wrote to Kernighan a few years ago, when "D is for Digital" came out, to ask him, in effect, if troff had been made obsolete by TeX. At the time I had begun using it, and wondered if it was worth investing my time to learn.
He answered that he still preferred to use groff, having tried alternatives. Since then I have used groff exclusively for writing documentation and papers. It never runs out of gas, and always produces attractive output. It offers a complete suite -- math, tables, pictures, graphs -- all in one package. It's fast, adaptable, and easy to learn. It has the highest ratio of text to markup of any typesetting system, including TeX and Markdown. It is the single most overlooked and under appreciated project on the planet.
Whether it's had "significant" development in the past 10 years depends on what that means. Current groff produces PDF format directly, without a Postscript intermediary. The PDF macros support links and TOC generation. I think that all took place in the last few years.
He answered that he still preferred to use groff, having tried alternatives. Since then I have used groff exclusively for writing documentation and papers. It never runs out of gas, and always produces attractive output. It offers a complete suite -- math, tables, pictures, graphs -- all in one package. It's fast, adaptable, and easy to learn. It has the highest ratio of text to markup of any typesetting system, including TeX and Markdown. It is the single most overlooked and under appreciated project on the planet.
Whether it's had "significant" development in the past 10 years depends on what that means. Current groff produces PDF format directly, without a Postscript intermediary. The PDF macros support links and TOC generation. I think that all took place in the last few years.
What platform do you write on? Linux? Mac? I'm genuinely curious.
I have been looking for substitutes to Latex ever since writing my dissertation in Latex and having to debug/tweak an already horrendous Latex .sty file. I basically decided that I would either write technical documents in Tex or not at all after that. For me, Latex is great, until it isn't, and then it's Jinga-on-a-unicycle frustration.
I have been looking for substitutes to Latex ever since writing my dissertation in Latex and having to debug/tweak an already horrendous Latex .sty file. I basically decided that I would either write technical documents in Tex or not at all after that. For me, Latex is great, until it isn't, and then it's Jinga-on-a-unicycle frustration.
The problem with LaTeX is that you are given an already unstable jinga (ie., style file etc.,).
You can always go back to the basics and use simple TeX for your documents if you think groff/troff is simpler.
However, I'd encourage you to give ConteXt http://wiki.contextgarden.net/ a try because it is the spiritual successor to TeX (one tool for all the typesetting) instead of having to depend on "third party" stuff like LaTeX, most of which very few of us (outside the original sty author) understand.
You can always go back to the basics and use simple TeX for your documents if you think groff/troff is simpler.
However, I'd encourage you to give ConteXt http://wiki.contextgarden.net/ a try because it is the spiritual successor to TeX (one tool for all the typesetting) instead of having to depend on "third party" stuff like LaTeX, most of which very few of us (outside the original sty author) understand.
> We wrote a variety of small Go programs and scripts to fix things up, including one that rewrites troff output to do vertical justification so all pages are the same height.
That sounds weird, why would they want that, wouldn't that make the pages pretty irregular internally? Like varying line heights..?
That sounds weird, why would they want that, wouldn't that make the pages pretty irregular internally? Like varying line heights..?
HTML with CSS with lend itself to generate PDFs. Are there open alternatives to Prince XML? Browsers don't support all page-layout related CSS2/3 attributes, like a page break.
wkhtmltopdf has some extensions for forcing a page break etc.
I used it for generating some documents from a groundup rebuild of bootstrap meant purely for printing (meant we could just use a lot of the web templates with minor changes) and it worked pretty well for our uses.
PrinceXML however is in a class of its own (as you'd expect for $3.8k per server)
I used it for generating some documents from a groundup rebuild of bootstrap meant purely for printing (meant we could just use a lot of the web templates with minor changes) and it worked pretty well for our uses.
PrinceXML however is in a class of its own (as you'd expect for $3.8k per server)
Generating PDF is not the same as typesetting.
GNU troff is looking for a maintainer. If you’re interested, please take a look at this general information about GNU packages and being a GNU maintainer, and then email [email protected] with a bit about your background and particular interest in this package. Thanks.
Please note that maintaining this package requires assigning copyright to the FSF, and ensuring that any future contributors also execute papers, since that is what past authors and contributors have done.