Read it first when I was 16, at the recommendation of my literature teacher in high school. For the past... - darn, this was long ago! - ...more than 20 years, I've read it at least once a year, and intend to do that for as long as I can read.
You will need to enter the repositories by hand, though. Mind you, writing a small tool that'd list the most popular/forked/starred/etc C repos, and do a search for you in those should be a few minute task.
Is there anything like blessed + blessed-contrib for either of those? It'd be fairly trivial to write the bulk of potential-happiness in Java, if there was a display library like blessed + blessed-contrib in the JS world.
Author here. I'm running this in "production" (kinda, on laptop, looking at data from my own servers), and works for me. There are some annoying shortcomings, though, which I will fix, eventually.
Among them is error reporting and handling: there's pretty much none. If the network dies, potential-happiness will not be too happy: it will cry and crash.
Bug reports, feature requests and whatnot are of course appreciated. I wrote this mostly for my own use, so it isn't exactly friendly just yet. (That, and my JavaScript is terrible.)
I remember http://lamsonproject.org/ being a python SMTP server library thing. It may be possible to piece together something based on it. That covers the SMTP part.
Mind you, the git repo looks dead, with the last commit being over two years old.
> you're saying that yours is unconditionally better
I don't think I'm saying that. The article presents two setups and a few related use cases, where I believe binary log storage is superior.
> With the services you run, you might be able to dictate that the log formats are restrictive enough that writing a parser for each one isn't a problematic overhead.
I don't need to dictate all log formats. If I can't parse one, I'll just store it as-is, with some meta-data (timestamp, origin host, and so on). My processed logs do not need to be completely uniform. As long as they have a few common keys, I can work with them.
For some apps or groups of apps, I can create special parsers, but I don't necessarily need that from day one. If I'm ok with only new logs being parsed according to the new rules (and most often, I am), I can add new rules anytime.
> Parsing up-front, assuming you know what data you can safely throw away, might appear to some as a premature optimisation.
>> We have well documented tools and workflows, so anyone new to the system can catch up and start working with the logs within minutes.
> It sounds like this is something which could be usefully open-sourced, to show how it's done.
LogStash is a reasonable starting point. Our solution has a lot of common with it, at least on the idea level.
> Pre-parsed binary logs in a locked-down environment might be as flexible as freeform text, but I'd need to see a running system to properly judge.
Only our storage is binary. That is all the article is talking about. Within that binary blob, there are many traces of freeform text, mostly in the MESSAGE keys of application logs which we care less about (and thus, parse no further than basic syslog parsing). You still have the flexibility of freeform text, even if you store it in a binary storage format.
I assume this comment is related to the journal. The article is not.
But, to reply: yes, many organisations have fully functional, well-debugged logging infrastructures. A lot of them also use binary log storage, and have been for over a decade, and are more than satisfied with the solution.
Both rsyslog and syslog-ng have been able to assist with setting such a thing up for about a decade now.
> Where are the rsyslogd and syslog-ng competitors to systemd's journald? Where is the interoperability? Where is the smooth, useful upgrade mechanism?
The journal has a syslog forwarded, but both rsyslog and syslog-ng can read directly from the journal. Interoperability was there from day one. Smooth upgrade mechanism took a while to iron out, but it's there now, too.
Why would I need to be able to parse everything up front? Taking the syslog example, that has a commonly understood format. As a default case, I can just split the parts and have structured data (esp. with RFC5424, where structured data is part of the protocol to begin with).
Then, I can add further parsers for the MESSAGE part whenever I feel like it, or whenever there is need. I don't need that up front.
With a binary log storage system, nothing stops you from browsing all logs that happened around the time of the incident. Instead of locating the files, you just tell the engine to show you the logs from that time onwards (or from a little bit before).
As for our logs being too verbose: nope, read the article.
Also, it's not an one-size-fits-all solution: I have no problem with people using text. All the article wants to show, is that binary logs are not evil, bad, useless, etc, and that there are actually very good reasons to use them.
For example, storing logs in a database is one kind of binary log storage: most databases don't store the data as text.
The machines I'm administering will all log the same way, therefore, within that context, they are universal. We have well documented tools and workflows, so anyone new to the system can catch up and start working with the logs within minutes.
We don't unexpectedly find machines that don't conform to our policies. We control the machines, we know where and how to find the logs. If we found any where we had to grep, we'd be having a very bad day.
Our lowest common denominator is not text, because we control the environment, and we can raise the bar. Being able to do that is - I believe - important for any admin.
* Why would you need a proprietary tool?
* What if they get broken? I don't want to look at them raw anyway.
* Text logs are easy to corrupt as well. Oh, append only? Well, you can do that with binary storage too.
And again, there is no need for proprietary tools at all. Everything I want to do is achievable with free software - so much so, that I use only such software in all my systems.
As for compressing - yeah, no. Please try compressing 100Gb of data and tell me the performance cost is nonexistent.
As for LogStash & ES: Guess what: their storage is binary.
Also note that my article explicitly said that the Journal is unfit for my use cases.
I've worked with a number of implementations, both embedded and others (ranging from a PC under my desk, through dozen-node clusters to ~hundred nodes). For most cases, binary storage triumphed. Most often, we kept text based transport.
Again, transport and storage are different. While I prefer binary storage, most of my transports are text (at least in large part, some binary wrapping may be present here and there).
My small system is usually two computers. Having a router/proxy/firewall box at home is not all that uncommon, and some examples I gave apply there nicely.
You're missing the point. I'm not using a custom logging format. I'm using binary log storage, with emphasis on the storage. There is a database and a search engine behind it.
Logging format and log storage format are two very different things.
Also, I'm not shocked people prefer text files. I'm shocked why they're so much against binary log storage. There's an important distinction between the two: you can prefer text, if that fits your case better, without hating on binary storage.