Hey mcint, fyi both of these issues are addressed: the localhost one is addressed for real, and a Match issue is worked around: while it's still not properly implemented, at least it doesn't prevent Nerdlog from starting now. Just in case you wanted to give it another try.
Thanks for sharing! Good to know that nerdlog turns out to be helpful not only for devs (the original use case), but also for DevOps :)
Fyi, support for journalctl was added to master, in case you wanted to try it out. I didn't yet add automated tests with the mocked journalctl, but my manual tests show that it's working fine.
If a system doesn't have either `/var/log/messages` or `/var/log/syslog`, nerdlog will now resort to `journalctl` by default.
It can also be selected explicitly by specifying `journalctl` as the file, e.g. `myserver.com:22:journalctl`.
I see, interesting. If you don't mind me asking, is it a sysadmin kind of job? Just trying to understand the use case better.
Regardless, journalctl support is the single most requested feature, so yeah I'll at least try to make that happen; hopefully on the upcoming weekend if I'm lucky.
I only had the blood circulation issues from compression during sleeping, and also had no idea until today that the nerve damage like that is possible. I wonder how could it be prevented, if it can.
I seriously hope I won't have to deal with it, but thanks for expanding on it and the treatment.
I don't think there's anything wrong with this format, it looks good. The most important thing for nerdlog is that all components of the timestamp must be at the fixed offsets from the beginning of each line. So I believe it can be implemented, I can look at it on the weekend. Feel free to create an issue on Github (I won't be able to test it for real, so would need your help with that)
Just posting it in case you want to subscribe to it. Looks like it's a popular demand indeed, so I'll at least poke it and see what kind of performance we can get out of it.
If so, then yeah it's totally doable to make this format supported.
Re: config.yaml, yeah I thought of that, but in the long term I rather wanted it to be nerdlogrc.lua, so a Lua script which nerdlog executes on startup. Similar to vim (or rather, more like neovim in this case since it's Lua). Certainly having config.yaml is easier to implement, but in the longer term it may make things more confusing if we also introduce the Lua scripting.
Re: the date/time format, I was thinking about implementing support for an option like timefmt, so you'd be able to do :set timefmt=2006-01-02T15:04:05Z07:00 , but postponed for now.
That's not hard to implement, however to make it persistent requires implementing some config / scriptability, which is a whole other thing and requires more thought.
Re: runit, I never tested it, but after looking around briefly, it sounds like there is no unified log file, and not even unified log format? I mean it's possible to make it work, treating every log file as a separate logstream, but I've no idea what these logs look like and whether supporting the formats would be easy.
Thanks. And no, as of today, there's no way to define a group like that. Might be a viable idea though.
However, before we go there, I want to double check that we're on the same page: this `log_files` field specifies only files _in the same logstream_; meaning, these files need to have consecutive logs. So for example, it can be ["/var/log/syslog", "/var/log/syslog.1"], or it can be ["/var/log/auth.log", "/var/log/auth.log.1"], but it can NOT be something like ["/var/log/syslog", "/var/log/auth.log"].
I first responded before your edit about ssh and localhost, so: yeah, as briefly mentioned in the article, as of today there's no shortcut even for localhost. I was debating whether I should implement this feature before open sourcing it, but I had to draw the line somewhere (I have TONS of ideas what could be implemented), and since reading local logs isn't the primary focus of nerdlog, I decided to skip it for now.
But yes the bypass for localhost can definitely be implemented.
Yeah it would be great, and I do want to support it, especially if the demand is popular. In fact, even if you ungzip them manually, as of today nerdlog doesn't support more than 2 files in a logstream, which needs to be fixed first.
Specifically about supporting gzipped logs though, the UX I'm thinking about is like this: if the requested time range goes beyond the earliest available ungzipped file, then warn the user that we'll have to ungzip the next file (that warning can be turned off in options though, but by default I don't want to just ungzip it silently, because it can consume a signficant amount of disk space). So if the user agrees, nerdlog ungzips it and places somewhere under tmp. It'll never delete it manually though, relying on the regular OS means of cleaning up /tmp, and will keep using it as long as it's available.
Not really, at least not yet, because nerdlog's focus is very different than that of lnav. There is a section about it in the article as well.
In fact nerdlog doesn't even support anything like -f (realtime following) yet. The idea to implement it did cross my mind, but I never really needed it in practice, so I figured I'd spend my time on something else. Might do it some day if the demand is popular, but still, nerdlog in general is not about just reading a continuous stream of logs; it's rather about being able to query arbitrary time periods from remote logs, and being very fast at that.
Cheers.