Stop using tail -f (mostly)(brianstorti.com)
brianstorti.com
Stop using tail -f (mostly)
http://www.brianstorti.com/stop-using-tail/?utm_content=buffer13c97&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
210 comments
With less, you can use m<letter> to mark the current position and '<letter> to go to a marked position. h will show you lots of useful help.
GP isn't meaning it like that, they mean being able to add some spaces so that the next log entries stand out for quick visual identification as you're making changes and reloading the application.
Use '/' (search) to enter a regexp to match a pattern on that line, and it will be highlighted.
This has nothing to do with what is being discussed in this subthread.
Sure it does. If your log lines are distinct (e.g. they have timestamps or unique IDs) then you can use less's search highlighting to provide a visual marker for a specific line, similar to what you can do by manually inserting a bunch of blank lines on the console.
This trick doesn't work if your log file has a bunch of identical lines and you want to keep an eye on their rate, though.
This trick doesn't work if your log file has a bunch of identical lines and you want to keep an eye on their rate, though.
Having to remember & type a timestamp has much more mental overhead (planning & memory) than "scan/scroll back to last block of vertical whitespace".
That's why suggestions of either named-marks or back-searches aren't considered equally-attractive alternatives to marking the scrollback with a batch of <return>s.
That's why suggestions of either named-marks or back-searches aren't considered equally-attractive alternatives to marking the scrollback with a batch of <return>s.
imagine the scenario: "I want to see everything that happens in a single request"
Enter method:
Enter method:
1. Press enter a bunch of times
2. Reload browser
3. Press enter a bunch of times and scroll up
1. Search for last line in output to highlight it?
2. Reload page
3. Try and figure out where stuff starts and ends with loads of visual noiseI won't argue that for this specific use case, tail isn't friendlier than less.
But the original poster posted a useful tip, and is now getting aggressive downvotes and comments like "This has nothing to do with what is being discussed in this subthread." I think that's unwarranted.
But the original poster posted a useful tip, and is now getting aggressive downvotes and comments like "This has nothing to do with what is being discussed in this subthread." I think that's unwarranted.
less method:
1. ma
2. Reload browser
3. 'a
1. ma
2. Reload browser
3. mb
4. 'aUnfortunately mark doesn't seem to work while you're following.
A bunch of blank lines is a lot more grokable than timestamps lost in a bunch of text.
An example of some of the vi commands that less(1) has adopted. You can also:
* page forward and backward, including using numeric prefixes to indicate a lineno or indicate "repeat 'n' times".
* launch vi if less(1) is working on a file (versus (eg) stdout of some process)
* search fwd/backward
* start examining a new file w/o leaving less(1)
* ...Another less-vs-vi question:
As a junior programmer I got chewed out by my IT department because I was examining a (production) log file in vi. The sysadmin told me I should use less (actually more---this was on Solaris in ~2001). To this day I only read log files with less, but I've never figured out his objection. Negatives to using vi I can imagine are:
- I might write to the log file. That seems like a real worry, although I could also say `vi -R` to prevent it.
- Starving the production system of memory. I'm pretty sure he expressed this concern. Any insight into whether it is legit? Is less actually any better?
Obviously you really should have a log shipping & aggregation service so you can read logs offline, etc, but not every project is large enough for that, nor every org organized enough. So for the sake of argument my premise is, "Assuming you want to read a log on production . . ."
As a junior programmer I got chewed out by my IT department because I was examining a (production) log file in vi. The sysadmin told me I should use less (actually more---this was on Solaris in ~2001). To this day I only read log files with less, but I've never figured out his objection. Negatives to using vi I can imagine are:
- I might write to the log file. That seems like a real worry, although I could also say `vi -R` to prevent it.
- Starving the production system of memory. I'm pretty sure he expressed this concern. Any insight into whether it is legit? Is less actually any better?
Obviously you really should have a log shipping & aggregation service so you can read logs offline, etc, but not every project is large enough for that, nor every org organized enough. So for the sake of argument my premise is, "Assuming you want to read a log on production . . ."
From sysop standpoint there is a huge difference:
1) for user: less chance (pun intended) to actually change the file when all you wanted was to read it
2) for sysadmin: if sysadmin sees "less somefile.log" in bash history, he knows the user just read the log. If he sees "vi somefile.log" then he doesn't know if the user has also changed the log file (maybe not even knowing it).
The assumption is that you deal with non-malicious users who just make mistakes (which is often the case).
1) for user: less chance (pun intended) to actually change the file when all you wanted was to read it
2) for sysadmin: if sysadmin sees "less somefile.log" in bash history, he knows the user just read the log. If he sees "vi somefile.log" then he doesn't know if the user has also changed the log file (maybe not even knowing it).
The assumption is that you deal with non-malicious users who just make mistakes (which is often the case).
for sysadmin: if sysadmin sees "less somefile.log" in bash history, he knows the user just read the log. If he sees "vi somefile.log" then he doesn't know if the user has also changed the log file (maybe not even knowing it).
Ha ha, this reminds me of the days I had sudo access to `vi` but not a lot of other commands, on a box that IT didn't really want to support. . . .
Thanks for your reply! Since I do a lot of devops I've tried over the years to formulate a mental model of how a sysadmin thinks and what they care about. I'd never have thought about this auditability concern, so it's something to add to my list! :-)
On large files less fires up immediately and vim is quite slow. I think that vim is loading an entire file in the memory while less just loads visible chunk. May be other vi implementations are more efficient with large files.
Besides — vim really shouldn't be used on log files. It's editor, not viewer.
Besides — vim really shouldn't be used on log files. It's editor, not viewer.
> Besides — vim really shouldn't be used on log files. It's editor, not viewer.
I think the two concerns I stated are probably legit, but this one is boring to me. Vim is nicer than less for reading files. I have more navigation commands. I can yank part of the file and save it somewhere else. I can pipe a range of lines through awk/etc. I can switch between files. I can split my screen. Etc. Some of these are probably available in less too (more than more(1) supported in 2001), but I doubt all, and I already know the commands in vim. I'm interested in not clobbering my logs and not crashing the server, but if you tell me vim is an editor not a viewer, I'll ask Why?
Btw re-reading my words I don't mean to sound combative. But the point of my original question was to understand. I've been cargo-culting "use less for logs" for 14 years already.
I think the two concerns I stated are probably legit, but this one is boring to me. Vim is nicer than less for reading files. I have more navigation commands. I can yank part of the file and save it somewhere else. I can pipe a range of lines through awk/etc. I can switch between files. I can split my screen. Etc. Some of these are probably available in less too (more than more(1) supported in 2001), but I doubt all, and I already know the commands in vim. I'm interested in not clobbering my logs and not crashing the server, but if you tell me vim is an editor not a viewer, I'll ask Why?
Btw re-reading my words I don't mean to sound combative. But the point of my original question was to understand. I've been cargo-culting "use less for logs" for 14 years already.
You can save a part of the file, pipe a range of lines, or switch between files in less, too, and the commands are mostly the same as in vi. It'd sure be nice to be able to split the screen, though!
> I think that vim is loading an entire file in the memory while less just loads visible chunk
No, vim also only keeps part of the file loaded, however it will try to count how many lines the file has.
No, vim also only keeps part of the file loaded, however it will try to count how many lines the file has.
Vim ships with a command called `view` that will start it in read-only mode, and makes it very much a viewer — not just an editor.
True, and I like using this for syntax-colored views of files. But it's got an annoying tendency to switch to edit mode with very little fuss.
Sometimes I want a viewer to just be a viewer.
Speaking of which: are there any Linux/Unix viewers that do have generalized syntax highlighting support?
Sometimes I want a viewer to just be a viewer.
Speaking of which: are there any Linux/Unix viewers that do have generalized syntax highlighting support?
You can give vimpager a try. It even supports vimrc. But I have felt it to be considerably slower than less.
With vim, I usually use 'vim -u NONE <file-name>' which skips the startup file and makes loading zippy
It may also try to do syntax highlighting and other formatting stuff, taking up even more resources and cpu time.
And an editor should not be used on log files?
Frequently, frequently I am in the position that I need to manipulate an overly-verbose log file to condense out the information I want. I could spend a half hour concocting wizard-like shell invocations, OR I could do it interactively in five minutes with vi...
Frequently, frequently I am in the position that I need to manipulate an overly-verbose log file to condense out the information I want. I could spend a half hour concocting wizard-like shell invocations, OR I could do it interactively in five minutes with vi...
I think your idea of `vi -R` or `vim -R` is a good idea. `less` uses less memory and may load faster but looking at logs for things like valgrind vim will give you syntax highlighting that I am not sure you can get with `less`.
BTW, starting vim as "view" (through e.g. a symlink) is equivalent to "vim -R", precisely for this use case [0].
[0] http://vimdoc.sourceforge.net/htmldoc/starting.html#view
[0] http://vimdoc.sourceforge.net/htmldoc/starting.html#view
I have used `view` but it didn't give me syntax highlighting. On my system (archlinux) view is actually a symlink to ex (which must change its behavior based on its name?).
readlink /usr/bin/view
exI think Edix's answer is the most pragmatic.
Also, someone at work actually crashed a service by opening the log files in vim.
I think the problem was that the memory and processor were already getting stomped on (thus the need to look at the logs) and vim tried to do a lot of fancy stuffs to get more info on the file as a whole.
I think the problem was that the memory and processor were already getting stomped on (thus the need to look at the logs) and vim tried to do a lot of fancy stuffs to get more info on the file as a whole.
If I'm logging into a server later to resolve an issue, one thing I will probably check is the command history. If you open a file with "vi", how do I know that you only looked at it, and did not make any changes to it? When I see "less" in the command history, I know for certain that it was a read-only operation.
Why do you let revs login to production machines. That's your first problem.
It's a good practice for sure, but your sysadmin was probably a little over-hash.
I'm generally in the habit of using `view` to do read-only vim. `less` works as well.
I'm generally in the habit of using `view` to do read-only vim. `less` works as well.
Funny - I always used to use less, until this popped up on HN:
http://seclists.org/fulldisclosure/2014/Nov/74
Now I mostly use vi.
http://seclists.org/fulldisclosure/2014/Nov/74
Now I mostly use vi.
I would never "yell" at anybody for using vi vs less, but I might curiously ask their reasoning behind it. I'm generally for less bikeshedding and cargo cultism when possible. Whatever works to get the job done.
Yes, but cpu is normally the problem. vim is very inefficient with long lines, combine that with all of vim's features (syntax highlighting, etc.) and you have the potential to make a problem worse.
I don't know his exact reasoning, but the biggest thing I can think of you already touched upon: there's more of a danger that you can write to a file accidentally with vi vs with less.
Beyond that, I would probably correct a junior employee as well. Even if there's nothing wrong with it, it's not the right tool for the job. When I first started I got 'yelled' at for checking to see if a machine was on the network using tracert instead of ping. It works, but it's not the right tool for the job.
Beyond that, I would probably correct a junior employee as well. Even if there's nothing wrong with it, it's not the right tool for the job. When I first started I got 'yelled' at for checking to see if a machine was on the network using tracert instead of ping. It works, but it's not the right tool for the job.
The sysadmin was low knowledge.
You should download the log file and view it locally. You should never run ad hoc commands on a live production sytem.
And there is no reason that you should have edit privileges on the log file anyway.
You should download the log file and view it locally. You should never run ad hoc commands on a live production sytem.
And there is no reason that you should have edit privileges on the log file anyway.
'&pattern' filtering is huge for me.
Display only lines which match the pattern; lines which do not match the pattern are not displayed. If pattern is empty (if you type & immediately followed by ENTER), any filtering is turned off, and all lines are displayed. While filtering is in effect, an ampersand is displayed at the beginning of the prompt, as a reminder that some lines in the file may be hidden.
Display only lines which match the pattern; lines which do not match the pattern are not displayed. If pattern is empty (if you type & immediately followed by ENTER), any filtering is turned off, and all lines are displayed. While filtering is in effect, an ampersand is displayed at the beginning of the prompt, as a reminder that some lines in the file may be hidden.
Just the other day I was looking for a way to do this in less:
yank 10 lines, open a new file, paste the 10 lines, save the file.
Is that possible? My Googling last week didn't find a good solution.
yank 10 lines, open a new file, paste the 10 lines, save the file.
Is that possible? My Googling last week didn't find a good solution.
If you wanted to grab 10 lines from /var/log/messages, starting at the first line that has "pjungwlr" in it, you could:
$ less /var/log/msgs
/pjungwlr^M
^G (note line informational line numbers)
v (launch vi)
[double check your line #s, etc]
:d1,. (delete from 1st line, to current line)
10j (go down 10 lines)
:d,$ (delete to EOF)
:w my_newfile
:qThis is such a common problem there's a stackoverflow article on it.
http://stackoverflow.com/questions/17908555/printing-with-se...
(Actually, there's probably more than one despite their really good duplication grooming, but this was the first hit.)
http://stackoverflow.com/questions/17908555/printing-with-se...
(Actually, there's probably more than one despite their really good duplication grooming, but this was the first hit.)
`v` was the missing key for me. Looks like it even preserves my position in the file. Thanks!
Usually I `ma|acat > newfile` in less, which saves by default a whole screenfull. There must be a better way to do this in less.
If you're using screen/tmux as myself, tail -f remains a good option, as you can enter gaps or search, or do 1000 more things as compared to plain terminal session
That doesn't change the actual log file, however - it's only visual whilst in the screen buffer
On OSX/iterm you can do command+k to clear the screen and less lets you navigate up if you need.
Even when you don't want less +F, a good alternative to tail -f is tail -F. It survives log rotation, and the file being temporarily inaccessible:
--retry
keep trying to open a file even if it is inaccessible when tail
starts or if it becomes inaccessible later; useful when follow-
ing by name, i.e., with --follow=name
-F same as --follow=name --retry+1 for -F. Note that --follow and --retry are gnuisms, while -F works on most implementations of tail (even though it's not part of POSIX).
on OpenBSD -f does:
Do not stop when end-of-file is reached, but rather to wait for
additional data to be appended to the input. If the file is re-
placed (i.e., the inode number changes), tail will reopen the
file and continue. If the file is truncated, tail will reset its
position to the beginning. This makes tail more useful for
watching log files that may get rotated. The -f option is ig-
nored if the standard input is a pipe, but not if it is a FIFO.
Is there a bug report open for that yet? FreeBSD has this so they should be able to resync easily enough
Why would they? It's not needed - '-F', '--retry' and the like are redundant. IMVHO, OpenBSD's '-f' behaviour should have been the default elsewhere, too. How many times in the past have you deliberately followed the file descriptor (coreutils' default '-f' behaviour) instead of its name? Every single time I used it in the past on Linux, this is what I thought:
"For f***** sake! I should have used '-F'! Arghhh...!"
I'm not saying that the OpenBSD behaviour is wrong but that compatibility is a worthwhile goal. Adding a simple -F alias for their existing behaviour would allow people's habits & shell scripts to just work without changes.
Why? OpenBSD's implementation makes sense to me.
> FreeBSD has this
OpenBSD ≠ FreeBSD, and that's fine.
> FreeBSD has this
OpenBSD ≠ FreeBSD, and that's fine.
I don't think its a bug
If you want to be pedantic, no, it's not a bug but in the vast majority of the software world people use the term “bug tracker” to refer to a system which also tracks new work which isn't strictly a defect.
And, since we're being pedantic, when I said “vast majority of the software world”, that includes the OpenBSD project:
“Sending in bug reports
If possible, use the sendbug(1) command to get the bug into our tracking system. Sendbug requires that your system can properly send Internet email. If you cannot use sendbug on a functional OpenBSD machine, please send your bug report to [email protected].
Perhaps what you are sending in is a feature request, not necessarily a bug. New features are accepted, especially with code that implements your suggested new feature.”
http://www.openbsd.org/report.html#bugtypes
And, since we're being pedantic, when I said “vast majority of the software world”, that includes the OpenBSD project:
“Sending in bug reports
If possible, use the sendbug(1) command to get the bug into our tracking system. Sendbug requires that your system can properly send Internet email. If you cannot use sendbug on a functional OpenBSD machine, please send your bug report to [email protected].
Perhaps what you are sending in is a feature request, not necessarily a bug. New features are accepted, especially with code that implements your suggested new feature.”
http://www.openbsd.org/report.html#bugtypes
Ok, but it has an -f behavior that seems to do the right thing. I don't think I would file a request to add flags that aren't needed.
I'm not being "pedantic", I just don't think there is anything to change.
I'm not being "pedantic", I just don't think there is anything to change.
The idea is that if everyone else is converging on "tail -F" supporting that as an alias means that people's habits & shell scripts just work without modification. That seems worth adding a single line to a switch statement to me.
This is my standard now. Logs rotate all the time and it very convenient to have tail continue working without my intervention.
Yeah, tail -F is the way to go imo.
Also, if you are using a proper ssh client, opening a second ssh connection to manage the reacting to the log files is habit for me at this point. The only time I'm using tail -F is after a new configuration deployment. Otherwise, I'm looking at archival data in ELK.
Also, if you are using a proper ssh client, opening a second ssh connection to manage the reacting to the log files is habit for me at this point. The only time I'm using tail -F is after a new configuration deployment. Otherwise, I'm looking at archival data in ELK.
[deleted]
Have you tried using screen[1]? Might make things a little easier than have two separate SSH connections.
[1] https://www.gnu.org/software/screen/
[1] https://www.gnu.org/software/screen/
Yes. I use it when I need to resume a SSH session. Given proper configuration, switching SSH tabs/connections is as easy as switching browser tabs...there is no real need for me to use screen for such a purpose.
Thank you for the suggestion tho. :)
Thank you for the suggestion tho. :)
c.f. tmux
This _almost_ touches on the useful part of this:
If you search in less and then put it in follow mode, it continues to highlight the search terms. This is very useful for trapping an exception or webcall in the wild.
The downside: less buffers and tail -f prints directly. On a slow printing log this can cause events to show up slowly, and can cause performance issues on a fast moving log. If you're piping through a script for processing, tail -f is still the best bet. If you need multiple files, multitail is probably better. less +F hits a quick+easy operational niche and is available almost everywhere (whereas multitail is not).
The downside: less buffers and tail -f prints directly. On a slow printing log this can cause events to show up slowly, and can cause performance issues on a fast moving log. If you're piping through a script for processing, tail -f is still the best bet. If you need multiple files, multitail is probably better. less +F hits a quick+easy operational niche and is available almost everywhere (whereas multitail is not).
The other useful thing is filtering - on sufficiently modern versions of less [1], you can use & to filter the lines, in the same way you use / to search them - only lines matching the filter are shown, and the display continues to update. Use &! for a negative filter. This has replaced grep | tail for me, with the advantage that it's non-destructive, so you can undo the filter, reapply a different one, etc.
[1] Everything i've used on Linux in the last few years, not the vanilla one on the Mac, but the one in Homebrew
[1] Everything i've used on Linux in the last few years, not the vanilla one on the Mac, but the one in Homebrew
& is very handy, however it is also excruciatingly slow. It looks like it uses an O(n^2) algorithm, just from observing how slow it works.
+1 (upvoted).
$ less ./somefile
$ less ./somefile
/somesearch^M (<--- does a search)
F (<--- puts less(1) in "follow mode", highlighting your previous search-term if it occurs in the future)
But how do you combine the two on the command line, so less is tailing the file AND highlighting a pattern when launched?
I typically read logfiles with less +F, but it would be nice to create an alias for patterns in specific types of logs, so I don't have to remember them or rely on command history.
I typically read logfiles with less +F, but it would be nice to create an alias for patterns in specific types of logs, so I don't have to remember them or rely on command history.
$ less +F +/mypattern^M /my/file/to/search_and_follow
* Note that I needed to "quote" (^V^M) that carriage-return to get the search pattern to work.
Happy searching-and-following.
* Note that I needed to "quote" (^V^M) that carriage-return to get the search pattern to work.
Happy searching-and-following.
Thanks! Any idea what the quoted carriage return does? I wouldn't have thought of that in a million years.
> Any idea what the quoted carriage return does?
From creating vi (nvi, on NetBSD) macros, I'm used to thinking in terms of competely replicating the keystrokes that one would do interactively... so I tried it both ways (with and without ^M). The way I published is the one that works. Why the ^M ? Because if you're working interactively the search-pattern isn't submitted until you press Enter. nvi(1) (what NetBSD (and Free and OpenBSD) uses) will accept "-c" "command" arguments which are similar to the less(1) "+"... so, you can:
$ vi -c 123 ./myfile
and start editing "./myfile" at line 123. Nice for edit/compile/edit/compile dance that might happen if you're developing software. Play with that (and try your imagination with other ideas).
Have fun, happy exploring.
From creating vi (nvi, on NetBSD) macros, I'm used to thinking in terms of competely replicating the keystrokes that one would do interactively... so I tried it both ways (with and without ^M). The way I published is the one that works. Why the ^M ? Because if you're working interactively the search-pattern isn't submitted until you press Enter. nvi(1) (what NetBSD (and Free and OpenBSD) uses) will accept "-c" "command" arguments which are similar to the less(1) "+"... so, you can:
$ vi -c 123 ./myfile
and start editing "./myfile" at line 123. Nice for edit/compile/edit/compile dance that might happen if you're developing software. Play with that (and try your imagination with other ideas).
Have fun, happy exploring.
bch has it. Less has ancient command line args, they aren't getopt style.
I was going to mention multitail as well, which is a fantastic utility. That and rsstail, both by Folkert Van Heusden, are quite slick. The mark it leaves (a red line across the screen) is even more visible than those given by less.
I've commented a bit more on them (in the context of other RSS tools) here:
https://www.reddit.com/r/dredmorbius/comments/1udv6i/further...
Folkert's site: http://www.vanheusden.com/ http://www.vanheusden.com/multitail/ http://www.vanheusden.com/rsstail/
Though not quite applicable to logfiles, I've written a script to tail my Newsbeuter feed URLs with Multitail support, for a console / terminal-based RSS feed.
I've commented a bit more on them (in the context of other RSS tools) here:
https://www.reddit.com/r/dredmorbius/comments/1udv6i/further...
Folkert's site: http://www.vanheusden.com/ http://www.vanheusden.com/multitail/ http://www.vanheusden.com/rsstail/
Though not quite applicable to logfiles, I've written a script to tail my Newsbeuter feed URLs with Multitail support, for a console / terminal-based RSS feed.
You can also achieve this by doing:
tail -f whatever.log | grep --color=auto -C99 exception
And there are other ways pausing/scrolling back, such as tmux's scroll mode.
tail -f whatever.log | grep --color=auto -C99 exception
And there are other ways pausing/scrolling back, such as tmux's scroll mode.
> We all have been there: You are watching a file with tail -f, and then you need to search for something in this file, or just navigate up and down. Now you need to exit tail (or open a new shell), and ack this file or open it with vim to find what you are looking for. After that, you run tail again to continue watching the file. There's no need to do that when you are using less.
There's also no need to do that if you're running tmux, screen or have navigation tools baked into your terminal emulator.
Not that I'm trying to dismiss the effectiveness for +F for those inclined, but as the author acknowledged, +F also comes with some usability issues that -f does not (namely working with multiple files, and piping output into grep et al). Plus learning to navigate though terminal history is a useful skill to have in other situations anyway, and utilities like tmux are actually a very handy tool to use for a whole plethora of additional reasons too.
So kudos to the author for his recommendation and providing helpful write ups to future sysadmins, but I'm inclined disagree with him and instead recommend people stick with -f and learn tmux instead.
There's also no need to do that if you're running tmux, screen or have navigation tools baked into your terminal emulator.
Not that I'm trying to dismiss the effectiveness for +F for those inclined, but as the author acknowledged, +F also comes with some usability issues that -f does not (namely working with multiple files, and piping output into grep et al). Plus learning to navigate though terminal history is a useful skill to have in other situations anyway, and utilities like tmux are actually a very handy tool to use for a whole plethora of additional reasons too.
So kudos to the author for his recommendation and providing helpful write ups to future sysadmins, but I'm inclined disagree with him and instead recommend people stick with -f and learn tmux instead.
Not if you also need to search things that weren't at the end of the file when you started tailing - which is very common with log files.
Generally you wouldn't want auto updating like the less +F or tail -f if the content you wanted was at the start of the file. However you can still work around those rare instances by specifying the -n flag. eg
And the best thing about this method is you can still pipe grep (which you couldn't do with less) so if you are writing out ~2000 lines rapidly enough that the content you want wouldn't be at the bottom, you can manage the text stream more efficiently (ie grep -v out stuff you don't want or only grep in the content you do want)
I should add that I am a fan of less on many occasions, but in this instance the piping ability of tail is a deal breaker when working with larger files.
tail -n 2000 -f
And the best thing about this method is you can still pipe grep (which you couldn't do with less) so if you are writing out ~2000 lines rapidly enough that the content you want wouldn't be at the bottom, you can manage the text stream more efficiently (ie grep -v out stuff you don't want or only grep in the content you do want)
I should add that I am a fan of less on many occasions, but in this instance the piping ability of tail is a deal breaker when working with larger files.
Also searching with grep is a lot faster than searching with '/' in less when the file is really large.
Yes and grep isn't very good for interactive searching. There's lot of different ways you could do this, and they all have pros and cons depending on the situation. My point was that all these posts dismissing this blog post as pointless are just arrogant. Most people don't know about less +F, but it's pretty handy for some common use cases. So he didn't mention someone's favourite tool - it's okay.
At no point did I dismiss his post nor show any arrogance towards the author:
> Not that I'm trying to dismiss the effectiveness for +F for those inclined
> kudos to the author for his recommendation and providing helpful write ups to future sysadmins
The ironic thing is, you've been far more dismissive about other peoples suggestions than I had of the author's. As you said yourself, there are a lot different ways this can be done, so why limit us to discussing only one possible solution?
> Not that I'm trying to dismiss the effectiveness for +F for those inclined
> kudos to the author for his recommendation and providing helpful write ups to future sysadmins
The ironic thing is, you've been far more dismissive about other peoples suggestions than I had of the author's. As you said yourself, there are a lot different ways this can be done, so why limit us to discussing only one possible solution?
Fair enough. My apologies for how that was conveyed.
I see that tmux doesn't come as part of standard ubuntu installation. So I think that puts it at a disadvantage for not being omnipresent.
Neither is about 99% of the other software that Ubuntu desktops and servers run daily. Hence why we have software repositories and user friendly package managers to make installing new software a breeze:
apt-get install tmuxbyobu is a sink-included tmux distribution in Ubuntu, that does most of the hard setup work for you.
I still like tmux for getting several servers in debug mode on the same screen when using docker though.
I still like tmux for getting several servers in debug mode on the same screen when using docker though.
If you're looking at log files, a tool I've started to play with is http://lnav.org/
= In Your Terminal = Many logging tools, like Splunk, provide great features but are optimized for large-scale deployments. They require installing and configuring servers before they can be effectively used. There is still a need for a robust log file analyzer for the terminal.
= Easy to Use = Just point lnav to a directory and it will take care of the rest. File formats are automatically detected and compressed files are unpacked on the fly.
= Improved Presentation = Log files are a wealth of information, lnav can help highlight the parts that are important and filter out the noise.
= In Your Terminal = Many logging tools, like Splunk, provide great features but are optimized for large-scale deployments. They require installing and configuring servers before they can be effectively used. There is still a need for a robust log file analyzer for the terminal.
= Easy to Use = Just point lnav to a directory and it will take care of the rest. File formats are automatically detected and compressed files are unpacked on the fly.
= Improved Presentation = Log files are a wealth of information, lnav can help highlight the parts that are important and filter out the noise.
That looks very nice! Thanks for the info. 8)
If you're using a modern terminal with scrollback, you already have the ability to pause, and scroll back.
# tail -f busy.log
< see something of interest >
ctrl+s # Stops flow of output.
pgup/pgdn for navigation # fn+up/dn for OS X users
ctrl+q # Continues flow.
This is not a replacement for searching for a specific string in `less` which highlights the matches, and eliminates the need for scanning the text manually which is harder and prone to missing information.
Correct, however the author is saying that there's no reason to ever use tail unless you're tailing multiple files, which is false. Most of the functionality you want is already built into your terminal emulator. If you want slower updates (`less` uses 1 second polling, vs `tail` [inotify]), and the ability to search, by all means, use `less`.
Hmm, someone should add inotify handling to `less`.
Most terminals also offer searching capability, which do not result in disk seeks on the remote host.
But it won't show me old parts of the log and I usually want this.
Or use multitail. It supports syntax highlighting, among other features: http://www.vanheusden.com/multitail/
multitail is great. I really cannot use command line utils that don't support colors/highlighting anymore. htop instead of top, etc. It just gives me much better visibility and readability. I'm surprised that so many non-color utils are still being used. It just feels so 1994 to me to stare at a white on black display. I guess there's nothing more slow moving and conservative than shell interfaces, thus articles like these that shame us into using different tools.
Personally, I'd love to see some hot young talent just do a 100% redo of the standard gnu utils from an interface perspective. Just go crazy with new interface and display ideas, novel presentation modes, novel navigation, etc while still maintaining backwards compatibility. I could see a big disruption here.
Personally, I'd love to see some hot young talent just do a 100% redo of the standard gnu utils from an interface perspective. Just go crazy with new interface and display ideas, novel presentation modes, novel navigation, etc while still maintaining backwards compatibility. I could see a big disruption here.
I wrote a Python script (called synesthesia)[1] that'll colorize input based on regex matches, and any matched text will be the same color for the same content.
The use case that drove its development was needing to keep track of UUIDs across multiple logs - and grep --color will colorize its matches, but not differentiate between ones that have different content. With this, I could watch both logs as data was passed from one to the other and keep track of e.g. the orange one.
I also thought it would be nice to be able to use patterns from logstash's grok, so I wrote grokpat[2] to find patterns for me. A lot of grok's patterns use atomic groups, which aren't available in Python 2, so I wrote redi[3] to convert them from grok's syntax to Python compatible syntax.
So I can now colorize logs easily as follows:
The use case that drove its development was needing to keep track of UUIDs across multiple logs - and grep --color will colorize its matches, but not differentiate between ones that have different content. With this, I could watch both logs as data was passed from one to the other and keep track of e.g. the orange one.
I also thought it would be nice to be able to use patterns from logstash's grok, so I wrote grokpat[2] to find patterns for me. A lot of grok's patterns use atomic groups, which aren't available in Python 2, so I wrote redi[3] to convert them from grok's syntax to Python compatible syntax.
So I can now colorize logs easily as follows:
tail -F program.log | synesthesia "$(redi "$(grokpat uuid)")"
that's way better than my alternative which was a bash function:
function hl() {
local R=''
while [ $# -gt 0 ]; do
R="$R|$1"
shift
done
env GREP_COLORS="mt=38;5;$((RANDOM%256))" egrep --color=always $R
}
I use htop and turn the colours off... thing I like about htop is I can use the mouse. It's so F-key-based, which, especially on a laptop, means you have to take your eyes off the screen to find the keys.
Of course, I'm not opposed to colours, I'm opposed to colours that look tacky / l33t hax0r, which would be most terminal stuff. I like 𝐛𝐨𝐥𝐝 better than trying to find colours I won't hate.
Of course, I'm not opposed to colours, I'm opposed to colours that look tacky / l33t hax0r, which would be most terminal stuff. I like 𝐛𝐨𝐥𝐝 better than trying to find colours I won't hate.
Switching between many programs and systems, I have a hard time keeping track of what colors mean what. Especially when people start customizing their colors. There are enough tools to show you what a file is, I don't need directories printed in red and files in blue and symlinks in green. ls -l is good enough and works everywhere.
Also take a look at ccze which is great for syntax-coloring all kinds of log files heuristically.
For example: tail -F somefile.log|ccze -A
For your shell, there's a lot of funky colorful interactive customizations you can get on zsh and fishfish.
For example: tail -F somefile.log|ccze -A
For your shell, there's a lot of funky colorful interactive customizations you can get on zsh and fishfish.
That's really nice – thanks for the pointer. "ssh … varnishncsa | ccze" worked first time without any config, exactly as you'd want.
Be very, very careful with this: http://seclists.org/fulldisclosure/2014/Nov/74
But really, it's 2015. Are you really using a terminal emulator without a buffer search? Are you not using screen or tmux, both of which can do these things natively, with no external tools? Why? And what is wrong with backgrounding the tail command, running your grep, and then foregrounding the tail command?
But really, it's 2015. Are you really using a terminal emulator without a buffer search? Are you not using screen or tmux, both of which can do these things natively, with no external tools? Why? And what is wrong with backgrounding the tail command, running your grep, and then foregrounding the tail command?
Personally? The reason I'm not using screen is that there's no way to atomically [attach, create new window] in screen and I value my gnome-terminal tabs.
If tmux can do that, then I'd consider it reason to switch.
If tmux can do that, then I'd consider it reason to switch.
Check out 'byobu'. Stupid name, but it's built on screen and can use the screen commands. Byobu has F2 for new terminal tab, F3/4 to shuffle between them.
Depending on what you mean by "atomic" (do you really mean atomic, or do you just not want to have to type anything after you've attached to create the new window?), you can do this with screen's -X option. If that's what you want to do I can try to dig up the way I used to do this.
Do you mean creating a new window every time you attach to an existing session? Or do you mean that each of those actions ("attach" and "create new window") should be atomic?
The problem is that less +F polls every second while tail -f uses inotify, which is both more efficient and responds faster to change.
There was a time when tail didn't know inotify. It was so, so much better when it learned that trick.
There was also a time when there was a utility called inotail, bridging that gap until tail proper was improved. I very much preferred it over regular tail.
There was also a time when there was a utility called inotail, bridging that gap until tail proper was improved. I very much preferred it over regular tail.
I always thought it was such a shame that you couldn't use plain poll() for this. poll()-ing on a read fd at EOF for a regular file should work like poll()-ing a network socket.
The problem is in what concepts are being considered "the same" when mapping two mostly-dissimilar things (a socket or pipe, vs a file).
poll(), select() et al don't define "readable" as meaning "a byte of data is available". They define it as "read() will not block".
When you read to the end of a file, read() returns "". This is the same as when you try to read from a closed socket. Conceptually, they are linking the concepts of "EOF" and "closed", not the concepts of "EOF" and "waiting for data". And indeed, if you call poll() on a socket that has been closed on the remote end, you will find it is "readable".
Ultimately, regular files just were never intended to be used as pipes. The abstractions just weren't chosen to work in that way.
poll(), select() et al don't define "readable" as meaning "a byte of data is available". They define it as "read() will not block".
When you read to the end of a file, read() returns "". This is the same as when you try to read from a closed socket. Conceptually, they are linking the concepts of "EOF" and "closed", not the concepts of "EOF" and "waiting for data". And indeed, if you call poll() on a socket that has been closed on the remote end, you will find it is "readable".
Ultimately, regular files just were never intended to be used as pipes. The abstractions just weren't chosen to work in that way.
> poll(), select() et al don't define "readable" as meaning "a byte of data is available". They define it as "read() will not block".
If that were true, poll() wouldn't return read-ready on a file-based fd until the data was actually sitting in buffers. After all, that's the only way to guarantee it won't block. That would actually be a useful semantic.
But what actually happens is that poll() on a file-based fd is basically a no-op that always returns true immediately, AFAIK. Someone could pull the disk drive cable before you actually call read(), in which case read() will indeed block, forever.
The existing semantic is useless. What argument is there in favor of a useless semantic?
This semantic wouldn't solve the "tail -f" problem, but at least it would be useful: http://cr.yp.to/unix/asyncdisk.html
If that were true, poll() wouldn't return read-ready on a file-based fd until the data was actually sitting in buffers. After all, that's the only way to guarantee it won't block. That would actually be a useful semantic.
But what actually happens is that poll() on a file-based fd is basically a no-op that always returns true immediately, AFAIK. Someone could pull the disk drive cable before you actually call read(), in which case read() will indeed block, forever.
The existing semantic is useless. What argument is there in favor of a useless semantic?
This semantic wouldn't solve the "tail -f" problem, but at least it would be useful: http://cr.yp.to/unix/asyncdisk.html
You can use kqueue/kevent to poll on a file. That's what tail(1) and other system utilities are using.
Not convinced, still like tail -f because you can hit enter a couple of times and there you are... a nice visual mark to wait for new data without getting confused by existing content.
All this in my humble opinion. less is useful and I may use F to wait for data if I was looking for something in old logs, but for new data I still like tail -f :)
All this in my humble opinion. less is useful and I may use F to wait for data if I was looking for something in old logs, but for new data I still like tail -f :)
It's a nice feature of less, but less is a pager, so you can't pipe the result to `grep` or `sed` or `awk` or something.
I'd wager that 90% of the time that I'm using 'tail -F' I'm also piping it to grep.
This. I use:
tail -f /var/log/x.log | grep foo -A 20 -B 20
Most of the time.
tail -f /var/log/x.log | grep foo -A 20 -B 20
Most of the time.
While you may not be able to do transforms on the output via `sed` or `awk`, you can get a grep-like filtering by using the `&` character with `less` open.
I use tail -f inside tmux anyways, so I already have the extra feature less offers ("navigation mode", searching, etc) without giving up the "watch multiple files" feature.
A much nicer alternative, IMHO.
A much nicer alternative, IMHO.
I use tail -f with tmux usually. Will less +F save the entire output into memory? In other words, if I accidentally leave it open in a tmux shell and the log grows to 300mb of output, will the server start swapping?
I usually run `less` niced and with a ulimit of 30 megabytes, but `less` also has an option to configure how much buffer space to use. As far as I can tell, the default is "all of your memory", which is a suboptimal default in my book.
That's an important point: never use this trick with huge logs...
There's also lnav (http://lnav.org), the Logfile Navigator. It can do what 'tail -f' and 'less' do plus: syntax coloring, filtering, loading compressed files, interleaves log messages from multiple files, and more.
Many of the comments I see on here about using '-F' with tail instead of '-f', live searching, and so on are handled in lnav by default. I would really encourage folks to give it a try.
Many of the comments I see on here about using '-F' with tail instead of '-f', live searching, and so on are handled in lnav by default. I would really encourage folks to give it a try.
One advantage that tail -f has is that you can insert newlines or other characters into the output to help you visually split log entries. Log files often look the same so being able to visually split up the log is very helpful IMO.
Huge caveat: "less +F" does not work with multiple files.
Yes, this is mentioned in the post, but I feel it is so big it would have deserved large, bold letters.
Yes, this is mentioned in the post, but I feel it is so big it would have deserved large, bold letters.
Interesting.
I always preferred
because of its nice highlighting feature.
I always preferred
watch -n 1 -d 'tail /path/to/log/file'
because of its nice highlighting feature.
watch is one of the great unsung heroes of unix tools. Discovering it was one of those "I wish I knew about this years ago" moments. Unfortunately it doesn't seem to have spread much beyond GNU/Linux platforms.
+1 for Commandline Fu. I love it when you see a colleague at a terminal and you say "Whoah, go back, what was that you just did?". It's kinda beautiful really, the way everyone has slightly different methods for common dev tasks at the commandline.
My method is usually '+G' to go to the end of the log file and check the damage report. Quick reverse search with '?' + search term to find last error, then probably hit 'n' to find the next previous any other occurrences to determine how frequent it's popping up. If it looks like it's a recurring problem i'll '+F' to see follow latest output. Then CTRL+c to quit that mode.
tail -f still has it's uses for cases as pimlottc mentioned. I like to put a bit of distance between the last bit of output. Especially if its a particularly repetitive and lengthy stacktrace.
My method is usually '+G' to go to the end of the log file and check the damage report. Quick reverse search with '?' + search term to find last error, then probably hit 'n' to find the next previous any other occurrences to determine how frequent it's popping up. If it looks like it's a recurring problem i'll '+F' to see follow latest output. Then CTRL+c to quit that mode.
tail -f still has it's uses for cases as pimlottc mentioned. I like to put a bit of distance between the last bit of output. Especially if its a particularly repetitive and lengthy stacktrace.
I use tac file |less often to start at the end of the file..
I use tail -f mainly for tracking something end-to-end through multiple log files because it's pipeable to grep or awk, where less is not. less +F is cool too, but tail -f is more useful when doing complex log analysis on the fly because it follows standard UNIX I/O behavior.
I just tried this on an extremely active logfile and it didn't work very well for me. I'd recommend multitail for watching constantly changing files.
Stop using [commonly accepted, proven method] and use [obscure alternative]
Textbook HN hit.
Textbook HN hit.
I use tail -f and when I need to search, I use iTerm's search (Cmd+F) which highlights matches and even supports regex. I also have an unlimited scroll buffer so I can just scroll up or down to get the context of a particular match.
This only works if what you are searching for has scrolled past since you started following. If you want to see other occurrences in the file you're out of luck. For example, let's say you are watching to see which error message is generated when something goes wrong, then you want to see when else that error message has been thrown.
What do people (you) think about browser based log viewers like e.g. logio.org? I like to comfortably have all the logs in the browser but do not want to go full ELK (or anything in this category) stack just for a few servers. But I am not sure if there is some better solution? Multitail is great, but sometimes just opening a new browsertab is even greater...
For lazy people out there:
There is also the command `tailf` which is quite similar to 'tail -f'.
2 characters make so much difference :)
There is also the command `tailf` which is quite similar to 'tail -f'.
2 characters make so much difference :)
alias tf='tail -f'
I saved you three more!
I saved you three more!
GRUB_CMDLINE_LINUX_DEFAULT="init=/usr/bin/tail"
Maximum savings!
Maximum savings!
For semi-large file (1/2GB+) less takes seconds if not minutes to count the number of lines (it read the whole file) so you can navigate and search across the file.
In that cases tail -f and grep are still way faster than less, though less is a handier tool
You don't have to wait. You can turn that off with '-n'.
When you jump to the end of the file ('>' command) you'll see "Calculating line numbers... (interrupt to abort)" and interrupt (ctrl-C) here will also turn them off.
When you jump to the end of the file ('>' command) you'll see "Calculating line numbers... (interrupt to abort)" and interrupt (ctrl-C) here will also turn them off.
I didn't know about the -n switch. Usually I need to get to the bottom of the file and find the last occurrence of a string in the logs (G, CTRL+C to stop counting lines, ?string to search) and the slower part is waiting for less to match a string going backwards on the file.
Will check if the -n switch solves the problem
Searching with less is a cool feature, but I think the best thing I learned was to use -F instead of -f with tail thanks to the comments on the article and HN.
Of course I'll probably fail to use any of it because of finger memory just typing out tail -f for me.
Of course I'll probably fail to use any of it because of finger memory just typing out tail -f for me.
I would be careful about such casual use of less: http://seclists.org/fulldisclosure/2014/Nov/74
It's amusing to read this just as, for the first time in months, I have a "tail -f" running. I just converted a medium-sized system from Python 2 to Python 3 (this is not fun; I had to fix or work around five bugs in external modules), and deployed it this morning. So I have a "tail -f" running in "putty", watching an Apache server error log. (I know, retro.) This is just in case there's some new Python-level error; operational errors are logged to the database. The log is quiet, and soon I can stop watching.
I was interested in the significance of the plus for this flag. If anyone else was curious: from the manual, the + flag is for initial commands to less, so this is equivalent to running less and typing F.
Thanks! This is really great trick. I will obviously use less +F for tailing single file.
Meanwhile, tail -f still useful for simple tailing to multiple machine, like
Meanwhile, tail -f still useful for simple tailing to multiple machine, like
function dogfight_tail {
logfile=/var/log/nginx/access.log
pids=""
for box in 02 03; do
ssh lb$box tail -f $logfile | grep " 500 " & # find error machines
pids="$pids $!"
done
trap 'kill -9 $pids' SIGINT
trap wait
}
dogfight_tailI use "tail -f" most of the time. Sometimes, I just dig more and use grep with it.
tail -f sys.log | grep NETDEV
But I would definitely give this a try, thanks for sharing.
tail -f sys.log | grep NETDEV
But I would definitely give this a try, thanks for sharing.
I think that it's worth noting that you should still use `-f` if you are viewing a file over SSH that is being appended to a ton and you don't need every line. SSH will require that every line makes it to your terminal over it's secure connection, so you will be bottlenecked and most likely won't be viewing the latest messages. Everything will be queued.
I think I'll do the following (as a joke, surprised no one did it already):
$ alias stalk='less +F'I just had an SRE screening and one of the questions was
"What is a linux command that's mostly not known?".
I could not remember anything that's special, at least for me at the time but "less -F" poped up to my mind. Thank you :)
Oh now I can think of ccze, column and tac. Interviews..
"What is a linux command that's mostly not known?".
I could not remember anything that's special, at least for me at the time but "less -F" poped up to my mind. Thank you :)
Oh now I can think of ccze, column and tac. Interviews..
With `tmux` there is a visual mode that allows to `tail -F` with search, selection etc.
For something simple and very lightweight for Windows development, check out BareTail: http://www.baremetalsoft.com/baretail/
Haha I didn't know a lot of people didn't know about this. I am one of those people who likes to read man pages :D Another favorite one of mine is the `v` switch to quickly jump to the editor from `less`. Pretty handy!
While I understand that the author is just trying to be helpful, the provocative title of the article makes me want to scream: "Stop telling me which tool is right for MY job."
I've literally never had this problem and nothing is wrong with tail -f for me. Quitting tail to use grep on the file is anything but inconvenient for me.
[deleted]
This is pretty awesome. I have always thought that "tail -f" was a little limited but still useful. Any how, great post!
For log files that move particularly fast, I prefer to use "watch tail -n 100" so I don't choke my connection.
Good share! Command line fu is always nice.
$make -j16 showcommands > build.log 2>&1 | less +F
ctrl+c...
<shit its not working>
ctrl+c * 1e9 impatiently
...
build stopped
oops
ctrl+c...
<shit its not working>
ctrl+c * 1e9 impatiently
...
build stopped
oops
It would be awesome to remove the "Waiting for data... (interrupt to abort)" message.
Passing -Pw without a prompt at least gets that down to "... (interrupt to abort)"
Or just tail -f in an emacs buffer
You mean like in multiterm or something? I guess so. In emacs there are always many ways to do things, which is awesome.
You can also open the file and engage auto-revert-mode or auto-revert-mode-tail. In my status bar it shows up as ARev mode.
Also you can bind those to your auto-mode-alist so all .log files get the tail treatment or whatever.
Scrolling can get weird.
Theres some google-able settings for the timeout to revert, default is about a sip of water (a few seconds?) but I guess you could crank it down to 1 second or something.
I like doing this in emacs so its all in the same ecosystem, if I need to cut and paste something weird to test it or put the message into code or even more likely to cut and paste actual error messages into docs or comments, I can do it inside emacs.
You can also open the file and engage auto-revert-mode or auto-revert-mode-tail. In my status bar it shows up as ARev mode.
Also you can bind those to your auto-mode-alist so all .log files get the tail treatment or whatever.
Scrolling can get weird.
Theres some google-able settings for the timeout to revert, default is about a sip of water (a few seconds?) but I guess you could crank it down to 1 second or something.
I like doing this in emacs so its all in the same ecosystem, if I need to cut and paste something weird to test it or put the message into code or even more likely to cut and paste actual error messages into docs or comments, I can do it inside emacs.
It looks like from ~24.4 onwards there's support for various notification event sources for auto-revert[-tail]-mode, as determined by the file-notify--library variable. I don't appear to have anything compiled into my prebuilt OSX emacs though, time to update the ol' source tree and see if I can remember how to build it.
"here you have pretty much the same behavior"
what is meant by 'pretty much?'
what is meant by 'pretty much?'
Perhaps neovim with terminal mode and tail -F might be the best of all worlds.
Doesn't read ANSI color codes! Looking through manual for that.
SOLVED: less +F -R <file>
Interprets color codes as RAW, relaying them to the screen as-is.
Interprets color codes as RAW, relaying them to the screen as-is.
less +F is not such a good alternative to "tail -f" or better "tail -F" especially when you are filtering the output of tail using grep or awk.
I imagine you could probably do "tail -f | grep/awk ... | less +F".
It works fine indeed. I'm using it all the time.
Imma just gonna leave this here: http://seclists.org/fulldisclosure/2014/Nov/74
Why not just pipe the output of tail -f through less?
Put that kind of suggestion in coderwall.com
Solaris 10 doesn't have the -F flag.
Can I hack someone's pay face book
U dont tell me what to do!
i like my tail
/thefox
/thefox
Can you hack face book account
its not better for me, hitting enter is essential for me when using tail
...or use multi-tail
wow... this is really a nice coomand
Nice!
Wait... Isn't less gonna read the whole file at first? That might suck for files that are already huge.
No. I use less to navigate gigabyte log files quite often (it's more efficient then vim).
Linux just really needs PowerShell.
hahahaha, No.
Enjoy your parsing text files. I'll work with objects instead.
Eventually Microsoft will be able to reinvent Unix. Just keep the faith.
[deleted]
tail works best for me. I'm not going stop using it just because a person on the internet says not to.
The title is a bit clickbaity, but ignore that and it's a useful tip you might not know. The author doesn't actually demand you stop using tail -f if you like it better.
With less, you have to keep track of the current position by timestamp or other unique message, and it's easy to lose your place when the output starts streaming in. With tail, you only need a moment to mark your spot and then it's visually distinct even as more message come in.