Confusing Grep Mistakes I've Made(blog.robertelder.org)
blog.robertelder.org
Confusing Grep Mistakes I've Made
https://blog.robertelder.org/top-5-grep-mistakes/
50 comments
Or $'\t' can be used in bash to get some C-style escapes. (The ^V thing is epic though!)
By the way, that technique also works in vim and some other editors, I think.
Yep. Allows for some seriously cool programmatic editing:
g/^abc/norm ^3wciwHello^V^]2ei!
[deleted]
My favorite grep mistake is actually from a related tool pgrep and pkill.
pgrep foo -> finds things running matching foo
pkill foo -> kills things running matching foo
$ pkill foo
$ echo nothing happened?
$ pkill -9 foo
$ echo nothing happened still? huh?
$ echo ok, let's run this in verbose mode..
$ pkill -9 foo -v
According to the man page, related to the -v option:
> In pkill's context the short option is disabled to avoid accidental usage of the option.
So at least in the version that I have installed (procps-ng 3.3.15), pkill -v results in an error message, not the catastrophic situation suggested by the parent comment.
> In pkill's context the short option is disabled to avoid accidental usage of the option.
So at least in the version that I have installed (procps-ng 3.3.15), pkill -v results in an error message, not the catastrophic situation suggested by the parent comment.
It was fixed in 2012. The debian bug report (2009) cites numerous users that fell into the trap before the fix.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=558044
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=558044
commit 1af18c260a87dc38f0e33bfeb6de6163f91be4ad
Author: Sami Kerola <[email protected]>
Date: Sat Feb 11 20:33:17 2012 +0100
pkill: remove -v match inversion option
The option -v does not make much sense in pkill context.On many systems,
Usually what you want instead is
FWIW I cannot reproduce this on my system any longer; it seems to vary by distribution. See e.g. [1].
[1] https://unix.stackexchange.com/questions/15980/does-should-l...
grep '[A-Z]'
Usually what you want instead is
LC_ALL=C grep '[A-Z]'
grep '[[:upper:]]'
FWIW I cannot reproduce this on my system any longer; it seems to vary by distribution. See e.g. [1].
[1] https://unix.stackexchange.com/questions/15980/does-should-l...
> 3) Confusing '.' with '\.'
Because of how different languages handle escaping within strings and not wanting to have to think about it, I've started using [.] to get a literal dot because it always means what I want. I still don't like it.
Because of how different languages handle escaping within strings and not wanting to have to think about it, I've started using [.] to get a literal dot because it always means what I want. I still don't like it.
Another I've run into:
If you meant to search in ∗, but somehow completely forgot to type the ∗ at the end of the command line, you might do something like
(I don't find this conceptually confusing -- I expect many Unix tools, including grep, to act on their standard input -- but I've still sometimes simply forgotten the * and not noticed right away.)
If you meant to search in ∗, but somehow completely forgot to type the ∗ at the end of the command line, you might do something like
grep foo
(I don't find this conceptually confusing -- I expect many Unix tools, including grep, to act on their standard input -- but I've still sometimes simply forgotten the * and not noticed right away.)
With (g)awk it can be even worse.
$ awk 'BEGIN{while(getline<"phonebook")p[$1]=$2} $1 in p{print $0,p[$1]}' names.tsv
Don't feel bad, I've done this probably a thousand times. Probably because I normally am using -r which doesn't have that behavior. I usually wonder 'wow this directory was bigger than I thought' until realizing my mistake. In the modern SSD era at least, you don't waste -minutes-, usually.
To add to the confusion grep -r without a path will recursively search the . directory.
If you are interested in where the name "grep" came from:
g/re/p
g: global
re: regular expression
p: print.
https://tldp.org/LDP/abs/html/textproc.html
g/re/p
g: global
re: regular expression
p: print.
https://tldp.org/LDP/abs/html/textproc.html
Often my biggest mistake is using grep instead of rg.
Tripping over the escaping rules is a continuing pain. GNU sed does things differently, using \| for the regex or operator. [0] It's fiddly enough to baffle the occasional StackOverflow answerer. [1]
[0] https://www.gnu.org/software/sed/manual/sed.html#BRE-syntax
[1] https://stackoverflow.com/a/6388042/
[0] https://www.gnu.org/software/sed/manual/sed.html#BRE-syntax
[1] https://stackoverflow.com/a/6388042/
Just use egrep instead of grep. Much easier to remember / less surprising behavior, and it’s supported on every system out of the box (unlike rg).
Related, the ack website has a comparison chart of various grep competitors:
https://beyondgrep.com/feature-comparison/
https://beyondgrep.com/feature-comparison/
Most of the mistakes are not exclusive/related to grep, they are actually shell mistakes.
This is true, and I agree. But this kind of "language-within-language" problem (regex within sh) comes up a lot when formulating grep and sed command lines.
Come to think of it, I think this is the most frequently-encountered class of "this line didn't do what I thought it would" type errors that I get as a near-daily user of (ba)sh for a few decades now.
Question for the group: When you encounter this kind of issue, e.g. the shell is stealing a single or double quote meant to be in the regex, do you diagnose the problem w/r/t the shell precedence rules for quotes and backslashes, or do you just blindly put the opposite type of quote around the whole thing and re-run?
Because probably half the time I use a basic blind strategy, and that's not usually how I approach programming errors!
Come to think of it, I think this is the most frequently-encountered class of "this line didn't do what I thought it would" type errors that I get as a near-daily user of (ba)sh for a few decades now.
Question for the group: When you encounter this kind of issue, e.g. the shell is stealing a single or double quote meant to be in the regex, do you diagnose the problem w/r/t the shell precedence rules for quotes and backslashes, or do you just blindly put the opposite type of quote around the whole thing and re-run?
Because probably half the time I use a basic blind strategy, and that's not usually how I approach programming errors!
agree, and the author continues to use double quotes around search pattern despite showing an example where single quote was needed!
As a good practice, I always try to single quote the expression, even if it is not needed. Use double quote only when needed and even then, use it only for the portion required, not for the entire expression.
https://mywiki.wooledge.org/Quotes is a must read.
As a good practice, I always try to single quote the expression, even if it is not needed. Use double quote only when needed and even then, use it only for the portion required, not for the entire expression.
https://mywiki.wooledge.org/Quotes is a must read.
And simply regex mistakes.
I like `grep -F` - it treats the search as a literal; no more escaping regex when you are really wanting a ".".
I have a list of gotchas and tricks here: https://learnbyexample.github.io/learn_gnugrep_ripgrep/gotch...
As pointed out in other comments, many of the issues in the post is due to shell, not specific to grep. Especially quoting. Always use single quotes to specify the search pattern, unless other forms of shell quoting is needed. Otherwise, you'll face issues with commands like
As pointed out in other comments, many of the issues in the post is due to shell, not specific to grep. Especially quoting. Always use single quotes to specify the search pattern, unless other forms of shell quoting is needed. Otherwise, you'll face issues with commands like
grep ; ip.txt
$ echo '5*3-2=13' | grep '-2'
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
That's a really good article!
One of the most common mistakes I see is missing escaping. Not strange when the rules change between the various regexp modes.
Something like 'grep "file.exe"' is probably not what you want, and sometimes it can be easier to turn off regexp processing with -F in the the cases that doesn't use them.
One of the most common mistakes I see is missing escaping. Not strange when the rules change between the various regexp modes.
Something like 'grep "file.exe"' is probably not what you want, and sometimes it can be easier to turn off regexp processing with -F in the the cases that doesn't use them.
Quoting and parsing continue to surprise people, and I don't blame them -- you're embedding one programming language inside another (regex inside bash), and they use some of the same reserved symbols and have slightly different quoting rules. And, every language is "inspired" by the others, but have their own special rules, so the more you learn, the less sure of anything you'll ever be. (For example, '(' matches a literal parenthesis in Emacs Lisp regexes, and '\(' starts a capture group!)
For matching literal periods, I personally have gotten into the habit of using "[.]" instead of "\.". Less to go wrong in this double-embed scenario, and I have never ever regretted adding the extra byte to my regexp. (Of course, character classes have their own weirdness. Your editor that matches bracket paris will love the syntax for matching a literal '['.)
For matching literal periods, I personally have gotten into the habit of using "[.]" instead of "\.". Less to go wrong in this double-embed scenario, and I have never ever regretted adding the extra byte to my regexp. (Of course, character classes have their own weirdness. Your editor that matches bracket paris will love the syntax for matching a literal '['.)
As confusing as multi-layered escaping can be, I still vastly prefer the situation in Linux and other Unix-likes to that of Windows where each process is given nothing more than a mildly-processed command line from the shell (e.g. no globbing) as a string, and parses it however it wants to. Escaping in batch files is slightly different from interactive commands, each utility may have its own escaping conventions on top of those the shell has, etc. Linux and the like have a far more predictable and consistent experience, since argument splitting and globbing is always done once by the shell, and each process gets the already-split and expanded argument list.
Another basic problem I run into is that grep returns whether it found a match or not as its exit status. This means if you, for example, run a script with bash option
set -e
I run scripts using both dash and bash and always use `set -eu`. Returning a false exit status is not a problem if you wrap the `grep` inside an `if` statement:
if grep pattern file; then echo y; fiMost of these things boil down to a few very simple rules (because I rarely run into these problems and I certainly don't have more than a few simple rules).
1. If your pattern contains variable expansion wrap it in double quotes (but watch out for shell variable expansion characters like '$`')
2. Else wrap your search string in single quotes.
3. If you're gonna do any sort of regex at all just use the '-E' flag so it behaves like a proper regex, and learn how to do basic regex.
4. Know your shell. Some of these gotchas come from the way the shell interprets the command. Again, most shell gotchas boil down to a few basic rules too (eg the single or double quote thing). For example, in bash I always surround variables in curly braces: ${THIS}. It avoids accidental bash variable expansion, or confusion about the precise name of the variable when it is concatenated with other strings in the pattern.
One trick I like is how to use grep as a highlighter.
grep --color -E '(^|My text)'
This matches every line because of the ^ but has nothing to color except your string (since the start of line character is not visible)
Also, the -A and -B flags are useful for grabbing lines after/before the pattern. And while -C doesn't make as much sense as those, its meaning logically follows A and B: grab the lines before and after.
Lastly, if you want to search a gigantic directory structure for files containing an expression, but do not want to hit every single file (eg: restrict it to '.c' files), you can use this:
find /path/to/dir -name ".c" -exec grep -H "pattern {} \;
The -H forces grep to show the file name. This gets activated by default when you grep multiple files, but find command executes a separate grep on each individual file, so you lose the filename. -H explicitly adds it.
Once you the -H parameter, it's easy to remember that -h turns filename off*. For example, to make a playlist containing all Bob Marley songs that are used in other .m3u playlists:
grep -r -h "Bob Marley" /home/user/My Music/playlists" > marley.m3u
This simple approach avoids having to pipe the output into another command like sed or awk to strip away the path, which may not be as simple as it sounds because the file paths may contain spaces and all sorts of other junk.
1. If your pattern contains variable expansion wrap it in double quotes (but watch out for shell variable expansion characters like '$`')
2. Else wrap your search string in single quotes.
3. If you're gonna do any sort of regex at all just use the '-E' flag so it behaves like a proper regex, and learn how to do basic regex.
4. Know your shell. Some of these gotchas come from the way the shell interprets the command. Again, most shell gotchas boil down to a few basic rules too (eg the single or double quote thing). For example, in bash I always surround variables in curly braces: ${THIS}. It avoids accidental bash variable expansion, or confusion about the precise name of the variable when it is concatenated with other strings in the pattern.
One trick I like is how to use grep as a highlighter.
grep --color -E '(^|My text)'
This matches every line because of the ^ but has nothing to color except your string (since the start of line character is not visible)
Also, the -A and -B flags are useful for grabbing lines after/before the pattern. And while -C doesn't make as much sense as those, its meaning logically follows A and B: grab the lines before and after.
Lastly, if you want to search a gigantic directory structure for files containing an expression, but do not want to hit every single file (eg: restrict it to '.c' files), you can use this:
find /path/to/dir -name ".c" -exec grep -H "pattern {} \;
The -H forces grep to show the file name. This gets activated by default when you grep multiple files, but find command executes a separate grep on each individual file, so you lose the filename. -H explicitly adds it.
Once you the -H parameter, it's easy to remember that -h turns filename off*. For example, to make a playlist containing all Bob Marley songs that are used in other .m3u playlists:
grep -r -h "Bob Marley" /home/user/My Music/playlists" > marley.m3u
This simple approach avoids having to pipe the output into another command like sed or awk to strip away the path, which may not be as simple as it sounds because the file paths may contain spaces and all sorts of other junk.
Well, not so confusing, but accidentally printing negative matches. Once it locked up my shell instance, and I had to ssh in again to kill it.
How many of these gotchas exist with ripgrep?
Most of them are shell gotchas, and so would apply to rg
On Windows, if you want to match a line ending with $, then you have to use the --crlf option.
When enabled, ripgrep will treat CRLF (\r\n) as a line terminator instead of just \n.
I really wish this was enabled by default on Windows builds.
When enabled, ripgrep will treat CRLF (\r\n) as a line terminator instead of just \n.
I really wish this was enabled by default on Windows builds.
ripgrep at least won't suffer from the UTF-16 problem. It will automatically handle those files correctly by transcoding to UTF-8.
What's a reasonable grep alias I should add to my .bashrc?
The only flag I'd say I want 99% of the time is -E (same as calling egrep) because extended regular expressions are what you almost always want.
A sibling comment suggested -i, other useful flags are -w (find whole words) and -C (change the amount of context around matches), but I don't think I always use those.
A handy function for searching a tree:
All that said... when I'm searching a tree, I can usually use 'git grep'.
A sibling comment suggested -i, other useful flags are -w (find whole words) and -C (change the amount of context around matches), but I don't think I always use those.
A handy function for searching a tree:
greptree() {
find "$1" -name "$2" -type f -exec egrep "$3" '{}' +
}
greptree . '*.java' 'foo[123]'
All that said... when I'm searching a tree, I can usually use 'git grep'.
grep -P to use pcre (i.e. what everyone knows)
I have `gi` aliased to `grep -i`.
grep -r --exclude-dir=.git --exclude-dir=.svn --color=auto
[deleted]
[deleted]
Better title: grep newbie mistakes
For some of the mistakes (like .* vs '.*') maybe, but I doubt even experienced grep users are aware of all these things (e.g. how \t is interpreted in BRE vs. extended vs. PCRE modes).
On a linked page, he recommends to use -P in all cases without mentioning that the BSD greps don't support it. I think "newbie" is a proper description.
If you need to be told that grep supports different flavours of regular expressions, you are a grep newbie. And although even experienced users may be bit by errors related to the flavour of regular expressions in use, they surely don't need TFA to know the source of the errors.
Something that stood out for me is that the author did not mention ^V, which is very useful in quoting metacharacters. Take the tab example: The author seems to imply that PCRE is needed to match a tab because there is no \t escape sequence in BRE/ERE. Presumably he cannot just type in a tab because he's using a shell like bash, and tab has a special interpretation and cannot be typed in as a string literal.
The way around this is to use ^V as a terminal escape sequence, followed by simply pressing the tab key. This technique can be used to insert other control characters as string literals in arguments. Want to grep for EOF? "grep ^V^D" will get you there.