parsing - How to address error "bash: !d': event not found" in Bash command substitution

Question

This question already has answers here:

Answer 1

Bash history expansion is a very odd corner in the bash command line parser, and you are clearly running into an unexpected history expansion, which is explained below. However, any sort of history expansion in a script is unexpected, because normally history expansion is not enabled in scripts; not even scripts run with the source (or .) builtin.

How history expansion is enabled (or disabled)

There are two shell options which control history expansion:

• set -o history: Required for the history to be recorded.

• set -H (or set -o histexpand): Additionally required for history expansion to be enabled.

Both of these options must be set for history expansion to be recognized. (I found the manual unclear on this interaction, but it's logical enough.)

According to the bash manual, these options are unset for non-interactive shells, so if you want to enable history expansion in a script (and I cannot imagine a reason you would want this), you would need to set both of them:

set -o history -o histexpand

The situation for scripts run with source is more complicated (and what I'm about to say only applies to bash v4, and since it's undocumented in might change in the future). [Note 3]

History recording (and consequently expansion) is turned off in source'd scripts, but through an internal flag which, as far as I know, is not made visible. It certainly does not appear in $SHELLOPTS. Since a sourced script runs in the current bash context, it shares the current execution environment, including shell options. So in the execution of a sourced script initiated from an interactive session, you'll see both history and histexpand in $SHELLOPTS, but no history expansion will take place. In order to enable it, you need to:

set -o history

which is not a no-op because it has the side-effect of resetting the internal flag which suppresses history recording. Setting the histexpand shell option does not have this side-effect.

In short, I'm not sure how you managed to enable history expansion in a script (if, indeed, the misbehaving command was in a script and not in an interactive shell), but you might want to consider not doing so, unless you have a really good reason.

How history expansion is parsed

The bash implementation of history expansion is designed to work with readline, so that it can be performed during command input. (By default this function is bound to Meta-^; generally Meta is ESC, but you can customize that as well.) However, it is also performed immediately after each line is input, before any bash parsing is performed.

By default, the history expansion character is !, and -- as mostly documented -- that will trigger history expansion except:

1. when it is followed by whitespace or =

2. if the shell option extglob is set, and it is followed by ( [Note 1]

3. if it appears in a single-quoted string

4. if it is preceded by a [Note 2 and see below]

5. if it is preceded by $ or ${ [Note 1]

6. if it is preceded by [ [Note 1]

7. (As of bash v4.3) if it is the last character in a double-quoted string.

The immediate issue here is the precise interpretation of the third case, an ! appearing inside of a single-quoted string. Normally, bash starts a new quoting context for a command substitution ($(...) or the deprecated backtick notation). For example:

$ s=SUBSTITUTED
$ # The interior single quotes are just characters
$ echo "'Echoing $s'"
'Echoing SUBSTITUTED'
$ # The interior single quotes are single quotes
$ echo "$(echo 'Echoing $s')"
Echoing $s

However, the history expansion scanner isn't that intelligent. It keeps track of quotes, but not of command substitution. So as far as it is concerned, both of the single quotes in the above example are double-quoted single quotes, which is to say ordinary characters. So history expansion occurs in both of them:

# A no-op to indicated history expansion
$ HIST() { :; }
# Single-quoted strings inhibit history expansion
$ HIST
$ echo '!!'
!!
# Double-quoted strings allow history expansion
$ HIST
$ echo "'!!'"
echo "'HIST'"
'HIST'
# ... and it applies also to interior command substitution.
$ HIST
$ echo "$(echo '!!')"
echo "$(echo 'HIST')"
HIST

So if you have a perfectly normal command like sed '/foo/!d' file, where you would expect the single-quotes to protect you from history-expansion, and you put it inside a double-quoted command substitution:

result="$(sed '/foo/!d' file)"

you suddenly find that the ! is a history expansion character. Worse, you can't fix this by backslash escaping the exclamation point, because although "!" inhibits history expansion, it doesn't remove the backslash:

$ echo "!"
!

In this particular example -- and the one in the OP -- the double quotes are completely unnecessary, because the right-hand side of a variable assignment does not undergo either filename expansion nor word splitting. However, there are other contexts in which removing the double quotes would change the semantics:

# Undesired history expansion
printf "The answer is '%s'
" "$(sed '/foo/!d' file)"

# Undesired word splitting
printf "The answer is '%s'
" $(sed '/foo/!d' file)

In this case, the best solution is probably to put the sed argument in a variable

# Works
sed_prog='/foo/!d'
printf "The answer is '%s'
" "$(sed "$sed_prog" file)"

(The quotes around $sed_prog were not necessary in this case but usually they would be, and they do no harm.)

---

Notes:

1. The inhibition of history expansion when the following character is some form of open parenthesis only works if there is a corresponding close parenthesis in the rest of the string. However, it doesn't have to really match the open parenthesis. For example:

   # No matching close parenthesis
   $ echo "!("
   bash: !: event not found
   # The matching close parenthesis has nothing to do with the open
   $ echo "!(" ")"
   !( )
   # An actual extended glob: files whose names don't start with a
   $ echo "!(a*)"
   b
   

2. As indicated in the bash manual, a history-expansion character is treated as an ordinary character if immediately preceded by a backslash. This is literally true; it doesn't matter whether the backslash will later be considered an escape character or not:

   $ echo !
   !
   $ echo \!
   !
   $ echo \!
   !
   

   also inhibits history expansion inside double quotes, but ! is not a valid escape sequence inside the double quoted string, so the backslash is not removed:

   $ echo "!"
   !
   $ echo "\!"
   !
   $ echo "\!"
   \!
   

3. I'm referring to the source code for bash v4.2 as I write this, so any undocumented behaviour may be completely different as of v4.3.