batch file - Why is escaping exclamation marks not necessary in parameter of `for /F` or `for /R`?

Question

When delayed expansion is enabled, it is (usually) necessary to escape exclamation marks properly in order to get literal ones (like ^^! or, when being in between "", ^!, in order to get a literal !).

However, why is no escaping of exclamation marks necessary but even distructive within the parameters provided immediately after the /R or the /F switch of the for loop command?

---

With the aforementioned escaping rules in mind, I created the following batch script using for /R and as a parameter a directory with an ! in its name (stated after the /R option):

@echo off
setlocal EnableExtensions EnableDelayedExpansion

set "TARGET=excl^!dir"
set "FILE=empty_%RANDOM%.txt"
echo/
echo creating directory: !TARGET!
mkdir "!TARGET!"
echo placing empty file: !TARGET!\%FILE%
> "!TARGET!\%FILE%" break
echo/
echo adequately escaped `^^!`:
for /R "excl^!dir" %%F in ("*.*") do echo(found item: %%~nxF
for /R excl^^!dir %%F in ("*.*") do echo(found item: %%~nxF
echo/
echo improperly escaped `^^!`:
for /R "excl!dir" %%F in ("*.*") do echo(found item: %%~nxF
for /R excl!dir %%F in ("*.*") do echo(found item: %%~nxF
for /R excl^!dir %%F in ("*.*") do echo(found item: %%~nxF

erase "!TARGET!\%FILE%"
rmdir "!TARGET!"

endlocal
exit /B

Surprisingly, the directory is enumerated only without escaping the !. This is an output:

creating directory: excl!dir
placing empty file: excl!dirempty_18378.txt

adequately escaped `!`:

improperly escaped `!`:
found item: empty_18378.txt
found item: empty_18378.txt
found item: empty_18378.txt

I expect the directory to be enumerated in case the exclamation mark is in fact properly escaped, otherwise not; but the result shows the opposite behaviour.

---

A similar phenomenon arises with for /F, like in the following script with an ! in the option string:

@echo off
setlocal EnableExtensions EnableDelayedExpansion

set "STRING=EXCLAMATION ^! MARK AND CARET ^^ SYMBOL"
echo/
echo normal  expansion: %STRING%
echo delayed expansion: !STRING!
echo/
echo adequately escaped `^^!`:
for /F "tokens=1-2 delims=^!" %%K in ("!STRING!") do echo(%%K -- %%L
for /F tokens^=1-2^ delims^=^^! %%K in ("!STRING!") do echo(%%K -- %%L
echo/
echo improperly escaped `^^!`:
for /F "tokens=1-2 delims=!" %%K in ("!STRING!") do echo(%%K -- %%L
for /F tokens^=1-2^ delims^=! %%K in ("!STRING!") do echo(%%K -- %%L
for /F tokens^=1-2^ delims^=^! %%K in ("!STRING!") do echo(%%K -- %%L

endlocal
exit /B

The output looks like this, strangely:

normal  expansion: EXCLAMATION  MARK AND CARET  SYMBOL
delayed expansion: EXCLAMATION ! MARK AND CARET ^ SYMBOL

adequately escaped `!`:
EXCLAMATION  --  MARK AND CARET
EXCLAMATION  --  MARK AND CARET

improperly escaped `!`:
EXCLAMATION  --  MARK AND CARET ^ SYMBOL
EXCLAMATION  --  MARK AND CARET ^ SYMBOL
EXCLAMATION  --  MARK AND CARET ^ SYMBOL

Here I expect the whole string to be split into exactly two tokens: EXCLAMATION and MARK AND CARET ^ SYMBOL. But the second token is too short, everything beginning from the caret symbol ^ is missing; so I conclude that the ^ used for escaping of the ! is taken literally. With no (or poor) escaping, the returned tokens are the intended ones.

Answer 1

The options of FOR, IF and REM are only parsed up to the special character phase.
Or better the commands are detected in the special character phase and a different parser is activated then.

Therefore it's not possible to use delayed expansion nor FOR-Parameter in the options.

These tests fail with errors

for /F %%O in ("defined") do (
  if %%O var echo yes
)

set option=defined
if !option! var echo yes

This seems to work, but uses the wrong delimiters (D, e, i, l, m, s, y and ! )

set "myDelims=123"
for /F "tokens=1,2 delims=!myDelims!" %%A in ("Hello 1 world") do (
    echo Token1=%%A  Token2=%%B
)

And for REM

set "help=/?"
REM !HELP!  - No help will be shown