No ErrorDocument reaches limit of 10 internal redirects

Recently, I was blocking some HTTP_USER_AGENTs on our Apache 2.2.9 web server with

RewriteCond %{HTTP_USER_AGENT} ^Casper\ Bot\ Search$
RewriteRule . http://127.0.0.1/

This worked as expected. Whenever the bot hit the server it got a 302. But when I changed the RewriteRule to Fail

RewriteCond %{HTTP_USER_AGENT} ^Casper\ Bot\ Search$
RewriteRule . - [F]

the server produced a 500 Internel server error instead of the expected 403. The error.log showed the infamous message

Request exceeded the limit of 10 internal redirects due to probable configuration error.
Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug'
to get a backtrace.

It took me a while to find that the instruction ErrorDocument 403 default is actually what helped even though I did not have ErrorDocument set at all (i.e. in any other Apache configuration file) and hence the setting default should have been effective anyway. Or so I thought.

So, in my case the problem was not an incorrect ErrorDocument setting but the fact that it was not set at all.

ErrorDocument 403 default

RewriteCond %{HTTP_USER_AGENT} ^Casper\ Bot\ Search$
RewriteRule . - [F]