Because we were talking about fail2ban on a mailserver.
Mailservers receive mail from other mail exchangers (MXes).
Fail2ban looking for traffic from an MX IP hitting a webserver you may be running on your MX (which is not a guarantee, and 401 and 403 are both HTTP Status Codes, which means a webserver is running) assumes one helluva lot about the remote MX, almost all of which are almost certainly invalid assumptions.
If you want to prevent spam based on sender, just use postgrey or an equivalent.