The fact that you are not setting the data field implies that you are interested only in sending ether (rather than in executing a contract function).
If the destination address is of an externally-owned account, then you only need 21000.
If the destination address is of a smart-contract account, then you need slightly more than 21000.
However, even when ...
This seem to have been entirely a problem with Infura. I suspect it is in regard to their rate limits. Nevertheless we have now set up our own ethereum node using geth and are able to isssue as many transactions as we want pretty much problem free!
So I found the issue. I was testing with the BAT token (0xda5b056cfb861282b4b59d29c9b395bcc238d29b), but at this point in time the ETH/BAT trading pair had not been created yet.
I should have checked the pair, and liquidity of the pair before trying to make a swap.
My code works with the DAI token (0xc7ad46e0b8a400bb3c915120d284aafba8fc4735).