What exactly is the difference between HDWalletProvider and HttpProvider, and what is the proper scenario for using each?
An HttpProvider is an intermediary object that allows to send commands to an ethereum client through an http connection.
For example there are also IPCProvider and WebsocketProvider that use pipes or websockets.
On the other side web3 commands can be grouped on different areas wallet, contracts, blocks, etc. Such functionality can be provided by separate intermediaries on top of the base provider.
For example HDWalletProvider will offer wallet services for an HD wallet. Underneath it will use HttpProvider to send other commands that are not wallet related.