Let's say that I try to use Ethereum with a modified version of the client which does not follow all the rules of the protocol. What stops my client from being able to damage the Ethereum network?
If your (modified) client sends an invalid message to an Ethereum client, the message should be ignored. Correct and robust implementations of Ethereum clients is what keeps the network consistent and protects it from misbehaving clients. However, it is possible that you may send a message that is handled incorrectly by an Ethereum client and if you discover such cases, then you can apply for a monetary reward at bounty.ethereum.org.