If I in theory had the private key to a public contract on the blockchain. Would I be able to do anything with that key?
You won't be able to do more than what is coded in the smart contract, actually from a user point of view unless you are defining the smart contract as an owner of itself in theory you will be just a normal user.
However what you can do is, sign transactions and send them as if you were the smart contract? To be honest, though I think there is no way of getting a smart contract private key because the interaction between smart contracts is internal transactions that are not recorded in the ledger.
So, in theory, you can conduct transactions as if you were the smart contract.
In practice, smart contracts call each other through internal transactions so you can't do anything.