There are two contracts, the Manager and the Token, with the Manager being the actual world-facing address that people interact with, but all it does is delegate all calls back to the Token where the logic is, but the storage is in the manager, so if a bug is found the owner can swap the logic out and be fine. Also, though, the owner could swap that contract out with one that does nothing, so it would screw everyone over. Hence the trust. What's to stop me from changing the Token address on the manager to a dead end, or something that gives their tokens to me? Has anyone implemented this, or is there a project with a pattern like this implemented already, and how did it go? What do people think?
The trust issue is why contracts aren't upgradeable by default: For them to be upgradeable implies there is some person or process that can upgrade it, and people using your contract have to trust that person or process. If you don't want that, you should instead deploy a new contract and let users decide whether or not to use it.
The extent to which this trust is acceptable will depend on your situation. If the purpose of the contract is to control your funds on your own behalf, there should be no trust problem, because you already trust yourself. If it is supposed to manage a bet between you and one other person, it may make sense to require both parties to agree before it can be upgraded.
However, given the practical reality that contracts sometimes have bugs, it is often a good trade-off to allow contracts used by many different parties to be upgraded under some conditions. One strategy is to have a jury of trusted parties who all have to sign off on an upgrade. Another is to make the contract upgradeable only if it detects an error condition, like finding a mismatch between the amount of money it thinks it should hold and the amount of money it actually holds.