You need to invert your thinking which is admittedly hard to do. There is no "token" per se, and no way to account for the token outside of the token contract itself. It will never make sense when thinking about it as digital artifact, like a file, that users actually possess. What they have is a clear privilege (can spend) an in interference-resistant system.
The token contract keeps a ledger which is similar to a spreadsheet, except this spreadsheet is, generally speaking, the same to all observers. The contract enforces rules, chief among which is a rule that adjustments can be made only by instructions signed by "owners".
Imagine a spreadsheet with a row for every possible ethereum address, and a number representing an account balance denominated in whatever currency the contract defines. The rule is simple. Adjustments to account balances only happen by instructions signed by address corresponding to that row.
Thus, the only way to "spend" a token is to sign a transaction and send it to the contract itself; for example, an instruction to send 10 coins from Alice to Bob, signed by Alice. When this occurs, Bob's balance increases and Alice's balance decreases. It's self-evident to all observers that Alice signed a transaction, those are the rules, and the contract did the math.
Indeed, it's the strength of the enforcement of such rules that gives Alice's token's any worth. Considering the strength of the rule, all observers can see that Alice did indeed transfer some tokens to Bob, and now Bob is the only entity that can possibly spend them.
Hope it helps.