Let's consider storage first. Storage is extremely expensive. It costs 20000 gas to set a storage location from zero to a value, and 5000 gas to change its value. (You do get some gas back for resetting a location to 0.) The reason is that a contract's storage values are stored on the blockchain forever which has a real-world cost. It's also 200 gas to read a word, compared with 3 for memory.
So for storage, use it only when you really must.
When must you use storage? For values that have to be persisted between different contract calls. For example the balances of accounts for an ERC20 token. Storage updates are state updates of your contract and are not really for temporary data. Storage of one contract can also be read by another contract, or by querying the blockchain without running the contract.
You can think of it like hard disk/SSD, if you like. Storage retains its value when the computer's not running.
Memory is relatively much cheaper than storage. It costs 3 gas to read or write a word, plus some gas if you are expanding memory. For a few KB it's very cheap indeed, but the cost goes up quadratically the more you use: a megabyte of memory will cost a couple of million gas.
However, the memory is only accessible during contract execution; once execution is finished, its contents are discarded.
When should you use memory? Pretty much for everything non-permanent. It's the general workhorse.
You can think of it like a computer's RAM. It vanishes when the computer is not running.
The costs of storing data on the stack are similar to those for memory. Sometimes there will be less overhead from having to push memory addresses to the stack; sometimes there will be more overhead as the compiler has to shuffle things around on the stack. The stack has a maximum capacity of 1024 items, but only the top 16 are easily accessible. If you run out of stack then contract execution will fail. The compiler generally uses it for intermediate values in computations and other scratch quantities.
On the whole, I'd leave the stack to the compiler to work out.
The stack also does not persist after contract termination.
All the info above is buried in the Yellow Paper - I've read it so you don't have to :-)