I want to backup my accounts onto a USB or other external device for safe-keeping. How can I do that?
To backup your wallet you will need to make a copy of the key file(s) located in your ethereum directory.
The file should look something like this:
I've suggested just saving the keystore contents, but you can of course backup the entire /ethereum directory; note, however, that this includes the chaindata files, and that's over 115 Gb, as of July 2018.
To import saved wallet files, simply copy them into the keystore directory of your system. Presale versions will end in the
*.json extension, but they will be recognized by the ethereum client.
PLEASE MAKE SURE TO ALSO BACKUP YOUR PASSWORDS
Each key file (as described in @Ethan's answer) is encrypted and only usable with the correct password.
Another safety tip: Before deleting any key files, after you have done the backup, please try the import/restore process on another system to make sure you can send a small amount of Ether. (To import, copy the backed up files to the keystore of the other system.) When you can spend that small amount of Ether, that will give you more confidence that your backup actually works.