I do not have a full answer but I will share what I found out. I used Browser Solidity to compile contracts, and used Etherscan's disassembler to help me out with the result.
When I compiled contract A {} in Browser Solidity, I looked at "Assembly" (you need to "Toggle Details"). It shows:
.code
PUSH 60 contract A {\n}
// Some Stuff...
JUMPI contract A {\n}
tag 1 contract A {\n}
JUMPDEST contract A {\n}
// Unpack contract for deployment...
RETURN contract A {\n}
.data
0:
// Now we have the deployed contract here
.code
PUSH 60 contract A {\n}
PUSH 40 contract A {\n}
MSTORE contract A {\n}
tag 1 contract A {\n}
JUMPDEST contract A {\n}
PUSH [ErrorTag] contract A {\n}
JUMP contract A {\n}
.data
// Oh nothing more to show here, but that's where it gets interesting for us.
Then I took the content of "Runtime Bytecode" and pasted it in the disassembler.
The lines:
[17] PUSH6 0x627a7a723058 [18] SHA3 // Its opcode is 0x20
Correspond to the xxxxx5820 that you identified. And guess what:
> web3.toUtf8("0x627a7a723058")
"bzzr0X"
Then what comes right after that, in my case 81ef35e9ce2010474897d82da20c73d954e24c1e93fceaca1da5d1ed75650a26 so 32 bytes which, if you go back to Solidity, are the same ones as under "Metadata location".
So my understanding is that the code of the contract has not changed; only its Swarm address has. And contract verifiers need to abstract that part away.
Edit: I just found this Encoding of the Metadata Hash in the Bytecode