I had to dig around to find the hint that I was pretty sure existed :-)
It's in the Design Rationale document, under Compression algorithm.
The wire protocol and the database both use a custom compression
algorithm to store data. The algorithm can best be described as
run-length-encoding zeroes and leaving other values as they are, with
the exception of a few special cases for common values like sha3('').
The relevant code is in the
go-ethereum/p2p directory on GitHub.
I can't immediately see how the Wire Protocol code calls into where I assume the custom compression is implemented (which is either
compression/rle/read_write.go?), and there are actually some calls in the RLP code to Snappy (Google's compression library).
Perhaps further investigation is required to be sure...