I wrote this on the gitter.im channel, so here it is again anyway. Caveat is I have not looked at recent geth code for this, so there may be subtle discrepancies with actuality.
A key paragraph is this "Message IDs are assumed to be compact from ID 0x10 onwards (0x00-0x10 is reserved for ÐΞVp2p messages) and given to each shared (equal-version, equal name) sub-protocol in alphabetic order. Sub-protocols that are not shared are ignored. If multiple versions are shared of the same (equal name) sub-protocol, the numerically highest wins, others are ignored."
Basically the DevP2P messages are all identified by their type, which is the first integer.
The first 0x0F(?) are reserved for DevP2P messages, specified in the DevP2P-Wire-Protocol document.
The next message ids appear to depend on what protocols are commonly supported by the pair of peers. By commonly, I mean shared.
The sub protocols shared by the peers are established in the Hello message (0x00) And the key phrase in the docs related to that is "cap Specifies a peer capability name as a length-3 ASCII string. Current supported capabilities are eth, shh."
So let's say both peers after the Hello message only support "eth", then according to the Ethereum wire protocol doc, the next message identifier is offset + 0x00 for the Ethereum "Status" message (making it 0x10)
To your question, "Suppose, I want to introduce a new sub-protocol", I think the above answers that. You would need to make a new 3 letter identifier (like, ETH, LES, SHH) and advertise that as a capability in your DevP2P client. Clients would handshake and in the Hello message establish that they have your ??? protocol in common, and that would become part of the message id space.
So, to summarise: a "sub protocol" is a message id space identified by a 3 letter name. Peers handshake and in the Hello messages specify which sub protocols they support. For that p2p relationship, the common protocols are arranged in alphabetical order, and when messages are sent, their identifier is the offset relative to the start of the message id space given to the alphabetically ordered message id space.