I did not find anything in the official documentation (and also in the geth source code) that suggests, that changing the default port will make other peers refuse to connect to some other peers.
The reason is probably due to many factors. Here, I report the ones I individuated:
- There are not enough peers, e.g. in Ropsten, as reported in this github issue
- The other nodes are using a reputation system, as reported in the documentation:
Node identity and reputation
The identity of a ÐΞVp2p node is a secp256k1 public key.
Nodes are free to store ratings for given IDs (how useful the node has been in the past) and give preference accordingly. Nodes may also track node IDs (and their provenance) in order to help determine potential man-in-the-middle attacks. Clients are free to mark down new nodes and use the node ID as a means of determining a node's reputation.
Maybe, your peers are not always online, and so the other peers find your peers not so useful and their reputation is not so high.