admin.addPeer() takes a very long time to add the peer

Question

I am running a private network where nodes add and remove each other dynamically.

I have a problem where, the first time a node performs admin.addPeer() to a neighbor the peering is instant (as desired), but after that it can take 10 seconds or more before a connection is established (that is, once admin.removePeer() and admin.addPeer() are called).

If I wait long enough, then it becomes instant again. Could it be a sort of spam protection?

Cheers