How to verify access of child contracts?

Question

I am using a factory A to produce child contracts. Now I want to guarantee that only these child contracts have access to a different contract B.

Using the factory as a proxy is somehow a detour as I also would like to send stuff back from B to the children. I thought using the nonce of the factory A would be great but this is not accessible in Solidity. Is there any other way?

Thanks!