Why don't you tell the bmom the same thing you posted here: tell her you would like to see the biofamily as extended family, but you're not there yet, and when baby boy is older you will feel more comfortable with the request. But for now, you are more comfortable with her visiting him in your home.
I hope it works out, all the best.
-Manni
