Poor baby! He is at high risk of attachment disorder, so that may be what is going on. It's not you! Check out Nancy Thomas's amazing book, When Love is Not Enough. It has good, concrete suggestions about how to encourage babies (and older kids) to attach. Attach China also has a very good site:
http://www.attach-china.org/. Good luck! I never had a baby, my kids are all older. I'm sure it's tough! The good news is that it's great to start the healing when they're babies, they can really heal so much easier. I wish my kids had a caring foster parent like you when they were babies!