I'm not in an open adoption, but I guess the thing that leapt out to me is this...
If you toss out the "details" and are left with the question of "Will I regret this more if I DO visit or DON'T visit", how would you honestly answer it?
P.S. Not saying you shouldn't feel the way you do with the fear (it's a new situation for you and a change in your "agreement"), just saying at the root of it seems to be that you genuinely like her and you seem to want to do this.