When a friend of mine was close to her due date with her fourth child, her then 3.5 year old son announced one day that this baby would probably would be brown-skinned since they already had three white kids. His logic being my family had one white skinned child and two brown skinned children so they should be due for a brown skinned child too. It was cute because he was so serious about it.
I think it is all pretty normal at that age. My kids all started noticing difference (not only skin color but weight, height, etc.) between ages 3 and 4.
The problem is that at age 5 I STILL can't get DS#2 to stop making comments at full volume and pointing!