Bug&Bear...was it the tone with the statement? Because that definitely makes a difference.
As a mom of 4 kids ages 9,8,6&5, I get them all!
I can tell by the tone though who is being rude or those who are just taking an interest and making conversation. My kids are funny and very sociable so we often get comments. The "handful" comment...you bet. But it isn't that common these days I think to see many groups of 4 kids so close in age. Usually my response to that one is "Oh yes! A handful of love!" Or if they are not behaving well and that's why I get the comment, I might say with a smile "Ain't motherhood just grand sometimes?"
Sometimes I get rude ones...but mostly not. People are just naturally curious and I'll take the cute comments or the "your children behave so well!" anyday! Yes, sometimes the attention comes at a time I might not be in the mood for or etc., but as long as they are not being rude, I'm inclined to be positive.
Think the rudest comment I ever got was "Don't people believe in birth control anymore?" Now that one I did have a mean comeback for...and she was sorry for it. Had a great day after that one...

(yes, I do like to be snarky at times when it is warranted!!!)
And I think too, because I have gotten the above very rude and awful comment, I find most everything else to be of an inquiring state rather than an attempt to chastise me or my kids.