I used to have "store toys" in my diaper bag or purse. I had a small selection of things they only got when we went to the store or general public. So that might help if you have to trade an item with him that he wants.
I think in your situation today, I'd have given him a chance to cool off in the car. Stand outside the car if you have to! (I know not everyone can turn off their ears like me!
)
Or you could just turn on the music in the car. Usually a calmer start will ensure a better shopping trip.
I've sat in the middle of a grocery aisle with one or more of my kids in my lap waiting out the tantrum. I've also left the store. And you bet I've also given them a cracker or anything I could grab to calm them down. Not to reward them, but ME. Obviously as you said, you don't make a habit of this one, but sometimes you have to do what you have to do.
Try smiling and singing silly songs to him too. Something different that might make him go "huh?" Blow bubbles at him (must have for diaper bag, imo!!!) for distraction too.
And then there is the simply ignore it times and deal with the stares. Most times I think you'll get the sympathetic looks from moms who go /have gone through the exact same thing. It happens to everyone.