Dear John,
For one thing, context is crucial. We live in a world of steel, glass and car exhaust, and so it is surprising and pleasing to encounter the reality of bodies -- the smell of sunlight on someone's skin, or the moment in school when the girl several rows front of you lets down her hair several hours into the morning, still slightly damp, and a very light scent of her shampoo wafts back -- even for us other girls this is just purely pleasant (provided she's not one of the annoying girls who lets down her hair while wet and makes a mess -- but I digress).
Sometimes the smell of a man's sweat, if it's fresh with the pulse of life and work (rather than tired, stale) has this same effect.
So part of the answer to your question, I think, is that skin is something we smell (not to mention touch), and this can be strong for us.
I wonder, if we lived in a world with little clothing, if it would have the same effect? I've certainly been in crowds where I was not excited about the surrounding press of bodies. I wouldn't say "any skin we see" for that reason.