You wouldn't worry what people thought of you if you knew how seldom they did.
Everyone is concerned with #1, themselves. They don't care what you're wearing, as much as they care about what they're wearing. They aren't so concerned about what they think of you as they are as about what you think of them.
"Enough about me, let's talk about what you think of me" is a fundamental truth of human interation.
So when you're at a party, and you think everyone is noticing you, they're not, they're hoping that you'll be noticing them.