There's a certain false dichotomy that feeds the problem and that is people separate it into something like this:
Smart people watch Smart anime
Stupid people watch Stupid anime
That's certainly an oversimplification but this is where the kind of elitism comes from. The problem is when a lot of the elitists are actually stupid people watching smart anime to make themselves feel smart, or so they think. Of course this kind of thing can only work in a self-affirming (aka mob thinking) fashion in which a community of self appointed elitists define quality. This isn't just true in anime, but almost every hobby really. To make this distinction, this group needs to raise itself above "popular opinion", but yes, in the end they create their own brand of popular opinions.
There's a small chance people from this group know what they're talking about, but in my anecdotal experience like any group of fools needs to demean others in order to give them credibility. I've received plenty of private messages from pretentious hygiene products telling me why I am totally wrong for bashing their deep work or liking something pedestrian. Of course, most can't really explain what they think being the simplest of sentences, so there's that.
Tl;dr Don't be a hipster.