From what I've gathered, there are a lot of "normal" people who do like the anime fan scene -- but they wear masks in public or go to a fair amount of effort not to be labeled. Watch those "street dance moments" where a bunch of people break into dance -- most of them have face coverings for anonymity. Anime tends to be watched solo by these folks or with thoroughly trusted peers.
I suspect there's a bit of practical frugality lurking in lack of approval. Fans tend to "waste" a lot of money on impractical collections (figures, etc). There's probably a number of people who have done master/doctoral theses on the sociology behind the whole thing
-- because people will give public reasons for shunning anime, sometimes not realizing the actual underlying discomfort they're trying to rationalize.