[Qilin]

Just to add, there's also a widespread preconception regarding the harem genre as "wish fulfillment" in fiction. After all, it is a shared fantasy by a lot of men to be loved and surrounded by women. These sorts of stories cater to that desire. Main characters for these sorts of shows are made as bland as possible to make it easier for viewers to insert themselves in the place of the main character.

This is quite the cynical view, I admit, but I'm convinced that there is some truth to this notion. Whatever the case, it's this popular notion that adds a sort of stigma to the harem genre. The very idea of "wish fulfillment" in fiction is then perceived to be negative, and therefore derogatory. And by extension, this negative association carries over to the types of main characters you'd find in these sorts of works: the bland, featureless, and uninteresting ones.

If you have doubts, let's think back to how the label "Mary Sue" came to become negative to begin with. "Mary Sue" today is used to describe characters that are unreasonably good at everything, who all other characters fawn over for no apparent reason. However, the label was used to refer to author self-inserts in fanfiction. All the talent and attention is just a way for the author to live out his/her fantasies. Just from this, there's definitely something about inserting one's self into a work that is strongly reviled and condemned by society.