Polarizing characters only exist if there is more than one opinion.
I don't have any problems with polarizing characters, because I ignore all other input on the matter until I'm finished watching a series and try to avoid spoilers. And by that time, I have already finished my opinion of all the characters by myself.
If I then find out afterwards, a character was polarizing the internet community all along, it's simple:
One side is right (my side) and the others are wrong.
Sounds smug, but really what can be a better way of forming a solid opinion than actually watching the anime?
I did that, they did that. If we all watched exactly the same anime and then still came to different conclusions, no amount of internet discussions will make a difference.