I think how Homura acts towards Madoka is still believable regardless of sexuality, since I know at least one show where somebody does basically the same thing for their sister, although it's not an anime (but still Japanese). If you watch Kamen Rider Ryuki, the way Shiro Kanzaki is motivated in protecting his sister Yui is very similar to how Homura is towards Madoka, and I think the way that show played out was that they had abusive/neglectful parents, and the two siblings only ever hung out with each other during their childhood.
This is relevant since Madoka in general was compared quite a bit with Ryuki.