Could one ever explain love at first sight? It's something that is very real, but logically speaking, it is hard to explain. Also, Subaru is a teenager with raging hormones, so that makes emotional impulses even more likely.
He met Emilia first. She was beautiful, and he pretty much fell in love at first sight. There really isn't much to reason or explain love at first sight. It just happens, and he is committed to her. If he changes to another person so soon (I don't even think he has been here a month yet), then he will seem whimsical with his feelings or even unfaithful even. That's also why Rem accepted that Emilia is top in his heart.