I think you can say like that.
The problem is that it is abnormal for a male to contract spirit.
And because the only one in the story was the *demon king* people are wary of him.
Afterall ,if you recall the *demon king* was the one defeat by Est last patner.
Also all the misunderstandings just give him a bad reputation
(you know *demon king of the night* ecc).