In episode 24 it will be revealed that Handkerchiefman is Asa's father. That is why there was no defense force for Asa in the anime; he was trying to keep Rin away from the other girls so that he would focus on Asa. Primula was a wildcard that he did not anticipate, so there was no time to organize a force for her. She is also the reason Handkerchiefman has not been seen lately - who do you think she used to practice the levitation and freeze spells?
"Kaede. He broke. Kaede. Where is the broom?"