Thats pretty harsh on Yorito
But yea - seems like a pretty low blow towards Aono. I had though briefly about the fact that what Yorito was doing didn't make sense (telling Aono they are going to move, while getting closer to Matsuri at the same time), but this would certainly explain it. Though i don't know if this "lowering Aono's guard" makes that much sense either - its not like Matsuri was trying to sneak upon Aono while Yorito was delivering the "good news" or anything
And we still don't know whether it was Aono who wiped the memories clean ... it might as well be connected to what Matsuri was talking about illusions; that they gradually fade from people's memories.