It's true he's biased against Tokiomi (understandably so with his current mental and physical state) which blinds him. However that particular scene his enemity with the man isn't everything that was involved. The anime
cut this:
Spoiler for novel:
Haggard, Kariya left out a dry laugh from his theory.
He did it. That haughty magus was finally splattered with mud by an ordinary man like Kariya, after always looking down to the likes of him. In his heart, Kariya sneered abundantly at Tokiomi and Zōken, thinking, did you see that.
I'm not a loser. Nobody will call me a failure or an insect anymore. I can fight against you. I'll have you know what fear is...
This is enough for tonight. There is no reason to continue a battle that will pain him now that his sworn enemy, Archer, has withdrawn. Just let the other Servants kill each other as they wish.
As he reached his conclusion, Kariya was the most panicked of them all when he saw Berserker took Saber as his new target and charge at her.
"Stop... Come back! Come back here, Berserker!"
He called out to him. Such a simple instruction should be enough to reach him, and yet the black knight didn't respond. Instead, the amount of prana consumed by Berserker's excitation stimulated all the Crest Worms at once, and pain shot again through Kariya's body.
"Berserkeeeeer! Stoop iit!!"
With the pain, Kariya's voice was more like a scream. He didn't even have the mental freedom to use a Command Seal anymore. Swarmed by a torrent of agony, Kariya could only focus entirely on keeping a hold of his receding consciousness.
Berserker went so mad that, in order to fight Saber, he sucked Kariya's prana out and he was forced to be unconscious in agony. So yeah... He was being smart too, letting the Servants fight themselves, considering he's maintaining a friggin' Berserker. That's the way to go unless you're a prana battery like Illya.