Wild speculation time.
Now, time for some visual comparisons...
In my view, these three share noticeable facial structure characteristics, as though being close family members. Similar overall nose structure, and sunken eyes. Very sharp facial features in general.
So, here's a thought:
Kanba and Natsume are half-brother/half-sister. They share the same father, but have different mothers. Kanba's family was disowned from the Natsume clan because of his mother being an "outsider". This is why he hates the Natsume family. However, his biological connections to such a rich and powerful family nonetheless gives Kanba some sway with the criminal underworld, hence enabling him to gain money quickly.
By extension, Sho is also Natsume's half-brother, while Kanba and Sho are both half-brothers to Himari (same mother, different fathers).
By "the four of us reuniting", Natsume's father is referring to him and his three biological children (Natsume, Kanba, and Sho).
What do people think? It's pretty out there, but...