TBH I find it hard to believe that anyone would ever think Goku was voiced by a man at all.
I dunno, most of the time female voiced males are very easy to notice, even if their voice suits the character or even if they sound manly. Very few of them actually genuinely sound male, like Yumiko Kobayashi who has the eternal voice of a little boy IRL.
Current male seiyuu play just as much, if not more young male roles than they do older males and they still outnumber female seiyuu playing male characters. It's only when the character's voice is meant to sound younger and younger that females have the upper hand over male seiyuu who are typically in their twenties or older. So it's not surprising that they get more roles than males because the option to play males roles is there as opposed to men who typically don't have the option of actually playing girls other than as a joke.