Something to keep in mind here: From what I've seen, most of the girls in harem anime aren't interested in
just sex. Most tend to want a serious romance with the male lead (personally, I'm glad about this, as romance anime would otherwise simply be plot-thin hentai by another name
).
There are exceptions of course, but they tend to only be a harem member or two, with most of the harem wanting a serious relationship.
It makes some sense for the male lead to not want to get involved with "willing girls" when this willingness carries with it the implied (if not clearly stated) desire for serious romance. In other words, getting involved sexually with such girls would carry with it at least implicit agreement of entering into a romance with her, and
that's a big step that you wouldn't want to make with just anyone. I think that most guys would want to at least get to know the girl, over a certain period of time, before they'd be willing to enter into a romance with her.