His official harem as of volume 6 only includes 4 people (well, 3... Ena apparently isn't official, and is an "underground lover").
There are others who have some slight crushes on him, but nothing serious.
In volume 7, the following jump on the boat, though they don't become part of the official harem:
Also, some think that Doni Salvatore is part of his harem, albeit not officially.