> I assumed that slavery is just a fact of life in most settings.
This is a poor assumption. Slavery isn't a big thing at all in most settings, except for the purview of evil races. The typical medieval period most fantasy settings draw from doesn't have slavery as a major facet either, since serfs and peasants worked the land.
It's only later on when sailing, colonialism and trade begins where african slaves become an easy option, and before that you only had rome where slaves were those taken by war. Either way, both fall somewhat far out of the time period.
That's part of why I brought up dark sun, as that's a much earlier sort of bronze age setting where life is harsh and slavery can be expected.
In the typical D&D setting? No, slavery probably isn't allowed, and if it is, it'd likely be restricted to monstrous races captured in wars.
That aside, it's as you say. It's either going to eat up a bunch of his time taking care of a gaggle of followers, or it's going to be a long-term goal in which case there's no real point.
And of course, the former is going to be a mess, because having followers that you can't just leave unattended or trust is obnoxious. You can't leave your slaves outside with the horses and gear, since it'd be far to easy for them to flee while you're in a dungeon for hours.
It's just the sort of thing that's very out of place in a fantasy story except as a villain, and playing a villain in a typical party doesn't really work out.
If you really, really wanted to do this, the solution would be to make a setting where Slavery was more acceptable, and likely do a more evil campaign where everyone is doing more roguish and illicit activities as part of a grand crime organization. Your slave trading can be done in the same downtime as the ranger peddling drugs or whatever.