well you can have as few as 4 to complete the quest. but this will make training times be very very long. you wont grow your might as fast as say someone that has 10+. the more you have the shorter the training time is, it doesnt matter so much as to their level, but has more to do with how many you have. the most you can have and still have all the important buildings in your town, is 14, but then you are giving up the tavern and the market. you will need atleast 7 cottages for employment and taxes, and to have a large enough idle population to train from.
so the question how many do you need, 4 to finish the quest.