Enough to house enough villagers to operate all your resource fields and also for your army. Look at the population figure in the supply tab in the bottom right hand area of the frame. Keep the 'idle population' figure positive.
Also, the larger the population you have, the larger your income, as they will all be paying you tax. Unless, of course, you've set your tax rate to zero, which is quite a good way of going about things.