There are several ways to stop it, but most of the ways take time to work. The reason why your population would drop, is because the happiness is low. For example, if my happiness is usually 100 and it drops to 80, i will lose population because my citizens are unhappy. So when you bring your happiness up ur population will go up, when it is at a constant happiness your population will stay constant, and when its lower than average, you population will decrease. So to stop it, you can either bring your tax extremely low or use items that you can buy using gems, such as mass hypnosis, to bring your happiness up. Another way to stop the downward trend is to build or upgrade cottages. By doing so, u can house more people.