Castles were places where people lived and worked. As well as knights and soldiers, there were people like blacksmiths, grooms, masons, cooks, armourers, bowyers, fletchers, carpenters and leather workers. Sometimes their families lived in the castle. Sometimes they lived in towns or villages nearby.
In times of danger, everyone would come into the castle for safety, bringing their animals with them if they could. It might be very crowded.