Although they were not needed inside the castle, the horses were very important. With them, the knights could ride out to help their friends or attack their enemies.
There were never many knights in a castle, but they were the people who led its defence.