If quidditch was a warm-weather sport, we'd have too many bored athletes roaming the halls. That's worse than a bit of cold.

And I'm sure the teachers would beg to differ with you about exams. This is a school, as you've hopefully noticed.