People can spend hours outside at below 0C temperatures as long as they’re wearing the right gear. Some people even like doing sports outside when it’s -20C.
But, you’re right that most of the time people only care to the 5 degree range. It’s a bit different when it’s close to the ideal room temperature. If you personally like it at 22C and the room is set to 20C you will probably feel cold after a while. If it’s 24C you’ll probably feel overly warm. But, except for something like measuring a fever, people almost never care about fractional degrees.