The air temperature is just one factor when considering how warm it is. As a general rule, the higher the wind speed, the colder it will feel and the faster your body will cool down. This wind chill can be very dangerous on cold days. An outer shell, such as a well-fitting, quality raincoat and over-pants provides good protection from the wind.
When resting on cooler windy days find shelter from the wind. If there is a nearby hut us that, sit behind a rock, among dense vegetation, or at least behind you pack. If someone is unwell or you need to rest for a longer time consider pitching your tent to protect you from the wind.

The following table will give you a sense of how quickly wind chill affects the apparent temperature and the second table shows what that can mean for your body.