Walter White explained why this is on Breaking Bad.
It's because the part of your brain that registers taste and smell is situated right next to where your long term memory part is.
In essence, you are remembering the first time you registered that smell, and that was the weather and where you were at the time.
Or something like that...
You ever smell the air and it smells like the 4th grade scholastic book fair on a chilly tuesday in September of 2001