It is coldest just about the time of sunrise because this is the hour at which the atmosphere has been without the heat of the sun for the longest time. The lowest temperature of the day usually occurs then.
The atmosphere holds in the heat of the day for some time after the sun sets. The heat gradually dissipates over the hours of the evening. By just before sunrise, the heat is gone, and the atmosphere is at its coolest. This is also the time at which the dew or frost point is most apt to be reached, and dew or frost often appears on the ground around this time. It takes a little time before the sun's rays become strong enough to begin heating the atmosphere, allowing the ground to start absorbing the heat again.
Similarly, the hottest part of the day is generally in late afternoon, not at noon when the sun is at its highest point. Although the sun's rays are most direct from late morning until mid-afternoon, the Earth holds in the heat as the day progresses. By late afternoon, the heat of the day has reached a cumulative point, and the day's highest temperature generally occurs at that time.