Chimney downdrafts, where air or wind comes down the chimney, have several causes. If the fireplace is located in an area with lower air pressure than that outside, it may have a permanent downdraft due to the nature of fluid dynamics. High winds can also push air downwards into chimneys.
A remedy for a chimney in which the fireplace has lower air pressure than that outside is to equalize the air pressure by opening windows or doors as necessary. For locations where high wind speeds are common, a properly made chimney cap can prevent gusts of wind from blowing downwards.