Venus is hotter than Mercury because of its thick atmosphere. According to NASA, Venus' atmosphere is 96 percent carbon dioxide, which holds in heat. Mercury has a very thin atmosphere that does not keep heat on the planet.
The thick layer of carbon dioxide that surrounds Venus acts as a greenhouse. The gases let the heat from the sun in, but do not allow it to leave. This is the same process that happens on Earth with the increased amount of greenhouse gases, though it is much more severe on Venus because the atmosphere of Venus is around 90 times as thick. According to Discovery, carbon dioxide absorbs the heat that Venus tries to give off at night and keeps it on the planet, increasing the surface temperature to around 900 degrees Fahrenheit. The carbon dioxide also causes severe winds and storms on the surface of the planet, making it even more inhospitable.
Mercury has a thin atmosphere with a variety of gases. Because of this, the atmosphere does not have a severe greenhouse effect that acts as a blanket over the planet like Venus. Instead, the heat given off by the Sun is more easily siphoned off during the night.