Wood is a poor conductor of heat (as well as other forms of energy) because it is covalently bound as a compound. As a result, it does not have the free electrons that scatter about to conduct different forms of energy like metals and other strong conductors do.
In addition to a lack of free electrons, wood also has lots of air pockets inside it, and it even has some starches and proteins. These three properties make it hold onto heat rather than release it. Wood is quite porous, and the pores tend to soak up the waves of heat as well; this is why it is possible to put a wooden spoon into a pot of boiling water and touch the handle after a few minutes without burning one's hand, but why doing the same thing with a metal spoon leads to red, sore fingers.
In contrast, metals share electrons on the atomic level, and passing electrons allows energy (of which heat is a form) to move much more freely. One final difference involves the surfaces involved. Even sanded wood is rougher than metal, and rougher surfaces transfer less heat, because their edges do not have as much contact with the recipient as smooth surfaces, such as metal.