The main difference between sand and dirt is that dirt contains clay. Clay contains thousands of electrically charged particles that stick together thanks to water. Sand has been washed of clay. The remaining items in dirt are grains of sand ranging from 64 microns to 2 millimeters in diameter.
Clay is the element that makes dirt appear "dirty." It sticks to fabrics and absorbs into surfaces that allow for it, such as skin. Removing dirt from fabrics requires an electrical charge, typically provided by detergent. Sand does not stick to anything, including other particles of sand, because it consists of crystals that do not stick to anything.