The driftwood flit switch is earned after completing the quests "An Island Getaway," "Pest Control" and "Giftwood for Driftwood." After finishing the quests, the driftwood flit switch is obtained by revisiting the driftwood after a few days. Look for a wooden plank the player can walk up.
According to the Fable Strategy Guide, the switch is initially located on a pedestal on the far right of the town. It can also be accessed during the first stage of development. Hit it with weapons or magic as applicable. Follow it around the island until it comes to rest on the pedestal outside of the cave. Now is a good opportunity to stop and finish quests, and then sleep or rest while Driftwood enters the second stage of development.
Climb the new wooden ramp that appears during the second stage, and attack the switch with a melee weapon. Continue to follow the switch, and use magic or melee as applicable. Finally, the switch comes to rest on a third pedestal, which is not accessible until the third stage of development. After this is complete, continue to follow the switch until it opens the cullis gate, which allows the player access to the gold key.