Andrew Jackson's most significant failure as president was to allow the state of Georgia to evict the Cherokee Indians from their indigenous lands. His economic decisions contributed heavily to the Panic of 1837, and his practice of giving cronies political positions introduced the "spoils system" to American politics.
Andrew Jackson signed the Indian Removal Act into law to help the state of Georgia make the Cherokee nation move to Indian reservations in the American West. The Supreme Court, in the 1832 case Worcester v. Georgia, ruled that no entity could force the Cherokees to move. However, the state of Georgia violated the Supreme Court order, and President Jackson did not intervene. Even though their forced-walk west, known as the Trail of Tears, happened after Jackson left office, his intervention could have stopped the deaths of about 4,000 Cherokee.
Jackson vetoed the charter for the Second Bank of the United States, taking the money out and putting it in the coffers of state banks. These banks lent money without much due diligence, leading to inflation. Jackson's response was to require land purchases to be made in silver or gold, which led to the Panic of 1837.
Jackson also believed in giving political jobs in the federal civil service to those who supported him. While many presidents after him have also done that, he was the first to use the "spoils system" to turn the civil service into a way to reward friends and supporters.