Since 2004, an application for exemption from Social Security and Medicare taxes is required to bear the applicant's Social Security number. The Amish do not believe in insurance, which these programs are. Not all Amish have a Social Security number, but many obtain one to be eligible for the exemption.
Amish children generally don't apply for a Social Security number until they officially join the church as young adults. At that time, they file Form IRS-4029 for the exemption. The Amish do not need Social Security numbers to file their income tax returns but find that a Social Security number is necessary for many other formalities, such as opening bank accounts and getting a hunting license.