The first National Bank was so controversial because it was seen by many as a centralization of power in the national government. It also took power away from local banks and was seen as an effort to benefit businesses in the north and not agriculture in the south. It also brought up questions of constitutionality and the government's rights in relation to the rights of states and the people.
The biggest reason the first National Bank was opposed is because it was seen as a centralization of power in the national government. It was looked at as the national government monopolizing banking and controlling state interests. There were also problems, because the National Bank held the government's accounts but couldn't actually control state banks, which led to rivalries between them. Even more rivalries rose up when politicians began seeing the National Bank as a way to improve business interests in the north while neglecting the south, who traditionally supported state banks.
Another big problem was that by the early 1800s, the National Bank was primarily owned by foreign investors. This worried politicians and American investors alike.
Another big concern was the constitutionality of the bank. James Madison believed that Congress didn't have the power to make a bank, since it wasn't explicitly stated in the Constitution, and Thomas Jefferson argued that a national bank violated property laws. All of these factors combined led to the Bank's charter not being renewed in 1804, when the anti-bank Thomas Jefferson was in office. The main argument was the fact that the creation of a national bank wasn't stated in the Constitution.