He was initially elected to New York's 21st congressional district (since redistricted) in the United States House of Representatives during the heavily Republican year of 1946. He was a member of the freshman class along with John F. Kennedy of Massachusetts and Richard M. Nixon of California. He served from 1947 to 1954, then resigned his seat after his election as the New York Attorney General.
In 1956, he defeated Mayor of New York City Robert F. Wagner, Jr., in a U.S. Senate race to succeed the retiring incumbent Democratic Senator Herbert Lehman. Like Lehman, Javits was for a time the only Jew in the U.S. Senate.
Javits was generally considered a liberal Republican, and was supportive of labor unions and movements for civil rights. In 1964, Javits refused to support his party's presidential nominee, his conservative colleague, Barry M. Goldwater of Arizona even though Goldwater had said in 1962 that were he a New York voter, he would vote to reelect Javits.
Senator Javits sponsored (1) the first African-American Senate page in 1965 and (2) the first female page in 1971. His background, coupled with his liberal stands, enabled him to win the votes of many historically Democratic voters. He was highly successful in all elections in which he was a candidate from 1946 to 1974.
Javits played a major role in legislation protecting pensioners, as well as in the passage of the War Powers Act; he led the effort to get the Javits-Wagner-O'Day Act passed. He reached the position of Ranking Minority Member on the Committee on Foreign Relations while accruing greater seniority than any New York Senator before or since (as of 2007). He was also one of the main forces behind the 1965 Immigration and Nationality Act that by removing immigration quota that favored Western European nations helped to make the U.S. a truly diverse and multicultural country.
Following the primary defeat, Javits ran as the Liberal Party candidate in the general election, having split the Democratic base vote with United States Representative Elizabeth Holtzman of Brooklyn and giving D'Amato a plurality victory.
Among those who attended the funeral were Governor Mario Cuomo, Mayor Ed Koch, former President Richard Nixon, Attorney General Edwin Meese, former Secretary of State Henry Kissinger, Senator D'Amato, John Cardinal O'Connor, former Mayor Lindsay, former Governor Hugh Carey of New York, and former State Attorney General Louis Lefkowitz.
Also there were U.S. Representative Bella Abzug of Manhattan; then Senators Nancy Kassebaum Baker of Kansas, Bill Bradley of New Jersey, Lowell Weicker of Connecticut, and Gary Hart of Colorado; David Rockefeller, the banker; Arthur Ochs Sulzberger, publisher of The New York Times; Victor Gotbaum, the labor leader; Kurt Vonnegut, the writer, and Douglas Fairbanks, Jr., the actor.
New York's Javits Center is named in his honor, as is a playground at the southwestern edge of Fort Tryon Park. The Jacob K. Javits Federal Building at 26 Federal Plaza in lower Manhattan's Civic Center district, as well as a lecture hall on the campus of the State University of New York at Stony Brook on Long Island, are also named after him.