Knox was born in the Pittsburgh suburb of Brownsville, Pennsylvania, and graduated from Mount Union College in 1872. He was admitted to the bar in 1875 and practiced in Pittsburgh, Pennsylvania. From 1876-1877 he was Assistant United States Attorney for the Western District of Pennsylvania and became President of the Pennsylvania Bar Association in 1897.
Knox was a leading Pittsburgh attorney in partnership with James Hay Reed, their firm being Knox and Reed (now Reed Smith LLP). Knox was also a member of the South Fork Fishing and Hunting Club, whose earthen dam failed in May 1889, causing the Johnstown Flood. The Club had been formed by Henry Clay Frick, at the suggestion of Benjamin Ruff. Situated in the mountains above Johnstown, Pennsylvania, the South Fork Fishing and Hunting Club's charter members were: Benjamin Ruff; T. H. Sweat; Charles J. Clarke; Thomas Clark; Walter F. Fundenberg; Howard Hartley; Henry C. Yeager; J. B. White; Henry Clay Frick; E. A. Myers; C. C. Hussey; D. R. Ewer; C. A. Carpenter; W. L. Dunn; W. L. McClintock; A. V. Holmes.
The Club grew to include sixty-odd members, all of them leading business tycoons of Western Pennsylvania and included among their number Andrew Mellon, Philander Knox and James Hay Reed, as well as Frick's business partner Andrew Carnegie. The Club members created what was at that time the world's largest earthen dam behind which formed a private lake called Lake Conemaugh. Less than 20 miles downstream from the dam sat the city of Johnstown, and not incidentally, Carnegie Steel's chief competitor, the Cambria Iron and Steel Company, which at that time boasted the world's largest annual steel production.
Poor maintenance, unusually high snowmelt and heavy spring rains combined to cause the dam to give way on May 31, 1889 resulting in the Johnstown Flood. When word of the dam's failure was telegraphed to Pittsburgh, Frick and other members of the South Fork Fishing and Hunting Club gathered to form the Pittsburgh Relief Committee for tangible assistance to the flood victims as well as determining to never speak publicly about the club or the flood. This strategy was a success, and Knox and Reed were able to fend off all lawsuits that would have placed blame upon the Club’s members. Although Cambria Iron and Steel's facilities were heavily damaged by the flood, they returned to full production within a year and a half. However, by that time, Carnegie's steel production had outstripped Cambria's.
Knox was a member of the Duquesne Club. Along with fellow South Fork Fishing and Hunting Club member Jesse H. Lippencott, Knox served as a director of the Fifth National Bank of Pittsburgh. Henry Clay Frick, Andrew Mellon and Philander Knox were directors of the Pittsburgh National Bank of Commerce.
Knox was a lifelong friend of President William McKinley. His nickname was “Sleepy Phil” which is said to have been a) because he dozed off during board meetings or b) because Knox was cross-eyed making it difficult for his two eyes to track together.
Knox married Lillie Smith, the daughter of Andrew Smith of the firm Smith, Sutton and Co.
As counsel for the Carnegie Steel Company, he took a prominent part in organizing the United States Steel Corporation in 1901. He served as Attorney General in the cabinets of Presidents McKinley and Theodore Roosevelt from 1901 to 1904. While serving Roosevelt, Knox worked hard with the concept of Dollar Diplomacy. In June 1904, he was appointed by Governor Samuel W. Pennypacker of Pennsylvania to fill the unexpired term of Matthew S. Quay in the United States Senate; in 1905 he was re-elected to the Senate for the full term (to 1909). After an unsuccessful bid for the Republican Presidential nomination in 1908, he served as Secretary of State in President Taft's cabinet from March 6, 1909 until March 5, 1913.
As Secretary of State, Knox reorganized the Department on a divisional basis, extended the merit system to the Diplomatic Service up to the grade of chief of mission, pursued a policy of encouraging and protecting American investments abroad, and accomplished the settlement of the Bering Sea controversy and the North Atlantic fisheries controversy.
Following his term of office, he resumed the practice of law in Pittsburgh. He was again elected to the Senate from Pennsylvania and served from 1917 to 1921. In April 1921 he introduced a Senate resolution to bring a formal end to American involvement in World War I. It was combined with a similar House resolution to create the Knox-Porter Resolution, signed by President Warren G. Harding on July 21. Knox died in Washington, DC later that year.
He is well-known for famous quote to Roosevelt: "Mister President, do not let so great an achievement suffer from any taint of legality," made in regards to the construction of the Panama Canal. A slightly rephrased version of this quote was spoken by Brian Keith as Roosevelt in the film The Wind and the Lion.