Born in Dorchester County, Maryland, Frazier entered the Navy as a seaman in 1802. Serving in the Mediterranean Sea during the war with Tripoli, he took part in Stephen Decatur's immortal foray into Tripoli Harbor to destroy the captured Philadelphia, and was again with Decatur in the attack on Tripoli 3 August 1804. He was severely wounded in the gunboat attack, and is believed by many to have received these wounds while protecting Decatur with his own body. It is not clear whether he or Reuben James was the brave seaman who thus saved his commander's life. He died in New York City about 22 April 1833.
USS Frazier (DD-607) was named for him.