During the Civil War, he organized the 126th New York Infantry in August 1862 and became its first Colonel. His regiment was among the troops defending Harpers Ferry against Joseph B. Kershaw's Confederate brigade of MG Lafayette McLaws's division during the Maryland Campaign. He was severely wounded with a gunshot through his lower jaw in fighting on Maryland Heights during the Battle of Harper's Ferry. Sherrill was captured and later paroled. The wound never healed, but he temporarily rejoined his regiment at Union Mills, Virginia, in October 1862. After a furlough for further recuperation, he returned for duty on January 27, 1863.
He commanded the Third Brigade, Third Division, Second Army Corps, after Col. George L. Willard’s death on July 2, 1863, at the Battle of Gettysburg. Sherrill, in his first full day commanding the brigade, was positioned near Ziegler's Grove on Cemetery Ridge, when he was mortally wounded on July 3 by a musket shot in the bowels. Carried off the field by men of the 39th New York Infantry, he was taken to the XI Corps field hospital, where he died about 8:00 a.m. the next day. His body was sent by the regimental surgeon to Baltimore for embalming. LTC James M. Bull, 126th New York Infantry, who filed the brigade's report, giving a detailed account of the artillery barrage on July 3 but not much on the repulse of Pickett's Charge, places Sherrill's death at 4:00 PM.
He was buried in the Washington Street Cemetery in Geneva, New York. Nearly 10,000 people attended his funeral.