He was the eldest son of Richard Woodville, 1st Earl Rivers and Jacquetta of Luxembourg. Like his father, he was originally a Lancastrian, fighting on that side at the Battle of Towton, but later became a Yorkist. He succeeded his father in 1469.
Rivers became very influential at the royal court after his sister Elizabeth married Edward IV. He joined the king in his temporary exile in 1470, and returned with him the next year, where he was wounded at the Battle of Barnet.
He was married to Elizabeth de Scales, Baroness Scales in her own right, daughter of Thomas de Scales, 7th Baron Scales, and widow of Henry Bourchier, younger son of Henry Bourchier, 1st Earl of Essex.
In 1473 King Edward IV appointed Rivers Governor of the Prince of Wales' household, and Rivers went with the prince to Ludlow Castle. His duties included the administration of justice throughout the principality. When the king died in 1483, he accompanied the Prince, now King Edward V, on the way back to London. However, they were waylaid by the Duke of Gloucester, who imprisoned the Earl and then had him beheaded at Pontefract Castle on 25 June 1483 as part of his path toward becoming king (as Richard III).
After his wife's death in 1473, Anthony was summoned to Parliament in her right as Baron Scales. On his death in 1483 the peerage fell into abeyance.
Rivers was evidently quite learned, and no doubt had learned excellent French from his mother. He had met with the earliest English printer William Caxton when in exile in Bruges, and there in 1475-6 Caxton published Cordyale, or Four last thinges, Rivers' English translation from the French of Jean Miélot of Les quattres choses derrenieres, itself a translation of the Cordiale quattuor novissimorum. After both of them had returned to England, one of the first, if not the first, books printed in England was Rivers' translation from French of the Dictes and Sayings of the Philosophers, printed by Caxton at Westminster in 1477.. Lambeth Palace Library has an manuscript illustration showing Rivers presenting a copy of this book to Edward IV.