The village is split by the county border, and constitutes two civil parishes, one called Richard's Castle (Hereford), the other called Richard's Castle (Shropshire).
Richard Fitz Scrob (or Fitz Scrope) was a Norman knight granted lands by the Saxon King Edward the Confessor before the Norman Conquest, in Herefordshire, Worcestershire and Shropshire as recorded in the Domesday Book. He built Richard's Castle before 1051. The castle was a motte and bailey style construction, one of very few castles of this type built before the Norman conquest. Most were built after the conquest. Richard was last mentioned in 1067. His castle passed to his son, Osbern Fitz Richard, who married Nesta, the daughter of King Gruffydd ap Llywelyn of Wales.
Osbern died around 1137 and was succeeded by his grandson, Osbern Fitz Hugh, who died in 1187. Richard's Castle then passed to his brother-in-law, Hugh de Say, who died in 1190, leaving the barony to his son, another Hugh Say. In 1196 this Hugh fought at the battle at New Radnor and was probably killed there, his castles eventually passing to Robert Mortimer of Attleborough. In 1264 his son, Hugh Mortimer, was forced to surrender himself and Richard's Castle to Simon de Montfort, 6th Earl of Leicester. His grandson, the last Hugh Mortimer of Richard's Castle, was poisoned to death by his wife in 1304. The castle then passed to the Talbots. On December 3, 1329, Joan late the wife of Richard Thalebot, had noted in the Patent Rolls that she planned to leave Richard's Castle to John de Wotton, chaplain, and William Balle of Underlith, in fee simple. The Talbots were still living there in the late 14th century. By the 16th century it was in ruins.