John was a younger son of Robert de Gray of Rotherfield Greys in Oxfordshire, and descended from the Norman knight, Anchetil de Greye. He entered Prince John's service by 1196 and was keeper of John's seal by 1198. After John's accession in 1199 he was rapidly promoted in the church, becmong Archdeacon of Cleveland in March of 1200 and Archdeacon of Gloucester before April of 1200 then quickly being elected bishop of Norwich in September of 1200. He was elected about 7 September and was consecrated on 24 September. He also served as John's secretary. In 1203 he went with Archbishop Hubert Walter of Canterbury on a diplomatic mission to King Philip II of France. He was also instrumental in securing the selection of his nephew as Lord Chancellor after Walter's death in 1205.
King John's attempt to force him into the primate's office of Archbishop of Canterbury in 1205 started the king's long quarrel with Pope Innocent III. After Hubert Walter's death in July of 1205 the selection of a successor was hindered by doubts about what the proper procedure should be. King John postponed a decision while delegations from both the bishops of England and the monks of the cathedral chapter went to Rome to seek guidance from the pope. However, while the delegations were in Rome, the monks of Canterbury decided to hold a secret election and elected their prior Reginald to be archbishop. Reginald was sent to Rome to join the delegation. When King John found out that the monks had elected someone without any regal input he forced the monks to elect John de Gray as archbishop. Some stories have the election of Reginald taking place before the sending of the first delegation to the curia. Another source, Gervase of Canterbury has the king telling the chapter they could choose their own nominee after six months, while the king secretly sent envoys to Rome to secure the election of de Gray.
De Gray was postulated to Canterbury on 11 December 1205 and the nomination was quashed by Pope Innocent III about 30 March 1206, along with Reginald's claim. The monks then elected, with Innocent's approval, Stephen Langton.
De Gray was a hard-working royal official, in finance, in justice, in action, using his position to enrich himself and his family. In 1209 he went to Ireland to govern it as governor. He adopted a forward policy, attempting to extend the English frontier northward and westward, and fought a number of campaigns on the River Shannon and in Fermanagh. But in 1212 he suffered a great defeat in Fircal in County Offaly. He assimilated the coinage of Ireland to that of England, and tried to effect a similar reform in Irish law.
During the interdict that innocent III placed on England during John's reign, de Gray stayed in the England and helped govern the kingdom, even after the king was excommunicated, along with Peter des Roches, Bishop of Winchester
De Gray was a good financier, and could always raise money: this probably explains the favour he enjoyed from King John. In 1213 he is found with 500 knights at the great muster at Barham Downs, when Philip Augustus was threatening to invade England. When John and the pope concluded the treaty where John gave England to the pope and received it back as a vassal, John de Gray was one of the witnesses to the treaty. After John's reconciliation with Innocent de Gray was one of those exempted from the general pardon, and was forced to go in person to Rome to obtain it. At Rome he so completely gained over Innocent that the pope sent him back with papal letters recommending his election to the bishopric of Durham in 1213; but he died at Saint-Jean-d'Angély in Poitou on his homeward journey on 18 October 1214. He was buried in Norwich Cathedral.
As bishop, he settled a long running dispute between the monks of his cathedral chapter and the bishops.