As well as being a talented musician, Berners was a skilled artist and writer. He was a ubiquitous presence in British literary circles of the 1920s and 1930s. He appears in many books and biographies of the period, notably portrayed as Lord Merlin in Nancy Mitford's The Pursuit of Love. He was a friend of the Mitford family and close to Diana Guinness.
Berners was notorious for his eccentricity, dyeing pigeons at his house in Faringdon in vibrant colors and at one point having a giraffe as a pet and tea companion. He was also subject throughout his life to periods of depression. These became more pronounced when Berners, who had lived in Rome during 1939-1945, found himself somewhat out of favour after his return to England. He bequeathed his estate to his companion Robert ('Boy') Heber Percy, who lived at Faringdon until his own death in 1987.
Berners obtained some notoriety for his roman-à-clef The Girls of Radcliff Hall, (punning on the name of the famous lesbian writer), in which he depicts himself and his circle of friends, such as Cecil Beaton and Oliver Messel, as members of a girls school. This frivolous satire, which was published and distributed privately, had a modish success in the 1930s. The original edition is rare; rumor has it that Beaton was responsible for gathering most of the already scarce copies of the book and destroying them. However, the book was reprinted in 2000.
His other novels, including Romance of a Nose, Count Omega and The Camel are a mixture of whimsy and gentle satire.
His epitaph on his gravestone reads: "Here lies Lord Berners / One of life's learners / Thanks be to the Lord / He never was bored"