Coming to professional life after WWII, Broyard was reluctant to discuss his history publicly. Because of this, he was sometimes accused of being a black man "passing" as white by some who criticized that he did not openly support African-American causes during the Civil Rights Movement or publicly identify himself as black. He had grown up in a Creole culture, however, that had different characteristics than identifying as black in New York. Broyard did discuss his ethnicity with a variety of friends, who were well aware of it. The fact that he was part-black was well-known in the literary community of New York from the early 1950s, but it was an environment in which people from a variety of backgrounds remade themselves as members of an artistic milieu.
In 1961, Broyard married Alexandra (Sandy) Nelson, a white woman of Norwegian ancestry, who knew of his background. They had two children, Todd, born in 1964, and Bliss Broyard, born in 1966. (Broyard had previously been married to Aida Sanchez, a black Puerto Rican with whom he had had a daughter Gala, but the couple divorced after Broyard returned from military service in World War II.) The Broyards raised their family in Connecticut.
In 1997, the scholar Henry Louis Gates discussed his view of how Broyard had concealed his African-American ancestry in an essay in his book Thirteen Ways of Looking at a Black Man, developed from an article in The New Yorker entitled "The Passing of Anatole Broyard." In 2007, Broyard's daughter Bliss published a memoir, One Drop: My Father's Hidden Life -- A Story of Race and Family Secrets, in which she described her journey of learning about family mysteries.
Broyard and the way he dealt with his ethnic background were said to have been the inspiration for the character and situation of Coleman Silk in Philip Roth's acclaimed novel The Human Stain. Roth however states there is no connection as he only learned about Broyard being black from a New Yorker article published months after he started writing the novel.
Broyard's cause of death was prostate cancer, diagnosed in 1989.