The site in the canton of Ancy-le-Franc near Tonnerre in Burgundy he inherited from his mother. He demolished the old château-fort and used some of its foundations. The new structure was symmetrical: four ranges enclose a central court, and at each corner a pavilion was erected. Though it was never expected to be defensible, French tradition ran so strong that it was surrounded by a moat. The exteriors of the ranges have a rusticated ground storey, Doric pilasters, niches on the piano nobile. In the central court, an unbroken corbelled cornice runs above two storeys, where rectangular windows alternate with shell-headed niches that are flanked by double pilasters with Corinthian capitals. Niches that span the corners are an anomaly. On the ground floor the fenestration is in blind arcading. The main door, approached by a flight of steps has a typically Serlian runsticated arch, crammed tightly between pilasters.
The interiors have the richest array of sixteenth-century frescoes and wall-paintings after the Château de Fontainebleau.
Ancy-le-Franc had a parterre garden centered upon one of its façades.
In 1683, the Clermont sold the château to Louis XIV's minister Louvois. His eventual heirs sold it in 1844 to Gaspard-Louis-Aimé de Clermont-Tonnerre, a descendant of the builder. After being the property of the princes of Mérode, and of other owners, it now belongs to Paris Investir, a French company dedicated to the rehabilitation and restoration of the château.