He entered the "Alumnat" or little seminary at Vienna in 1794. Here he formed an intimate friendship with Vinzenz Darnaut, the future professor of church history, and with Jakob Frint, later Bishop of St. Pölten. The three men were again united as court chaplains, and remained friends for the remainder of their lives. Meanwhile, Milde became catechist in the Normal High School and successor of Augustin Gruber, and occupied also the chair of pedagogics at the university.
Later, as court chaplain at Schönbrunn, Milde spoke comfortingly to the Emperor Franz I, after a battle lost to Napoleon. The emperor named Milde Bishop of Leitmeritz in 1823, and in 1831 Prince-Archbishop of Vienna.
The year of the Revolution (1848) brought him bitter enmities and severe illness. He was between two fires. On 13 March the storm broke, and four days later he warned his clergy, in a circular letter, not to overstep the bounds of their calling: "Priests are not intended to advise regarding the earthly affairs of men, nor to regulate them, but should only concern themselves with interior matters pertaining to the salvation of souls." But the revolution soon menaced the archbishop. Mock serenades were held repeatedly outside his palace and its windows were broken. On the other hand, a portion of the clergy clamoured that he should be declared incapable of managing the affairs of the diocese and expressed the hope of being led to victory by a stronger personality.
A deputation of the clergy represented this to Milde, who complied as far as possible by retiring to his castle of Kranichberg. When the draft of the fundamental laws of the Austrian constitution was discussed by the assembly of the States of the Empire at Kremsier, the archbishop drew up an address to the assembly:
At the assembly of bishops in Vienna (1849), Milde was chosen one of a committee of five to continue the negotiations with the state. When finally in 1850 the imperial decisions were promulgated, which at first dealt a blow to the existing Josephist system, Milde published a pastoral for the purpose of stilling the tumult: "The uneasiness is indeed in great part the result of misunderstanding, but often also the result of malicious misrepresentation, since, through some newspapers and through speeches made by certain men inimical to the Church, the words of the august decree were distorted, and erroneous representations spread abroad."
The monument erected to him in the left wing of St. Catharine's chapel in the cathedral of St. Stephen in Vienna portrays a catechist bending over two children, inscribed "Charity", to the left, a priest in the act of elevating the Blessed Sacrament, attended by a young priest and a clerk, inscribed "and Prayer". Under these two inscriptions, and extending across the whole length of the monument are the words "link together the inhabitants of this world and those of the next".
Milde's "Lehrbuch der allgemeinen Erziehungskunde" was well known (Vol. I: Von der Kultur der physischen und der intellectuellen Anlagen; Vol. II: Von der Kultur des Gefühls- und des Begehrungsvermögens, Vienna, 1811-13, 3rd ed., 1843). A compendium of the Erziehungskunde was published in 1821. J. Ginzel edited Milde's "Reliquien" (2nd ed., Vienna, 1859), which contained discourses and addresses which he delivered as bishop and archbishop.