One key distinction between Renaissance and Baroque instrumental music is in instrumentation; that is, the ways in which instruments are used or not used in a particular work. Closely tied to this concept is the idea of idiomatic writing, for if composers are unaware of or indifferent to the idiomatic capabilities of different instruments, then they will have little reason to specify which instruments they desire.
Howard Brown, while acknowledging the importance of vocal transcriptions in Renaissance instrumental repertoire, has identified six categories of specifically instrumental music in the sixteenth century:
While the first three could easily be performed vocally, the last three are clearly instrumental in nature, suggesting that even in the sixteenth century composers were writing with specifically instrumental capabilities in mind, as opposed to vocal. In contention of composers' supposed indifference to instrumental timbres, Brown has also pointed out that as early as 1533, Pierre Attaignant was already marking some vocal arrangements as more suitable for certain groups of like instruments than for others. Furthermore, Count Giovanni de' Bardi, host of a gathering of prominent 1580s scholars and artists known as the Florentine Camerata, was demonstrably aware of the timbral effects of different instruments and regarded different instruments as being suited to expressing particular moods.
In the early Baroque, these melodic embellishments that had been improvised in the Renaissance began to be incorporated into compositions as standardized melodic gestures. With the Baroque's emphasis on a soloist as virtuoso, the range of pitches and characteristic techniques formerly found only in virtuosic improvisation, as well as the first dynamic markings, were now written as the expected standard. On the other hand, some of the instrumental genres listed above, such as the prelude, toccata, and intonation, were improvisation-based to begin with. Even in the early sixteenth century, these genres were truly, idiomatically instrumental; they could not be adapted for voices because they were not composed in a consistent polyphonic style.
Thus, idiomatic instrumental effects were present in Renaissance performance, if not in writing. By the early Baroque, however, they had clearly found their way into writing when composers began specifying desired instrumentation, notably Claudio Monteverdi in his opera scores.
The application of this principle to instrumental writing was partly an extension of the forces of change in vocal writing stemming from the Florentine Camerata and their head Count Giovanni de' Bardi, who deliberately sought to change the way music was written, and adopted an overarching goal of a music renascence. In a c. 1580 letter to Giulio Caccini, a composer and member of the Camerata, Bardi decried counterpoint's obscuring of the text in vocal settings and advocates a return to the music of the ancient Greeks, which he believed consisted of a single singing line and simple accompaniment, allowing direct, intelligible expression of the text. He instructed Caccini to "make it your chief aim to arrange the verse well and to declaim the words as intelligibly as you can." While Bardi's letter dealt with vocal music, the principle of a single, clear melody dominating a simple accompaniment easily carries over to the instrumental realm. This is seen in the proliferation of hitherto unknown solo instrumental sonatas beginning shortly after Caccini's Le Nuove Musiche in 1601.
The rise of instrumental monody did not have its roots exclusively in vocal music. In part, it was based on the extant sixteenth-century practice of performing polyphonic madrigals with one voice singing the treble line, while the others were played by instruments or by a single keyboard instrument. Thus, while all voices were still theoretically equal in these polyphonic compositions, in practice the listener would have heard one voice as being a melody and the others as accompaniment. Furthermore, the new musical genres that appeared in the late sixteenth and early seventeenth centuries, especially the instrumental sonata, revealed a transition in ways of thinking about composition and performance, from a collaboration of equals to a soloist backed up by a relatively unimportant accompaniment. In addition, even in the mid sixteenth century, most works for voice and lute were conceived specifically as such. In the realm of English ayres, for instance, this meant that composers such as John Dowland and Adrian LeRoy were already thinking of a dichotomous melody and bass, filled in not with counterpoint but with chords "planned for harmonic effect."
This necessarily led to a change in the types of instruments that were preferred by composers, for many instruments of the Renaissance were greatly limited in pitch range, being designed only to play a discreet role in a consort, as well as in dynamic scope. Entire families of instruments, such as racketts and shawms, were unsuited to carrying a solo melodic line with brilliance and expressivity because they were incapable of dynamic variation, and fell into disuse or at best provided color in string-dominated ensembles. The low instruments of the woodwind consorts were all but abandoned. Even in the string family, members of the viol family – except for the bass viol, which provided the necessary basso continuo – were gradually replaced by the new and highly virtuosic violin. The lute and viola da gamba continued being written for in an accompanimental role but could not compete with the violin in volume. The shawm was replaced by the oboe, which had a more refined sound and was capable of dynamic nuance. The cornett, which in the Renaissance tended to function as the soprano member of the sackbut family, survived in the early seventeenth century as a solo instrument, even having a large repertoire rivaling that of the violin, but eventually disappeared as well. However, Renaissance instruments did not vanish from use quickly; contemporary references indicate such instruments survived in chamber or military contexts well throughout the seventeenth century and even into the eighteenth.
As a general rule, however, one can see in the Baroque an overwhelming preference for those instruments that were capable of carrying a melodic line alone: those that were louder and higher, that could achieve a variety of dynamics, and that lent themselves to virtuosic display and emotional expression, none of which the Renaissance instruments were designed to do. Lower-pitched instruments, those that could not vary dynamics, or those that were cumbersome, were deprecated. Thus, the supremacy of melody in the Baroque mind had wide-reaching consequences in the instrumental choices made by composers and makers.
Such music allowed for highly dramatic effects, with sudden shifts in volume, articulation, timbre and texture, for not all of the choirs were the same size, and could be made up of radically different combinations of voices and instruments. With the addition of the basso continuo in the early seventeenth century, the concertato style (stile concertato) had essentially been developed, featuring a larger overarching ensemble out of which smaller groups were selected at will to play successive musical phrases in different styles, or to perform simultaneously in different manners. Thus one phrase might be soloistic, the next set in imitative polyphony, the next homophonic, the next an instrumental tutti, and so on. Alternatively, a chorus could declaim a text homophonically while violins played in an entirely different style at the same time – in a different register, in a different location in the church, all performed over a basso continuo. The stile concertato spread throughout Europe and was particularly dominant in Italy and Germany, later forming the basis of the Baroque concerto, the concerto grosso, and the German cantata.