Tanistry was a system for passing on titles and lands. In this system the Tanist (Irish Tánaiste; Scottish Gaelic Tànaiste; Manx Tanishtagh) was the office of heir-apparent, or second-in-command, among the (royal) Gaelic patrilineal dynasties of Ireland, Scotland and Man, to succeed to the chieftainship or to the kingship.
In Ireland, it remained fully in force among the main dynasties, as well as lesser lords and chieftains, until the mid sixteenth century, and lingered, albeit in much reduced form, until as late as the 1840s. When in 1943 the Irish Free State appointed it's first new Chief Herald tanistry was not reintroduced. Rather courtesy recognition was granted to Irish chiefs based on primogeniture from the last known chief. 
The Gaels exported their customs, this included, to those parts of Scotland they came to control after 400AD. Contrary to some beliefs, apparently the Picts (the other foundational ethnicity of today's Scotland), did not share the succession principles of their distant Celtic relatives of Ireland and Scottish Gaels. There is basically no female succession or even allowing for through female links in the Irish and Gaelic succession model, i.e ideal form of this tanistry, whereas Pictish succession more than regularly used links through females (maternal grandson apparently was preferred heir to grandfather; and/or maternal nephew to uncle in Pictish custom). Tanist also is a foreign term to the Pictish. The royal succession in Celtic Scotland was limited strictly to the elective agnatic or male line of the Siol Alpein or House of Alpin until the accession of King Malcolm II in 1005. This monarch was the first to introduce the concept of hereditary monarchy in Scotland. He did so to try to eliminate the strife caused by the elective law which encouraged rival claimants to fight for the Throne. However, since Malcolm had only daughters, he also introduced the right of female line succession in Scotland, but not without great conflict and unrest for generations thereafter. The Irish monarchies never at any stage allowed for female line succession, a position which is maintained to the present day.
The usual rules for qualification as a roydammna was that a candidate had to be a member of the "Derbfhine", a kindred all descended in the male line from a common ancestor (usually a great grandfather or great-great grandfather). This is recalled in the coats of arms of representatives of the many clans and septs descended from the Uí Néill royal dynasty, many of which feature the Red Hand. The joints in the fingers, the fingernails, and the hand itself, represented the four/five generations that qualified for inclusion within the Derbfhine. This meant that the group itself became highly exclusive, keeping the kingship within the dynasty and not the wider clan, many of whom were reduced to mere gentry or even peasant status (though they might too share the surname). (These features make tanistry as a clearly agnatic succession mode, and a succession by appointment, being obviously an elective monarchy. The basic requirement of the nature of hereditary monarchy, i.e the outcome of the succession being predictable up to the identity of successor and next heirs by genealogy, is not fulfilled in tanistry.)
The downside of this large and equal group of eligibles was that proliferation of roydammna in each generation might lead to internecine dynastic civil war. Such was the case among the descendants of King and High King Tairrdelbach mac Ruaidri Ua Conchobair (1088-1156). His dynasty, the Sil Muirdeag (who took the surname hUa Conchobhair/O'Connor), had successfully ruled as Kings of Connacht since at least the mid fifth century. Their increasing consolidation of their position - via the annexation of the Kingdoms of Mide and Dublin, plus suborning neighbouring states and lordships to vassalage - paved the way for Tairrdelbach to become the first of his dynasty to become High King.
However competition between Tairrdelbach's many sons induced corrosive warfare between at least four competing main lines, in addition to allied lordships and kingdoms striving for the main chance. This, coupled with the incursions of the Normans from 1169 onwards (especially the machinations of the de Burgh lords of Clanricarde - fragmented O Conchobhar rule till by the early 1500s they were reduced to ruling a fraction of their former patrimony.
Another example of Derbfhine or Roydammna proliferation comes from the Annals of Connacht. It states that at the Second Battle of Athenry in August 1316, in addition to King Tadc O Cellaig of Hy-Many, "there fell with him ... twenty-eight men who were entitled to succeed to the kingship of Ui Maine."
A most publicized case was when the Bruce candidate to inherit the crown of Scotland in 1296 pleaded, among other grounds, the traditional tanistry in his favour. He was primogeniturally seen from a cadet branch of the old royal descent, and thus primogeniture would not have favoured him, but idea of rotation and balance (and his seniority in physical age and experience) made him a credible competitor. A presumably Pictish ingredient to the situation was that both Balliol and Bruce descended through female lines from the royal house, a relationship not dependable on any Irish principle of succession lines, and were allowed to present candidacy, Bruce also claiming tanistry through a female line. This may be an indication that in Scotland, Pictish and Irish succession rules were intermingled. (Although the judicial resolution of that quarrel, dictated by the feudally-leaning English king, went in favour of the Balliols on basis of primogeniture, the subsequent political events reverted that result to an incidentally more "clannish-tradition" direction, and Robert the Bruce, the grandson of the candidate who pleaded tanistry, ascended the throne despite of the fact of representing a rather junior cadet line of the original Royal House - all future monarchs of Scotland then were succeeding on basis of rights of the Bruce.)
Tanistry as the system of succession left the headship open to the ambitious, and was a frequent source of strife both in families and between the clans, but was conversely quasi-democratic. Tanistry was abolished by a legal decision in the reign of James VI of Scotland, who later became James I of England and Ireland, and the English land system substituted. 
The rules of succession of the dynasty of Alpin of kings of Scotland, a dynasty legendarily of mixed Pictish and Gaelic origin and their successors, abided by tanistry rules until at least 1034, used them in certain successions in 1090s, and were pleaded as a part of succession litigation as late as in 1290s. A similar system operated in Wales, where under Welsh law any of the sons or brothers of the king could be chosen as the edling or heir to the kingdom.
In the "Proteus" episode of Joyce's Ulysses the term is used in an ironic context. Stephen Dedalus recalls his meeting in Paris with the exiled Fenian Kevin Egan, who had "prowled with colonel Richard Burke, tanist of his sept, under the walls of Clerkenwell" as they planned an explosion to free an imprisoned Fenian.