A complex institution is any sort of governing structure that controls the behavior of people around it. The institution functions by means of of certain behavioral rules that are enforced by a person or group of people responsible for punishing people that do not follow them. One of the most common examples of complex institutions is a government.
Complex institutions are more commonly found among developed people groups. To be considered a complex institution, a society must have specialization of labor and a ruling hierarchy wherein the leaders are treated and regarded differently from the common people.
For example, the United States has a governing body that includes the President and other members of the Executive branch of government, plus the Legislative and Judicial branches. The Legislative branch of government creates laws which are then followed by the people. Within the Judicial branch, judges and law enforcement officers ensure that the laws are followed correctly and issue punishments when they are broken, jobs no other branches are authorized to do.
Discussions of complex institutions invariably refer to governing bodies of some sort. In some ancient civilizations, cultural norms dictated behavior without being enforced by a ruler or governing body. Complex institutions arose as a means to control and regulate these cultural behaviors in a way deemed appropriate to the individual civilization.