A formal justice system is one in which laws are set and enforced and punishments are administered by state institutions such as courts, police, judges and prisons. There are very few places that don't have a formal justice system, though in some areas, one coexists with an informal system.
The details and make-up of formal justice systems vary from place to place, but they do have some things in common. Formal justice systems are always based on a written code of laws and a list of punishments that can be assigned for violating each one. Therefore, punishment for a crime cannot be based on the whim of the judge or court.