The phrase, "something is rotten in the state of Denmark" means that something is wrong with a certain situation. There is cause to be suspicious of people and their motives. The line first appears in Act 1, Scene 4, of the Shakespearean play, "Hamlet," when a character describes the disturbing conditions that the country is facing.
When the line appears in "Hamlet," the title character has learned from a friend, Horatio, that there have been sightings of Hamlet's father's ghost. Hamlet decides to see for himself although Horatio and another friend, Marcellus, attempt to persuade him otherwise. Hamlet sees the ghost, dismisses his friends and follows the spirit. Marcellus utters the famous phrase, frustrated with the situation.