A secret weapon is either a concealed weapon, or a weapon that is not officially confirmed by the owner.
In terms of large-scale weapons, a secret weapon may refer to a newly-designed or invented weapon that the government denies the existence of. For example, during its development, the atomic bomb was considered a secret weapon.
In terms of personal weapons, a secret weapon is a weapon that is hidden, or a weapon of unexpected design. Examples include umbrellas that fire bullets (Penguin in the Batman series) and watches that fire lasers. The concept of secret weapons was widely popularized by the James Bond novels and movies.
The very nature of secret weapons is special, as they tend to not stay secret for very long, if they are actually used as weapons. The German WWII V1 flying bomb, for example, only stayed secret until they were fired against allied targets. The secrecy factor relies on how many people know of the weapons development and ultimately, how many people remain alive to report about the weapon's existence in the first place once the weapon is used.
In the example of a weapon being concealed (such as a gun underneath a trench coat) the weapon would remain a "secret" if a) there are no witnesses to the carrier of the weapon prior to the event and b) there are no witnesses left after the event (other than the carrier).
"Secret weapon" can also be a figurative phrase referring to an unknown technique of any kind that a person uses suddenly and surprises his opponent. For example, a debater may use references to a certain authoritative source as a 'secret weapon' to discredit the claims of the opponent.