A passive speaker of a language can have a basic command of the grammar and can use it to a small extent, but not be fluent. It may be implied that the passive speakers are not trying to improve their skills in the language as well.
The term may especially describe speakers of a minority language or language being eroded away due to cultural changes. Around 10% of the Ainu people who speak the language are considered passive speakers.
Passive speakers can also be people who understand a language but can not speak or respond in it. This is common in areas where people grow up hearing another language outside their family with no formal education. An example of this would be people who can understand, however vaguely, a Spanish speaker despite not being able to properly respond.
A passive language, however, is a term used in interpreting. Typically these are languages being spoken to (an) interpreter(s) that get translated to another language. There can be many of them, such as if an interpreter must translate Mandarin Chinese, Japanese, and German into English. English is the active language while the others are passive. If the situation were reversed, there would be one passive language (English) and three active languages (Chinese, Japanese, and German).
As a technical term, it is unrelated to the idea of passive voice.