The speaker of a poem is always going to be the "person" who is "speaking" the words of the poem. While the poet is the one who actually wrote the poem, the speaker is the character whom the poet intends to invoke—sometimes, however, the poet means to write as her/himself.
Notably, the speaker could be anyone, anything or even multiple people or things. This is determined based on the narrative perspective, tone and use of language in the poem. Narrative perspective will help to determine whether the speaker is a narrator (third person), a character in the poem (first person) or speaking to the reader (second person).