What we've got here, is a failure to communicate.
Yep, the speakers are one of the major things that make this poem so difficult to read, since they're constantly shifting without any sort of signal to the reader.
In the opening stanza, we seem to hear from a woman named Marie who is looking back with nostalgia on her childhood memories. Later, we hear from someone sitting on the bank of the Thames River and complaining about all the litter, and later still we get a woman chatting inside a bar. The one speaker who seems capable of inhabiting all these speakers, though, is the blind prophet Tiresias, whom Eliot called "the most important personage in the poem." Since he is a prophet or "seer," Tiresias is able to guide us through any scene that is happening at any point in history, anywhere in the world.
Eliot probably puts so much importance on Tiresias because this character allows Eliot to jump all over the place, giving us a cross-section view of modern Western culture and how it stacks up (not so well) against the greatness of the past. Also, Tiresias' visions come to him in little spurts, which allow Eliot to make his entire poem seem fragmented and disconnected. It's up to you, the reader, to try to put all the pieces together, because our modern world no longer references the classic parables or clear moral standards to make sense of everything in terms of a larger whole.
Ultimately, the reason Eliot makes his speaker so fragmented and difficult to follow is because he believes that fragmentation is basically a perfect metaphor for what it feels like to live in the modern world. In former times, the world was held together by a belief in the greatness of high culture and a religious certainty that everything on earth fit into some sort of divine plan. Even while a guy like Dante might have written about terrible places like hell and purgatory, at least these places made perfect sense when you were reading about them.
The structure (or maybe anti-structure) of fragmentation was really popular with modern authors in general, who seemed convinced as a group that after the earth-shattering devastation of World War I, art was going to have to do something to convey the sense of shattered-ness that had affected the minds of everyone in Europe. Some writers would go on to embrace this sense of fragmentation as a good thing, since it opened the door to new ways of thinking about the world. Others like Eliot, though, chose to mourn for the past and to memorialize it as well as they could. And here, the fragmentation of the speaker is the perfect way for Eliot to mourn and embody the fragmentation of the past.