Where It All Goes Down
The South Seas, the South Pole and Other Points, Um, South
Poe establishes a fairly concrete time and place to start with, then keeps it intact even when more fantastic elements come into play. He's vague about the year, but the setting is very specific: we start the journey in Java and then move south towards Antarctica.
But that's where the certainty ends. People knew very little about the South Pole when he wrote it, and plenty of wild theories developed about what was there. Poe, for one, was fascinated with the notion of a hollow earth (yeah, that was a thing back then), with the outer crust hiding a giant inner kingdom.
So is that inner kingdom where these zombie sailors are headed? In that case, the whirlpool in the story may be an opening to the Hollow Earth… and might even pop out on the North Pole as well.
Poe doesn't specify, but returned to the notion with his later novel "The Narrative of A. Gordon Pym of Nantucket." (We'll forgive the fact that Antarctica is a landmass, which is something that was only just coming to light when Poe wrote the story.)
In any case, the gist of the setting of this story is that the narrator and his creepy crew are headed off into the great unknown. The setting, like everything else, is shrouded in mystery and the fantastical.