The poem starts with the speaker reflecting on a coastal town in Maine. His observations track an elderly, wealthy woman who seems to have a ton of property, but who is quite alone in her older years. Then he starts to describe the things that have begun to go wrong with the place – the millionaire who lived there for the summer is gone, the town decorator seems depressed and poor, and now the whole place is looking pretty sad for fall.
Then the speaker shifts the focus to himself. He remembers a drive he took through the town one night and what he saw as well as how he felt. It all seems pretty gloomy, and he admits to being depressed and feeling kind of crazy. What he observes after he lets us in on his mental state seems to be affected by how he's feeling, and as the poem progresses it gets pretty bleak.