Where It All Goes Down
There are at least two settings in this poem: the moment when the speaker and his love said goodbye, and the present in which he writes the poem. Okay, that's the good news. The bad news is he doesn't tell us a whole lot about either one. Now, we do know that on the terrible, terrible day of parting, it was cold and damp: "The dew of morning / Sunk child on my brow" (9-10). We also know that the whole thing went down in the morning. Without exactly saying so directly, the speaker paints a picture of a really cold, damp, early morning scene. Just imagine a winter morning, just before or just after dawn, where the temperature is really cold and you'll have a good idea where this goodbye happened.
Fast forward a few years and we have a new setting: the "present" in which the speaker looks back on this sad moment. He notes that people around him talk about his old fame; in fact, we get the sense that he's at some kind of social gathering or party where gossip is flying fast and furiously (note, for example in line 17, the present tense of "They name thee before me"). Other than this, we aren't given a lot to go on in terms of setting.
That, however, will not deter us. Byron had some very specific referents in mind when he wrote this poem (see "In a Nutshell" for more on this), and so it's perfectly acceptable to say that the "present" in which the speaker writes the poem is post-Waterloo, Regency England. Whoa, post-what and what? Post-Waterloo just means after the Battle of Waterloo (where Napoleon was finally defeated), and the period in British history known as the Regency, meaning the period when the actual King of England (George III) was mentally incompetent, and so the country was run by a stand-in or regent, his son George IV.
Now there are a ton of things we could say about this specific setting, but there are only a few important things to know. First, the post-Waterloo period was a time of great peace in Europe. Remember, Europe had been fighting Napoleon in sort of a mini-world war for some twenty years. After Waterloo, people were finally able to chillax a bit. The other thing to keep in mind is that England was still a very socially stratified place; there was still a large gulf between aristocrats and non-aristocrats. Aristocrats like Byron hung out, went to parties, played cards, courted each other, and basically did just about everything but labor. It is likely that, when the speaker talks about gossip in this poem ("they name thee before me"), he is at some type of aristocratic fiesta (a ball or a dinner), where gossip of all sorts was quite at home.
Both of these settings complement the poem's themes. The cold, damp, wintry morning: what better place for a sad, sad thing to happen? Okay, that makes sense, but what about the other setting, where the speaker seems to be hanging out with some friends who keep talking about his old flame? Well, the point there is simply this: even when he's around other people and doing other things, the speaker still feels sad and lonely. It's like he's in pain, and there's no place he can go where that won't be the case.