Where It All Goes Down
We're not in a garden watching a pair of swans do their thing. Thank goodness for that. Instead "[i carry your heart with me(i carry it in]" looks as if it takes place in the speaker's heart. How cool (gruesome) is that? Of course the speaker doesn't literally say we're in his heart, but we get the feeling that all the parentheses serve to mimic the feeling of being inside and outside of his lover's heart. Outside of the parentheses, things tend to look simple and less clichéd: "i am never without it" (2). But once we're in the parentheses (or perhaps in his lover's heart), the speaker can't hold back on the romance with endearing terms like "my sweet" and "my darling."
In a more symbolic/cosmic sense, we may even be looking at the "roots" of love by the third stanza. And those roots are mighty expansive and envelop the whole world in love. So, in a way, the poem's setting is love. We're right in the heart of it. And if love unifies the whole world, then we can also assume that the poem's setting is everywhere and everything. The speaker doesn't provide any specifics because of love's universal kind of setting—there's no need to box things into a specific time or place, because love is always somewhere to be found.