The speaker of this poem is an omniscient narrator—a "voice" that seems to know all the thoughts and feelings of the main character, but who doesn't take on a separate role in the poem. Even though Rich often spoke about women in the third person at this stage of her poetry career, we think it's kind of cool that in this case, the all-knowing "voice" seems kind of like another witness to the way the woman is living (like the milkman and the beetle).
Still, we don't get the sense that the speaker is sitting in judgment here. Nope. No black robes on this person (whoever that might be). Really, it's as though the speaker is sympathetic to the woman in the poem in some way. Even though the man is also described, his thoughts are not. The speaker gives us a peek into what's going on in the woman's mind, instead. It's her frustrated reality that we're being told about. In that way, the speaker is prodding us to contemplate the woman's troubled relationship, her disillusionment with her boring reality, and the ways in which her original fantasy of "living in sin" has fallen apart.