Stanza 1 Summary

Get out the microscope, because we’re going through this poem line-by-line.

We sat together at one summer's end,
That beautiful mild woman, your close friend,
And you and I, and talked of poetry.

Our speaker sits with two women, one of whom he is addressing. They are just chilling, enjoying the evening, doing some chit-chatting about poetry. What, doesn't everyone do that?
The other woman is a beautiful friend of the woman he is speaking to, and they all seem to have an informal, comfortable relationship.
By using words like "mild," "beautiful," and "summer's end," Yeats does seem to be establishing a pleasant moment here. Let's see if that lasts

I said, 'A line will take us hours maybe;
Yet if it does not seem a moment's thought,
Our stitching and unstitching has been naught.

As the three chat about poems, the speaker says that some lines of poetry can take hours to write, but when you read them, they should appear to have been written with just a "moment's thought." Basically, if they don't look like the came easily to the poet, then they aren't good lines.
He also uses a metaphor for writing poetry by comparing it to "stitching and unstitching." He's saying that poetry is like sewing, and sometimes you have to undo your "stitches" to get them just right, just like you have to re-write your lines. Bottom line: it's a lot of work.
But in the end, it can't look like it took work, or it isn't good writing. That's a pretty critical view of poetry. Do you agree?
In any case, his own lines might seem to have been written quickly, but if you check out the "Form and Meter" section, you'll see that there's a lot of background work going on, including the rhyming of the last words in each couplet (like "thought" and "naught" in lines 5-6). Judging by that, it would appear that Yeats the poet agreed with his speaker.

Better go down upon your marrow-bones
And scrub a kitchen pavement, or break stones
Like an old pauper, in all kinds of weather;

He says that it'd be better to work hard, doing physical labor.
The "marrow-bones" imagery emphasizes this work; marrow is inside the bones, so to work so hard you reach the marrow would be to work very, very hard. Let's hope he isn't being literal (or the poem would quickly become a horror movie).
"Paupers" are impoverished people. Our speaker's talking about menial (unglamorous) labor, done in all types of weather. It's pretty much the opposite of sitting at a nice desk and composing a poem. There, all the work is done in the mind.
So, why did the speaker jump from talking about poetry to talking about scrubbing floors? Let's read on…

For to articulate sweet sounds together
Is to work harder than all these, and yet
Be thought an idler by the noisy set

To "articulate sweet sounds together" is to make these lines of poetry; after all, a sentence is just a bunch of sounds put together.
The speaker's saying that writing these lines is harder than the manual labor he described in lines 7-9. That's quite a claim, right?
Most people wouldn't agree with him. The speaker says that writing is considered an "idle" type of work by society. He says that they are wrong. Though it might seem easy, it's actually pretty hard.'
The "noisy set" refers to people who like to make their opinions known. It sounds like he isn't a fan of those types of people. They are the ones that think writing poetry isn't real work.

Of bankers, schoolmasters, and clergymen
The martyrs call the world.'

And who are these "noisy" people who don't think writing poetry is real work? Well, they're bankers, schoolmasters, and clergymen—all members of professions that deal with everyday tasks.
Now, the speaker isn't being literal here. Not every banker, schoolmaster, or clergyman thinks writing is useless. It's just that all those professions are considered "useful" by most of the world, whereas writing isn't always thought of in that way.
And he calls the people who think this way "martyrs." A "martyr" is someone who dies for their beliefs. So… is he saying that those who work themselves to death because they believe manual labor is the only real work died for their beliefs? Hmm.
If so, you can see why they might not think poetry is a real profession. It isn't as physically taxing.