Symbolism, Imagery, Allegory
Every room in Darlington Hall has its purpose. Yeah, there are about a bajillion of these rooms, but trust us (or trust Mr. Stevens, at least)—they all matter.
For the butler, the pantry is a kind of office and sanctuary. The housekeeper has her parlour. These strictly defined spaces symbolize the distinct roles the butler and the housekeeper play, but they also symbolize their isolation from each other. They exist in different universes. They have separate jurisdictions within the house. It's all very orderly… and we know Mr. Stevens wouldn't have it any other way.
When Miss Kenton barges into his office, Stevens feels that she has crossed a boundary and his space is being violated. Likewise, he doesn't presume to walk in on her parlor uninvited, even when he hears her crying. This is symbolic on two levels. On the macro level, it shows how divided and stratified English society was back in the day on every level: the sexes didn't mix, different classes didn't mix, and even within a contained staff body there was a premium placed on, well, "place." You were either in your place or you were way the heck out of it.
On a micro level, these two domains show how isolated Mr. Stevens and Miss Kenton were from one another. Sure, they did have some close moments—the afternoon cocoa they share in Miss Kenton's parlor marks a moment when their friendship is at its most intimate—but ultimately Miss Kenton ends up holed up in her parlour.
It is a recollection of standing alone in the back corridor before the closed door of Miss Kenton's parlour; I was not actually facing the door, but standing with my person half turned towards it, transfixed by indecision as to whether or not I should knock; for at that moment, as I recall, I had been struck by the conviction that behind that very door, just a few yards from me, Miss Kenton was in fact crying. (7. 33)
This is the point at which we usually scream "Mr. Stevens, you idiot!" and throw our copy of The Remains of The Day across the room.
Compare these rigidly defined spaces to the guesthouses and inns that Stevens encounters on his road trip. The inns are often noisy: tons of people are all jumbled together. When Stevens gets lost in Moscombe, some villagers convert their son's bedroom into a guest room: sharing is caring in Moscombe.
When he finally arrives in Cornwall, the rain pours into a courtyard, driving everyone inside. These spaces are not restricted—they enable mingling in a way that Darlington Hall does not. Just another point of proof that life inside Darlington Hall (or, perhaps, in proximity to the classist stratification that crops up in places like Darlington Hall) is toxic to making friends and influencing people.