Symbolism, Imagery, Allegory
These two ideas come up a lot. And they really piss off the Underground Man. His argument goes like this: 1) free will means man gets to do whatever he wants. 2) The laws of nature say that 2+2 always has to equal 4. 3) He wants 2+2 to equal 5. 4) The laws of nature are a stone wall standing in the way of his free will.
The metaphor continues when the Underground Man compares the way a man of action would act as opposed to a man of consciousness. The man of action sees the wall and goes, "Oh, it's a wall," and leaves it at that. But the Underground Man is "not going to be reconciled to it simply because it is a stone wall." Rather, he's going to smash his head against it, even knowing the futility of his actions.
So how is this a symbol? Well, when the Underground Man talks about the rule of 2+2=4, he isn't just talking about 2+2=4. He's also talking about…3+3=6? Well, yes, but even more than that, he's talking about all rules of rationale and logic: the rule that if you're hungry, you'll eat; if a dog is threatened, he'll show his teeth; if someone hits you, you will feel pain. He finds all of these rules to be restrictive in the same way that this very specific example – the 2+2=4 example – is limiting to his free will. The stone wall, then, is a great way to represent the barrier that these rules construct.