Symbolism, Imagery, Allegory
Remember that an oak tree is the first thing to take a bite out of baby Grendel. When he catches his foot in the trunk of that tree, it's also his first close encounter with Hrothgar and his troops—and as we know, it doesn't end well. In part, the oak tree is another one of those unthinking robots of nature that so frustrate and isolate Grendel.
To be trapped by an uncaring, unspeaking, non-philosophizing, photosynthesizing monstrosity gives Grendel his first major reality check: the world doesn't care about him. It's a tree-eat-monster world.
Naughty Nature
This kind of "natural" hostility happens over and over again in the novel. Remember the vines that freak Grendel out because he thinks they are snakes—and sometimes they are (54)? Or the goat that won't get off Grendel's rocks (139)? Or the various woodland creatures that fear him and yet won't leave him in peace (174)? They're all reminders that he doesn't belong... anywhere.
The tree itself comes back to haunt Grendel's nightmares, this time in the form of roots. When things aren't going well, Grendel gets these premonitions of annihilation, and they appear as visions of tree roots and an abyss. It's like both the natural world and the human world are out to get him—and eventually, they succeed. But Grendel doesn't really get it until it's too late, when he's dying in the forest, and he trips on the roots of the oak tree (173).
All along, that oak tree has stood for his ultimate destruction.
Getting Back to Roots
If the significance of the tree and its roots seems random to you, allow us to try to change your mind. Gardner knew a lot about the poem Beowulf and the culture that produced it. In the mythology of Nordic countries like Iceland and Denmark (a.k.a. Grendel-land), there is one very famous and important tree: Yggdrasil.
This bit of shrubbery is the center of Norse cosmology. Wait, what? That means the tree represents a philosophy of heaven, earth, and everything in between. Its branches reach into the heavens, mankind inhabits the middle bit (called Migarðr, or "Middle-Earth"—yeah, Tolkien knew about this, too), and its roots reach deep into places beyond our world. They anchor the whole thing in place.
A naughty dragon named Nidhogg (Niðhöggr in Icelandic) eats up one of those three roots and threatens the balance of the whole set-up. The tree itself, its roots and the creatures that live in it, is a symbolic way of thinking about the universe, existence, and annihilation (that's what old bearded scholars call a cosmology). So if you're going to read this story like a proper Dane and a good medievalist, note that a tree is never just a tree, especially if its roots and branches play a big part in the story.