Symbolism, Imagery, Allegory
A lot of readers are tempted to look at the penal colony itself as an allegory for something. What's an allegory? Think of it as a story in which the parts of the story (characters, objects, events in the plot) mean or refer to something else, and, importantly, something else in particular. An allegorical story as a whole refers to something else and has a definite meaning or message. It's easier to think of this with an example.
Austin Warren , the first major critic in the English language to write about "In the Penal Colony" (in the 1940s), set the tone for later critics by reading the story as an elaborate religious allegory. In his take on it, the penal colony itself was "the world" – so we are all in the penal colony!
There's a bit more to Warren's interpretation than that. A penal colony is a place where convicted criminals are sent, usually to do labor. Everyone in the penal colony, then, is guilty (of something). This is one way of making sense of the officer's idea that "Guilt is never to be doubted" – it's not so much that guilt for a particular crime is never to be doubted, but that, since someone's in a penal colony, he must already guilty of something. In Warren's religious reading, the world is the penal colony because everyone in the world is a sinner, guilty before the law of God and deserving punishment (source). The idea that "we're all guilty" is a common theme in many religions, and that's one way of understanding the core of the officer/old Commandant's worldview.
So now let's fill out the allegory. Because Warren reads the story as an allegory, he can make sense of most of the characters and parts of it in a pretty neat way. The old Commandant is God, who gives the Law to the world after creating it (he made the penal colony and designed its judicial procedure) and before whom everyone is guilty. See, an allegory makes everything in the story "equal" something else. Old Commandant = God; founding the penal colony = creation of the world; designing its judicial procedure = giving laws/commandments (source).
As for the rest? Warren suggests the following. The machine = organized religion, which keeps all the believers together and imposes penalties on them for disobeying God's law. It's kind of like "the Church." The officer is an orthodox believer, probably a theologian or an Inquisitor (think Spanish Inquisition). The explorer = secular humanist, someone who doesn't believe in God or his commandments and has his own views about what's "humane" or "just" based on non-religious ideas (source).
And you'll see, in Warren's interpretation, the story has a definite point. Kafka, Warren thinks, is sympathetic to religion, and regrets that it is losing its place in the modern world – which is what he sees as the meaning of the old Commandant's loss of adherents and the collapse of the machine. The officer, then, is a kind of hero, who tries to defend it, though it's impossible and he dies with it in the end. The "non-believers" can then laugh at the machine, because they haven't understood it – like the people at the old Commandant's grave.
Warren's interpretation raises a whole host of questions: what does Kafka think the point of religion is? Why would he be sympathetic to it as it's "represented" in the story? We won't get into those here, but feel free to look up Austin Warren if you want to see his answers. (To learn more about Warren's interpretation of "In the Penal Colony, " check out his famous article on the subject: Warren, Austin. "An Exegetical Note on 'In The Penal Colony." The Southern Review, Vol. VII, No. 2,1941, pp. 363-369.)
Anyway, that's what an allegorical reading of "In the Penal Colony" looks like, and there are a number of them, which means there are many options for how you interpret the story, and they can be quite different from each other (though religious allegory is especially popular).