So, you could actually think of "In the Penal Colony" as having two endings, and both of them definitely pose a lot of questions. First, there's the ghoulishly gruesome ending of the main narrative proper – you know, where the apparatus falls apart and turns the officer into a bloody pulp, without any of the "transfiguration" he wanted? Then, after that, there's the little "epilogue," which is separated from the rest of the narrative. That's when, walking back to the penal colony, the explorer visits the grave of the old Commandant in the teahouse, sees the people there laugh at it, and then goes in a hurry (we think) to leave the soldier, the condemned man, and the penal colony behind.
Yes, the significance of each of these can change a great deal depending on how you read the story, on whether you use an allegorical frame and all of that. To avoid getting ourselves into a huge mess, we'll just ask a few questions which are relevant regardless of how you read the story symbolic reading.
Let's start with the ending of the main narrative – we have a lot more to say about that. The big question here is: why does the machine go haywire? You could offer several reasons. One is that it falls apart because it's just, well, old and falling apart. We already know from the officer that it has some hardware issues.
You could even see it as the final, big moment in the collapse of the world of the old Commandant, which had been going on for some time. We already know there aren't many (or any) adherents left, besides the officer, and we've also seen a number of things "go wrong" or "not as they should" before the big disaster at the end: the condemned man has vomited on the bed, a strap has broken, the officer has dirtied his hands (rather than cleaned them) by putting them accidentally in filthy water. All of this gives a sense that the whole world of the officer and his traditional penal system is on its way out. The machine's just the last piece to go.
On the other hand, you could also make something different of it. Perhaps the machine destroys itself because it can't perform the particular sentence the officer gives it: "BE JUST!" You might think this for several reasons. One is that the whole procedure is so unjust that telling it to be just destroys it, maybe because the destruction of the apparatus is the only way to be just. Or perhaps it destroys itself because there is no real justice. What would "justice" be? That's actually one of the big questions of the story, we think, and it's not guaranteed that there's an answer. Or maybe "BE JUST" is impossible because everyone is guilty and incapable of being just (like it seems the officer might believe). Finally, it could be that the apparatus can't "BE JUST!" because the officer has been perfectly just. Ooh, bet you didn't see that one coming.
Another important question relevant that's worth asking at the ending: is the officer a hero? Is he a victim? Is he a villain? What do you think? How you answer this question may also pull you in the direction of one or another answer to the first question about the machine. What seems clear is that the officer is unable to save the procedure he's so devoted to, and is ultimately destroyed by it. Not only is he destroyed by it (he kind of wanted to be, after all), he doesn't receive the "transfiguration" he wanted more than anything else.
If you view him as a hero, whether it be for his conviction or because you find something sympathetic in his worldview, the ending is tragic, and the officer a tragic hero of sorts. You could also look at him as the sorry victim of forces that he's powerless to stop – the "modern" direction taken by the new Commandant (to become more like the explorer), which sweeps away his old "tradition." Here you might want to adopt the "falling apart" interpretation of the machine in some form.
Then again, if you believe he's a madman, or brainwashed by the old Commandant, you could just look at him as a victim of the falsehood he believes in. Maybe in that case the idea that the machine destroys itself because it can't "Be Just" makes the most sense. And maybe the officer's failure to reach transfiguration was unavoidable: even if the machine hadn't gone ker-plooie he wouldn't have gotten it, because it never really existed. Or maybe he's even an actively evil guy for revering such a brutal and unjust procedure, and gets punished by the machine as a result. We don't want to stop you from exploring other options, either. These are just a few to get you thinking.
On the other hand, you could also wonder whether the Designer was still in fine working order, and whether or not it was the Officer's attempt to use it on himself that destroyed it. Specifically, the Officer tries to put himself through it with the Sentence "Be Just!" Does the machine fall apart because it is impossible for it to be just? Or does it fall apart perhaps because "being just" is itself impossible? Or does it fall apart because the officer has been just, and it can't sentence him? If you really want to run with it, you can ask whether there's any sense to justice outside of the kind of worldview the machine has – namely, an arbitrary one in which everyone is guilty.
Phew, that's a lot of possibilities! What to say about the epilogue? What's striking about it is the way it breaks with the rest of the story. So long as we were listening to the officer, the old Commandant still seemed like a pretty imposing figure, someone to be taken seriously. Either he really was God, or he was just a frightening man (though a man with vision – the system he designed does have this mysterious, ritualistic, religion-like aura about it, no doubt about it).
Once the officer dies, though, and we return to the penal colony, we find that to everyone else the old Commandant's just a joke, as is the officer's reverence for him. "The old Commandant will rise again!" Hah! How wack is that? And at that moment it does seem pretty crazy. Plus, the religious or godlike aura of the old Commandant is also defused when we learn that the priest wouldn't let him be buried in the churchyard. That almost says: regardless of what the officer told you, there's real religion here, and the old Commandant was just a madman who thought he could play God.
What you make of this depends on how you read the rest of the story, and it will influence how you understand the departure of the explorer, who definitely seemed disturbed by something. But what is it? If you look at the officer as mad or brainwashed, then, yes, the old Commandant is a joke, and the story takes on a darkly humorous sheen. Maybe the reason the explorer leaves is from a certain disgust: even if the officer was crazy, he had so much conviction (which the explorer admired), and here's everyone else laughing at it.
Or maybe the explorer feels disgusted for a different reason: that the officer and his whole system of justice could have brutally killed people for so long while taking seriously someone who was ultimately just ridiculous, and who should have been laughed at instead. That's human nature, isn't it? Laughter and horror or revulsion can go together in this case.
On the other hand, perhaps you think there's something to be taken seriously about the old Commandant. Perhaps, if he's not "God" per se, there is something prophetic or powerful about him, and there's something to be said for his worldview. Maybe it does represent some kind of religious way of looking at the world – one in which there's a possibility of "redemption through suffering," of feeling confidently that justice can be done, of feeling strongly that one has a place in the world and of achieving real satisfaction by submitting to something higher (law…or the Commandant). In that case, it's tragic that the old Commandant is being laughed at, and that's why the explorer leaves. Or perhaps the explorer is genuinely horrified at the brutality and gruesome character of the old Commandant's world, while still recognizing its power.
We'll end with one thought, as a way of summing up. Remember that, in many ways, the reader and the explorer share a similar position in this story: the position of watching the entire bloody spectacle unfold, and of doing so with (it's presumed) a similar set of "humane," "just," "modern" eyes. Why should we expect that the explorer knows what to make of what he's experienced any more than we do? Perhaps the safest way of explaining why he leaves is because he's overwhelmed and horribly troubled by what he's seen, in large part because he doesn't know how to understand it. Just like we don't. All we can say is that the ending is overwhelming.