The title A Great and Terrible Beauty comes from the following line in the book:
Our bodies feel like luxurious sighs as we stand in the great hall, completely invisible. Oh, God, the great and terrible beauty of it. (29.63)
In this moment, the girls are all invisible. They are surrounded by the members of their school, and yet suddenly above the school's rigid rules and expectations—they are powerful in a place they usually feel powerless, which is a beautiful thing, and yet there is something terrible about the fact that they have to be invisible in order to become so.
And this, of course, is true of the girls and their power throughout the book. As they become more powerful—as their power literally becomes greater—so too does their ability to do terrible things with it and to maintain it (here we can think of Felicity and the deer for an example). But at the same time, their power consistently comes at the price of invisibility. They must be intensely secretive about the realms, and for Gemma in particular, as she steps into her powers, she always has to hide them from society. It is great and terrible at once.
The title is also a reference to how much pressure is placed on the girls to be beautiful. Society wants them to be submissive wives, easy on the eyes and little else—but to become that sort of beautiful is a terrible experience, requiring the girls to let go of their own opinions and sit quietly in service of their husbands for the rest of their lives. Ugh.