What was Big Willy Shakes going for?
This quote would make a lot more sense if we told you that "protest" in Shakespeare's day meant "to vow." So when Gertrude says that the player queen, a.k.a. her doppelgänger protests too much, she's actually saying she's too insistent in supporting whatever's taking place. She's telling her husband one too many times that she will never remarry.
The lady loses her credibility with Gertrude because she's too elaborate in her promises. So what she's really saying is "Vows are silly." Hmm… what other vows do people make to one another? Marriage vows, perhaps.
Maybe Gertrude is hinting at the fact that marriage vows are not as important as people think. After all, she changed her marriage vows with the tide when Claudius became king. Or maybe she's saying the exact opposite: that marriage vows in the play she's watching are meaningless, but they aren't in real life.
Whatever she's saying, Shakespeare left this one obscure. That's one of the things we love (and hate) about Shakespeare. He loves to give us a little nugget of gold and then let us decide what it means.
We're not sure whether Gertrude was involved in her husband's death or not. We're also not sure if she cheated on her husband. We're not even sure how much she knows at this point. Even though The Mouse-Trap is supposed to catch Claudius, it's also aimed right at Gertrude. Hamlet wants to know what mommy dearest has been up to. And her little comment on the player queen doesn't given him any solid answers.