When it comes to drugs, Alvy just says no. To him, drugs symbolize a lack of control he just can't abide. That's why Annie's persistent need to smoke pot before they get it on irks him so much: If he gets high, he'll lose control of himself; when she gets high, he feels like he loses control of her and the experience. When Annie wants to light up, Alvy tells her:

ALVY: It ruins it for me if you have grass because, you know, I'm like a comedian, so if I get a laugh from a person who's high, it doesn't count, you know, because they're always laughing.

Basically, if Annie's high, Alvy can't have all of her. And Alvy is greedy.

Alvy's and Annie's differences in opinion when it comes to drugs also symbolizes their different attitudes about life outside of the sack. Annie's open to exploration and unpredictability. Alvy definitely isn't.

Take the scene where their friends offer them some cocaine, for example. Annie's game for a new experience (albeit a thoroughly stupid one). Alvy's not, and explains that he just doesn't see the appeal in letting go, while Annie accuses him directly of never wanting to try anything new. Whether it's a joint or a pile of cocaine sneezed across a coffee table, drugs are a symbol of change: Annie welcomes it, and Alvy rejects it like a $3 bill.