According to Martin Scorsese, the most crucial scene in the whole movie—the point it hinges on—is when Betsy rejects Travis over the phone.

As he tries to convince her to go out with him again, after that super-awk porn movie disaster, we can't hear her responses, but we understand that she—quite sensibly—isn't willing to go out with a guy who acted like a total creep on the first date. However, we're also meant to empathize with Travis because we understand how screwed up he is—we understand that he really isn't capable of interacting with humans in a reasonable way, and that's his tragedy. His entire spiel is desperate and still indicates he doesn't really understand what he did wrong:

"Hello Betsy. Hi, it's Travis. How ya doin'? Listen, uh, I'm, I'm sorry about the, the other night. I didn't know that was the way you felt about it. Well, I-I didn't know that was the way you felt. I-I-I would have taken you somewhere else. Uh, are you feeling better or oh you maybe had a virus or somethin', a 24-hour virus you know. It happens. Yeah, umm, you uh, you're workin' hard. Yeah. Uh, would you like to have, uh, some dinner, uh with me in the next, you know, few days or somethin'? Well, how about just a cup of coffee? I'll come by the, uh, headquarters or somethin', we could, uh... oh, okay, okay. Did you get my flowers in the...? You didn't get them? I sent some flowers, uh...Yeah, well, okay, okay. Can I call you again? Uh, tomorrow or the next day? Okay. No, I'm gonna... okay. Yeah, sure, Okay. So long."

As Travis gives this speech to her, the camera pans away from him and focuses on an empty hall—as Scorsese put it, the rejection is too painful to watch. Also, the empty hall is itself an image of urban loneliness, of New York desolation. When Travis finishes his phone call, he walks down that hall and into a deeper and even crazier form of loneliness.