Mystery
Okay, so it's pretty obvious that The Big Sleep falls under the genre of mystery since it's about a detective trying to solve a crime. We have many of the standard elements found in a good mystery story: blackmail, murder, gambling, gunfights and sex (or at least sexual situations).
But The Big Sleep isn't exactly your typical mystery story. For one thing, Marlowe doesn't have the superior intelligence of a Sherlock Holmes when solving the case. He makes blunders, some of them costly, and he's portrayed as a flawed, vulnerable person. And when Marlowe finally succeeds in solving the crime, we're not left with that warm and fuzzy feeling of exhilaration that the criminal has been caught and justice has been served.
On the contrary, The Big Sleep ends on a pretty bleak, cynical note, and Chandler doesn't tie all the loose ends neatly together in a big red bow (for example, we never know for sure whether Owen Taylor was murdered or committed suicide). So while The Big Sleep certainly belongs in the mystery genre, it's also a novel about a man named Marlowe who just happens to be a detective, and we spend just as much time figuring out what kind of person Marlowe is as we do piecing together all the clues.
Literary Fiction
Normally when we think of a detective novel, we'd associate it more with the genre of popular page-turners that tend to focus primarily on plot. But Chandler wanted to do more with the genre by bringing it closer to the realm of literary fiction, which emphasizes style, character development, and psychological depth. To pull this off, Chandler spends less time worrying about the plot (much to many critics' chagrin), and more energy developing the psychological complexity of his characters.
Quest
When we hear the word quest, we usually think of works like the Odyssey or The Lord of the Rings, where there's a whole lot of traveling involved. But quest stories don't necessarily have to include crossing endless miles of land and sea. In The Big Sleep, the idea of the quest is used more subtly. Marlowe has to navigate the seedy streets of L.A. to reach a goal (solving the blackmail case and the disappearance of Regan). He's gotta overcome numerous obstacles in his search for the truth. And most importantly, Marlowe sees himself as a knight-figure, and we all know that knights are constantly out on quests, looking for new adventure.