Symbolism, Imagery, Allegory
That poor, sad horse. It's basically the symbol of Tevye's sad-sack qualities. Tevye is always talking to or about the horse like it's the most bedraggled and miserably underappreciated creature. No matter how well fed and how underworked that animal is, Tevye just sees himself in it and can't help but identify with the way the horse is just forced to accept whatever happens to it.
So, why the horse? Why not his dairy cans, or his cows, or his wagon?
Well, the horse does almost the exact same job that Tevye himself does, delivering the dairy products to nearby towns. The horse is literally a beast of burden—exactly the archetype Tevye sees himself as. And without the horse, Tevye couldn't do his job. The two of them are a team, just as much as Tevye and Golde are.
And when does he most tend to bust out the horse-related references? Either when he's just been given some super disappointing news, or conversely when he's just experienced something awesome and so is obviously due for another setback any second now. Here's a great example, right when Tevye finds out that Tzeitl refuses to marry Lazer-Wolf, the rich butcher:
I sat and drank it all in as I was thinking how cleverly the Creator of the universe had made His little world so that every creature, from a man to a cow, forgive the comparison, should earn its keep—nothing comes free! […] You, horse, do you want to chew? Then run back and forth day in and day out with pots to Boiberik. And the same goes for you, O man. Do you want a crust of bread? Then go toil […] and drag yourself every morning to the Boiberik dachas, bow and scrape to the Yehupetz rich folks, smile for them, charm each one, and be sure they are satisfied and that their pride hasn't been hurt! (4.95)
See how you could easily switch the words "horse" and "man" in that passage, and come out with pretty much the same meaning? It's a neat trick.
Of course it gets a lot more serious in the final story, when Tevye finds himself having to sell his horse because the Russians are kicking him out of town. Selling his horse—selling himself. Exile isn't just leaving behind your home. It's also leaving behind a crucial part of your identity.
Yeah, major bummer.