While I follow the stakes races VERY closely, I don't wager much. I bet all the big race days, I go to The Spa (Saratoga for you unwashed heathens) about 10 days out of the meet, and I host a Breeders Cup Party every year. Betting on the horses tends to play havoc with my poker bankroll.
Once I began to follow the sport closely, the betting became second to the racing itself.