Unfortunate as it is, since horse racing was the only form of legal gambling aside from lotto of course it had a strong folllowing.
That is not to say that some didn't follow it for the "sport"
However in this age where everything around us moves at a much faster pace, people no longer want to go to the track and have to wait 20 minutes or more to see the result of their wagers.
Some tracks, like Roosevelt Raceway are gone completely.
Others have tried to add entertainment, restaurants, and generally tried to spruce up the tracks but it hasn't helped much.

I'm afraid you are right, it's only a matter of time. Perhaps aside from the Saratoga meet and events like the Belmont stakes, you will no longer see 30,40 and 50,000 people flock to the tracks