From the web...

"Benny served time in Leavenworth Penitentiary from 1953 to 1957 for tax evasion. He sold his share of the casino to fellow gambler Joe W. Brown to pay approximately $5 million in legal costs. It was generally understood, however, that Brown was only a caretaker, and Benny regained controlling interest in 1957. He did not regain full control, however, until 1964."

And, here is the Wiki page link:

http://en.wikipedia.org/wiki/Binion%27s_Horseshoe