I think some of his mob visited town one day and were asked to leave and to never return.

...per "Galveston -- Island of Chance" by Frank Chalfant