The interview I saw with Mr. Robbins, he indicated that he was not even under the impression that he was going to speak at the HOF event.

It would be perfectly acceptable for the HOF to ask all their invitees to refrain from making political statements at the event. However it appears that the HOF wasn't concerned about avoiding politics, they were seeking to punish or embarrass Mr. Robbins for his political views expressed elsewhere.

Major League Baseball has distanced themselves from the HOF's actions.