as soon as you can, remove the sponge/foam spacer! I have found that over time, the sponge will adhere to the dice, through a chemical process and that process WILL leave the dice with a blemish where there was contact with the sponge.

In my opinion, there is nothing intrinsically special about the sponge/foam material that warrants keeping it and you are running a risk of damaging the dice.

Jim