I am 100% sure that the inlay has fallen out. The linen pattern was on the bottom of the inlay when it was pressed into the soft clay.

I would guess that the chip was an old crest and seal.