I do see your point. If it were me however I would consider the source from which it was coming from. I think if I were put in that situation I would defend myself once and that would be it. People are going to believe what they want to believe. And frankly I don't really give all that much thought to what people think about me. My friends know who I really am and my true character and that is all that matters to me.