I can sort of understand the mild frustration of making an entirely non-racist comment and then being accused of racism for whatever reason. The frustration would be even more justified if someone were to falsely refer to you as a racist in order to defame you.
BUT... when people KNOW that they've said or done something racist, I've seen plenty of them howl "Accusing someone of racism is just as bad as/worse than racism itself!!!" What the actual fuck?
Hell, even if you don't think you're racist, it's still irrational to act like being called one is the worst offense there is. I've seen way too many people do this, and it's nothing short of pitiful.