Guy sounded like an aggressive prick that needed a beat down. If there are doubts that you made the scratches then immediately get a third party over (aka the police) to make a decision. I doubt he starts bitching at an officer's decision.
You seem like a decent enough guy. The only wrong thing...