Putting all of the peripheral concerns to one side for a moment, look at the circumstances of the accident.
- Narrow road, limited space for two cars to pass.
- Mirror to mirror clash whilst passing.
- Neither car was presumably over any central white line (because there wasn't one)
- No independent witnesses
Given all of the facts, any insurance company would record it as knock-for-knock. Personally, I would explain that to the chap, state their is similar damage to your car and suggest that you both accept responsibility for your own repairs. Explain that the alternative is to take it to the insurers who will do exactly the same but additionally raise both your premiums AND given the excess you'll still need to pay for the majority of the repair anyway.
Any right-minded person will see that there is no point persuing it in reality.
A lot of your decision making seems to be based on their behaviour in the heat of the moment. I would suggest you try and put that to one side and deal with the problem directly. If, after some days to reflect, they are still choosing to be unreasonable then you know the type of person you're dealing with. Until then, don't assume they are actually that unreasonable.
I had a similar knock-for-knock situation recently, well actually someone hit my car it was entirely their fault but the damage was so slight I just didn't consider it worthwhile to chase them for it, paid it myself and put it down to running costs.