The You Don't Understand Argument

I call this the reverse Voltaire. If I am listening to a person with whom I agree but whose accent, pitch, cadence, or inflection is grating to my ear, I sometimes say that I agree with what he says, but I will not defend the way he says it. 

If you want to continue the discussion, then I think the best way to do so is to ask how you did not understand the annoying Boston or Brooklyn accent (or whatever the annoying aspect was).

This could be amazing familiarity , assuming the person isn't close to the politician, and assuming the person cannot actually read your mind, which is most likely true.

Otherwise, this is an assertion without support.

Off the bat, the phrase "You don"t understand" could be taken as an ad Fidentia fallacy as it attacks, (undermines confidence) in your ability to comprehend or tolerate the speaker in question's speech challenges, style, or impediments as the cause of your annoyance.

Person 1: X says many things I agree with, but the way he says them is annoying.

Person 2: You only find him annoying because you don't understand why he speaks that way. (He talks like a New York cabbie, he has a speech impediment, he was born in a pickle barrel, he's an insufferable demagogue, it's his way of appealing to the common man, etc.). 

Therefore: X is not annoying

X may say many things you agree with AND can still be annoying. It has nothing to do with your comprehension skills or pet peeves. 

It reminds me of the affirming the consequent . It could be that
1)Some people don't understand why the politician is speaking that way and that annoys them
2)You are annoyed by the way the politician speaks.
3)Therefore, you must be one of them.
Definitely it shows poor communication because although you already told him WHY you found it annoying (so there's no need on his part to guess), he chose to believe that specific narrative of his and arbitrarily make you a part of it.