What is Whataboutism actually

I usually see Whataboutism in this form:

Person A makes claim, and targets person B with the claim
Person B does not refute claim, and targets person A with the claim
Person A claims Person B performed whataboutism and retargets person B with the claim

In some cases Person B will try claiming Person A's actions violated the claim even worse than Person B's actions and it becomes a fight over who was worse than the other.

No one seems to deny the claim, they just keep shifting it back and forth. 

I wrote about this in my book Reason, Books I&II.

People are called out on their “whataboutery” when they point out hypocrisy. For example, if a father tells his child not to smoke, and the child says “what about you? You smoke several packs a day!” The argument is that one shouldn’t smoke, not that the one making the argument is exempt from that rule. So a “what about you” response does not address the argument, thus, is fallacious. However, it is a valid question that does warrant a response. Perhaps the father would respond “I am an idiot with no self-control.” But if the father simply responded “that’s ‘whataboutery,’” it would be akin to the Argument from Fallacy , where the implication is that because the response is a fallacy, then it must be incorrect, unreasonable, or undeserving of a response. This is a deflection and an argumentative cop out. Feel free to point out the whataboutery, but respond to the accusation and don’t stop dialog because you have your opponent on a technicality.

This type of whataboutism is also known as  Ad Hominem - Tu Quoque. As you correctly say, the response doesn't deal with the argument and is thus a logical fallacy of irrelevance. 

I think it's worth pointing out that it's fair to highlight when someone is hypocritical and it's only a problem when the person fails to either acknowledge or address the charge against them. For example if someone were to respond "Sure, I am guilty of that, but so are you, so who are you to cast stones" I think that's fair enough. 

If the two people just keep going back and forth, they're both guilty of irrelevance. The first person shouldn't be drawn into responding to irrelevant comments and should steer the topic back to the original argument, however you really need to understand logical fallacies to keep on track and most people don't.