Is this ad hominem?

It depends how it is done. It would only be a fallacy if it is being used as support for the claim that the person is wrong in what they are claiming. For example,

"I hope you can see how toxic my opponent's ideas are. Therefore, what he is saying cannot be true."

It can also be a fallacy if it is implied, but not expressly stated that what the person saying is wrong by pointing out how "toxic" the person is. The reason for this is that the truth of a proposition isn't affected by its "toxicity."

As Dr Bo said, really. It's not fallacious to just note it - in fact, their toxicity could itself constitute a fallacy (ad hominem (abusive) comes to mind). It's what you do from there that might cause problems; for instance, if you use their toxicity to argue that what they are saying is wrong, or otherwise faulty. That would be a non sequitur.

You don't have to engage someone in a debate if they're being an asshole. You can accept that they may have a point (or even be correct), but not wish to discuss it with them because they're awful to talk to.