What is the Fallacy?

The fallacy is called the "Fallacy of Presumption" or more specifically, "Hasty Generalization." This involves making a broad or generalized conclusion based on limited or insufficient evidence. It can also be seen as a form of stereotyping or prejudice.

It’s not terribly clear who is accusing whom and of what, but the accusation appears to be an ad hominem Guilt by Association because the argument is directed “to the man” and not the claim.

Now, if the claim is being challenged simply because someone of ill repute or scandal had previously made the same claim it could be a Genetic Fallacy.