Is there such thing as a sexist fallacy?

A teacher tells the classroom before a test that boys are better at math than girls are. The girls will most likely second-guess themselves and do poorly on the test.

I don't think this would qualify as a fallacy, but it certainly is problematic. The phenomenon described in your scenario is what is referred to in social psychology as stereotype threat , which is like a self-fulfilling prophecy based around stereotypes. There has been much research exploring the possible reasons why girls tend to score lower on math, and all the research points to social and cultural issues—the strongest evidence pointing specifically to stereotype threat (see Tomasetto, Alparone, & Cadinu, 2011).

As for the comments of the teacher, she would be perpetuating this harmful myth and activating the stereotype threat, being a causal factor in the girls' reduced performance. This might be better classified as ignorance or bad teaching, rather than fallacious reasoning. As for the girls, they have every reason to trust their teacher and cannot be expected to know about stereotype threat. The effects of stereotype threat are unconscious, that is, there is no or little "reasoning" on the part of the girls who are given this information. It is the background processes at work that lead to the girls doing worse on the test.

References

Tomasetto, C., Alparone, F. R., & Cadinu, M. (2011). Girls’ math performance under stereotype threat: The moderating role of mothers’ gender stereotypes. Developmental Psychology, 47(4), 943–949. doi:10.1037/a0024047

One can make generally true statements about a group based on the group's statistically determined norm.

The trick with such statements is to notice the disconnect between group and individual, to remember the normal distribution, and that we are comparing two different groups.

With two groups whose norms differ by no more than about one standard deviation, a comparison between norms will not be absolutely true because with the normal distribution, a significant number from both groups will end up in the same value range. The smaller the difference between norms, the less valid the comparison between groups.

Or, in other words, such comparisons are themselves probabilistic, not simple binaries.

I do not see the connection between the idea of a sexist fallacy, and the failure of a group of girls to get a high mark on a math test.

First, the idea of a sexist fallacy is a stretch. I think the idea of citing sexist fallacy is actually what we might call a flummoxing response to a perceived sexist comment, or a comment that we would wish to dismiss as sexist, although in fact it is not. In this vein, many ideologues are expert at dismissing rational responses to their rhetoric simply by calling the debater a sexist. This would constitute a name-calling fallacy. Too, it would suggest an attempt at a straw man fallacy.

Next, the test moderator who might use such a statement at the beginning of a mixed-class math test, might be simply reporting a bit of research. Yet, he or she should not be doing research--that would be an ethics violation as well as a violation of the Geneva Convention. At any rate, to utter such a statement, even for humorous effect, would not be advisable, because a case can be made for a suggestible student getting a low grade because of the utterance, whether in fact or not. Unethical students, too, will attempt to make trouble for a teacher who they may not like.

Brain testing has shown the female brain more competent in logical arrangement (in this case, the arrangement of the numbers, functions, etc.) which would indicate to me that the girls should do BETTER than the boys.

This falls under other fallacies (more general than a "sexist" fallacy). It is highly likely that this is a "guilt by association" fallacy because the statement was that [all] boys are better than [all] girls at test taking and it is highly likely that there are overlapping ranges of scores between boys and girls.

Beyond this, it is possible that the statement from the teacher may be a "begging the question" fallacy if the teacher's statement is not backed by evidence.

Additionally, the statement that girls will second guess themselves may also be a "begging the question" fallacy because there isn't direct evidence that the teacher's statement will necessarily have the anticipated effect (especially for all students). Some female students may see the statement as a challenge and work harder to overcome the stereotype and disprove the assumption.

Sexist statements are usually fallacious, but those fallacies are general and apply to racist statements just as well.