Facial shape, size, and facial hair (as another commenter here already mentioned) can absolutely matter in mask fit and seal. This is a very common problem in CPAP masks, which use an inflated bladder to adjust to the face of shape. So I can't imagine this *not* being a problem in N95 masks which don't have a bladder but only an edge of slightly flexible material.
And if we're being pedantic, men typicall have ~10% larger lung capacity than women, which may well have an impact on mask performance as well.
So while the original commenter was factually incorrect, the concern doesn't seem terribly far fetched to me.