Greg Cochrane mentions that here, http://westhunt.wordpress.com/2012/08/22/paternal-age/ , in the context of the influence of paternal age on mutation rates, and excessive numbers of mutations having negative influences on fitness and health.
It could be that men with some medical condition are both likely to father an autistic child and to postpone child-rearing for health reasons. Heck, the autism itself in a very mild form might be such condition - a mildly autistic man could be likely to pass more severe autism to his children, and have problems finding a mate.
