As someone with graduate level math experience and many friends in mathematics grad school, I know what you mean, but it definitely depends on your idea of fun.
You have to enjoy bashing your head against a problem for a dozen hours. Or to be more precise, you have to enjoy the feeling of solving such a hard problem sufficiently much to find math fun.
Heh, fair point regarding the "idea of fun". Disagree, though, that you have to enjoy the feeling of banging your head against hard problems (no one does). You just have believe that getting through it will make you better off eventually.
You have to enjoy bashing your head against a problem for a dozen hours. Or to be more precise, you have to enjoy the feeling of solving such a hard problem sufficiently much to find math fun.