Exactly - Apple have always been completely crystal clear that your touch targets should be a minimum of 40x40pts. Your graphics themselves might be smaller, but the touchable area should be at least 40x40.

Text size, certainly, but touchable targets shouldn't be a problem assuming you're following Apple's guidelines. I appreciate that may not always be the case, but then you only have yourself to blame.