What you're describing is formal verification. While I agree with what you're saying, I'm not sure if you're just understating the the complexity of formally verifying systems or if you're implying that "being really careful and doing your due diligence" is practically invulnerable.