Not precisely. Sure, with a session open on your computer and the token inserted, anyone controlling the computer controls the session. But once you close the laptop or take out the key, the access is gone (this is how it works with the most up-to-date challenge-response systems). And yes, you will find out soon, since the results of the recent active session will be all wrong. An attacker can't use the credentials at a later date. I am not saying this is perfect (no security will ever be), but it's stronger than long-term passwords, with or without a password manager.
Even for credentials for multiple sites stored on one token (which is less safe), you at least get the ability - even offline - to see what credentials have been provided to whom. It's just a fundamentally better authentication paradigm, although certainly not the solution to all the world's security ills.
If you're giving the computer, your cryptographic keys from your security token in order to start a session then a malicious program on a rooted machine will have access to them too.
Good tokens don't work that way. Normally your giving out a 5-10 digit code that is valid for 30 seconds and can be compared with a secure server somewhere. Physical devices plugged into the machine often work the same way, but use a longer code that you don't need to enter. In either case the token never gives up it's private key's.
No one is questioning that it's better, just whether it makes sense to conflate a still unfortunately hypothetical system with something immediately available. Until the US financial system starts taking security seriously, I can't force them to switch to token-based auth but I can start using a password manager.
I also find the distinction above almost meaningless for webapps: if an attacker has privileged access, you're screwed. Maybe they can't make actions while you're not logged in but they can hijaack the session and control everything you see each time you do login. Using a more trustworthy client is your only realistic option until banks start requiring one-time codes for every transaction.
Even for credentials for multiple sites stored on one token (which is less safe), you at least get the ability - even offline - to see what credentials have been provided to whom. It's just a fundamentally better authentication paradigm, although certainly not the solution to all the world's security ills.