The issue isn't that you can hear the difference.

The issue is that you can't reencode. I can't take a 256kbps MP3 (or even 320) and then reencode it at 128kbps for my portable media player without significant lossage.