Ok, but in a multi-user system (e.g. a webserver), if user 1 triggers a (de-)optimization, then user 2 can tell that a previous user was in that code path. Now I don't know how to extract useful information from that fact, but it shows that at least some information spills over the user boundaries.
Oh I see what you mean - Spectre-like rather than specifically Spectre. Yes I suppose specialisation (more generally than speculation) could leak information, in the same way as cache status can leak information.