1. Does OS thread M get pinned to run only on a particular processor P? (It seems like "yes" when default.)
2. If M blocks in a syscall too long in the optimistic case:
2.a. is M unpinned from P but continues to block until the syscall returns?
2.b. is another thread from the pool used or new thread created, and pinned to P so that P can be used for other work? (I think this depends on configuration if there are fewer, same or more threads than processors.)
2.c. is there an upper limit on outstanding blocked syscall worker threads or will it simply be the last task any extra created threads beyond the normal limit would ever process?
An OS thread M can run on any available P. While there are some caches associated with each P, Ps are fundamentally there to insure that only so many CPUs worth of Go user code is ever running at once, so the important thing is that an M that wants to run user Go code has some P, not a particular P. Ms claim and release Ps as they go in and out of running Go user code, but I believe they don't release and then re-acquire a P as they switch between goroutines.
(I believe the actual implementation treats Ms as a sort of secondary thing. For instance, I think that the local list of runnable goroutines is attached to the P, not to the M. At one level, the M is just a context for running things on Ps.)
In the optimistic case when the system call blocks for too long, the M is unpinned from the P it was using and continues to sit in the system call (the Go runtime doesn't attempt to interrupt the system call itself). If there is another runnable goroutine and there are no free M's, the Go scheduler will create another M to run the goroutine on the now-free P. I think that the runtime directly allocates the free P to the newly created M rather than letting the new M try to contend with other things for the P, but I'm not sure.
I don't think there's any limit on the number of Ms (OS threads) that the Go runtime will create, but I haven't checked the code carefully. Idle Ms are reclaimed under some circumstances.
2. If M blocks in a syscall too long in the optimistic case:
2.a. is M unpinned from P but continues to block until the syscall returns?
2.b. is another thread from the pool used or new thread created, and pinned to P so that P can be used for other work? (I think this depends on configuration if there are fewer, same or more threads than processors.)
2.c. is there an upper limit on outstanding blocked syscall worker threads or will it simply be the last task any extra created threads beyond the normal limit would ever process?