There are OS internals books that document this kind of thing (e.g., various books on FreeBSD, Solaris, MacOS, etc.). You can also learn a lot just by reading the specifications for various object file formats, or the source code to tools like linkers and debuggers. There are also some dated books that are specifically on linkers and loaders (I've only leafed through them in a bookstore, don't know how good they are).

But on the whole, I'd say you learn on the job. Project, anyway.

In days of yore these systems-level structures often arose out of a project's specific needs and the individual experience of the people on staff. For example, I sat next to the person who designed the GEMDOS executable format (we needed one, the old one in CP/M-68K was terrible), and I think it was done in a day or two. The engineer in question had maybe 15 years industry experience, including some time as a systems programmer at IBM; I think the format would have been different (maybe better, maybe worse, how would we know?) if a different engineer had decided to do that work.

I used a couple tricks from the GEMDOS executable format to do some rather nifty runtime work at Apple (it's not like the format was secret or anything). That's cross-pollination for you.