SKUs of Haswell with the GT3e graphics configuration include 128mb of on package DRAM, which is intended to give more memory bandwidth to the GPU, but it also acts as a cache for the CPU.
Based on what you're saying, it seems as though AVX-512 and future models with larger, faster embedded DRAM might play very nicely together.
I should have been clearer. I was talking about the cache hierarchy and CPU memory pipes. I suppose there's a divergence in main memory too, with CPUs using DDR3 and GPUs using GDDR5, but as you say you can just throw cache at that problem.
Based on what you're saying, it seems as though AVX-512 and future models with larger, faster embedded DRAM might play very nicely together.