> the lack of innovation in programming techniques called out in this paper.
What kind of innovation? Compiler optimizations? New language paradigms?
Am I right to think computers should adopt a more parallel architecture and design, and to expand on techs like OpenCL and CUDA, and generalize those technique to everything done by a computer, because we've hit a frequency limit?
We often see software being either sharded, distributed, load balanced, etc, but it seems that there is much more performance being possible if we start building chips that enforce the division of a task on the programmer. Of course it seems like it is a very big paradigm shift, which might be too expensive and complicated, but since bleeding edge techs like deep learning cannot be properly accomplished with traditional computers, I tend to believe the computer model of today is outdated.
> "Am I right to think computers should adopt a more parallel architecture and design, and to expand on techs like OpenCL and CUDA, and generalize those technique to everything done by a computer, because we've hit a frequency limit?"
"...Amdahl's law is often used in parallel computing to predict the theoretical speedup when using multiple processors. For example, if a program needs 20 hours using a single processor core, and a particular part of the program which takes one hour to execute cannot be parallelized, while the remaining 19 hours (p = 0.95) of execution time can be parallelized, then regardless of how many processors are devoted to a parallelized execution of this program, the minimum execution time cannot be less than that critical one hour. Hence, the theoretical speedup is limited to at most 20 times {\displaystyle \left({\dfrac {1}{1-p}}=20\right)}{\displaystyle \left({\dfrac {1}{1-p}}=20\right)}. For this reason, parallel computing with many processors is useful only for highly parallelizable programs..."
In my field, computational chemistry, you are definitely right: when we have more parallelism available, we go for increased accuracy and scope, not reduced latency. The latency is set by human schedules (a coffee break, overnight, etc). So Amdahl's Law does not apply.
> Am I right to think computers should adopt a more parallel architecture and design, and to expand on techs like OpenCL and CUDA, and generalize those technique to everything done by a computer, because we've hit a frequency limit?
Intel Skylake has 8x execution ports with 6x way dispatch PER CORE. Typical code can achieve more than 1-instruction per clock tick, maybe 3 or 4 instructions/clock if you really work hard at your optimization. With 6x instructions/clock as the hard limit due to the uOp cache.
That's a normal CPU. Modern CPUs are incredibly parallel. They just "pretend" to be serial, so that the typical programmer doesn't have to think of those parallelization issues. A combination of compiler (aka: dependency cutting), and CPU (aka: Tomasulo's Algorithm) works together to achieve this parallelism (out-of-order, superscalar, pipelined).
----------
EPIC / VLIW are "leaky abstractions", which bleed the parallelism into the assembly language. But you don't get a lot of parallelism out of EPIC / VLIW, not compared to SIMD anyway.
So if you really have a huge amount of parallelism available, SIMD seems to be a superior methodology (and modern compilers can auto-vectorize loops when the compiler detects the parallelism).
I'm just not sure where EPIC / VLIW techniques come in handy. Its not parallel enough to compete with SIMD, but its still more complicated than traditional CPUs.
A lot of what is done in a computer is branching logic code, and that's not so easy to deploy on opencl or cuda which are to a first approximation arithmetic engines.
What kind of innovation? Compiler optimizations? New language paradigms?
Am I right to think computers should adopt a more parallel architecture and design, and to expand on techs like OpenCL and CUDA, and generalize those technique to everything done by a computer, because we've hit a frequency limit?
We often see software being either sharded, distributed, load balanced, etc, but it seems that there is much more performance being possible if we start building chips that enforce the division of a task on the programmer. Of course it seems like it is a very big paradigm shift, which might be too expensive and complicated, but since bleeding edge techs like deep learning cannot be properly accomplished with traditional computers, I tend to believe the computer model of today is outdated.