> In a media briefing ahead of today’s announcement at Oak Ridge, the partners revealed that Frontier will span more than 100 Shasta supercomputer cabinets, each supporting 300 kilowatts of computing.
So 30 megawatts of computing, plus cooling and other supporting services. How do you power something like this? Does ORNL have their own power station (given they have reactor(s) on site)? If power comes from an external station do they coordinate with the station operator when bringing a system like this online?
As has been noted in other comments, we do not have a power station at ORNL. We buy power from TVA at about 5.5 cents per kW hour which in part is because of the locality of the lab to TVA power plants.
TVA recently completed a 210 MW substation on ORNL's campus to better serve our needs. We do not need to coordinate with them for large runs on the machines.
Nice :-) Back in the day in the UK the RAE royal aircraft establishment twinwoods had a direct line to a local power station for their wind tunnels and used to control the speed form the power staion
With that much gear and those kind of loads do you still have a traditional UPS / transfer switch / genset arrangement for everything in the room? If not, how do you manage short duration power outages?
Yep, we have battery-backed generators for UPS and a transfer switch at the 480-V feed that comes into the room but it is not enough to power the compute nodes. The UPS allows cluster management nodes and the parallel filesystem (which is a small cluster by itself) to ride through full outages and other PQE.
Oak ridge national laboratory was built where it is partly because they could get lots of cheap power from the TVA, so probably from that. (TVA is a regional electricity provider that operates a lot of hydro plants.)
For those who are curious, a typical American home uses of order a kilowatt, time-averaged (10,400 kWh per year = 1.2 kW). So 30 MW is roughly the average power usage of a city of 30,000 homes, or 80,000 people, although total capacity will be larger to handle fluctuations.
...and yet, even a machine at this scale, or even 100 times that, could not come close to being on par, in terms of neural/neuron simulation, with that of the human brain.
We are definitely "doing something wrong" when it comes to artificial neural networking; even though are models are much simpler, it still takes an enormous amount of computing power, both in terms of raw CPU as well as actual electrical needs, just to be able to simulate things at a small scale (and if we use more accurate models, based on what we know about the brain and neurons, then at best our simulations can only be run to simulate, over our actual-time, what would be in actually fractions of a second in real-time).
That our brains can do so much using so little power (wattage), with such a high number of nodes and interconnections that dwarf anything we've so far have managed to simulate - it's a bit mind-boggling and humbling.
I just wonder where and what the issue actually is.
Why do our current practical models of a neuron, which are vastly simplified, require so much power to run at scale?
Is the issue related to the fact that they are simplified models, and actual neurons with their complexity are able to do things we don't yet understand or know about?
All of this is also related to back-propagation; such a thing doesn't seem to exist in nature (jury is still out on the theory, though) - so how do biological neural nets "learn"?
If we could eliminate or reduce the need for backpropagation, would that lower our power requirements for artificial network implementations?
As someone who has merely dabbled with artificial neural networks, these questions and conundrums fascinate me, and cause me to attempt to think up potential solutions, however far-fetched.
I highly doubt I will be the one to solve the issue, but I do hope to see it solved within my lifetime.
The main difference between the brain and a CPU is that the brain runs at a much lower, variable clock speed (10-100Hz) to reduce switching costs and makes up for this with extensive pipelining and parallelization when possible. The high node count and necessary connectivity is then possible due to the use of directed self assembly.
At any rate, it is quite likely that the neocortex of the brain simply computes a function(s) recursively upon sensory input (see Chomsky's minimalist program for suggestions on what it could be). What is unclear is what this function is, and how it comes about - for this reason the approach by some has been to attempt to simulate an entire brain to see what it does. But without the necessary abstractions, this will be inherently wasteful and generates nothing new other than validating your experimental data.
Probably current AI is base on FP64, FP32, biological neuron likely be large network of only one or few bits. Once we figure out how to build, use, optimized large network of nodes which only process a few bits each, the power consumption might just go way down.
Someone might also find that kind of network can run a KHz like human brain cells instead of MHz, GHz. The power usage will go down even more.
They might coordinate with TVA for transients (i.e. going from an idle machine to a full-machine run), but in normal operations these systems are at least 50% full. In my experience as a user these machines are more than 80% most of the time. I don't have hard numbers on the average utilization over the last week/month/year (these might even be classified), but you don't by and build such a machine to be idle.
I could find the numbers for two German super computers I have used in the past. SuperMUC (Phase 1 and 2 combined) had "above the desired 85%" utilization in 2017 and Hazel Hen at HLRS in Stuttgart reported a utilization "between 92% and 98%" in 2017.
> When I took a tour of the Oak Ridge Leadership Computing Facility a few years ago, Buddy Bland, who is Project Director, told me he could tell when he comes to the Lab in the morning if they are running the LINPACK benchmark by looking at how much steam is coming out of the cooling towers.
That is true. But very few super computers track power consumption for different codes and do power aware scheduling. SuperMUC and IBM actually do a lot of research on this, because it is a rather new field in HPC.
Most of the super computers today have their own power station on site. I know blue waters at UIUC had one, which I believe caused a power outage at one point.
Not a wildcard solution (or a particularly legit one or one a serious vendor is going to take up, but could work for DIY):
DNS poison within LAN (or proper internal DNS, or even HOSTS if DIY) for
.product.tld
Product initiates letsEncrypt for:
uid.product.tld
And posts some form of identity proof to vendor, setting up a link to carry out letsEncrypt challenges.
Verification negotiated between letsEncrypt and the vendor via public resolution of
uid.product.tld
Device completes cert process with letsEncrypt and installs cert to the device.
LAN now has TLS to uid.device.tld internally (via DNS spoof/poinon) and vendor hasn't seen the cert. (but, if not DIY, the vendor is distributing a DNS poisoner in their product)
() Caveat: I've still not set up letsEncrypt in the wild, so don't know it's limitations, but from the doc's the above looks doable.
The lack of ability to have a browser TLS to *.local without an internal CA or non-public root and the endpoint management to goes with either is a pain.
(Some?) modern cars expose a CAN bus via the OBD port. There typically are pins that adhere to the OBD2 spec as well as additional pins for the CAN bus.
On my car (Mk6 VW) there are multiple CAN buses which are separated by a gateway device. A device plugged into the OBD port has to negotiate with the gateway in order to access the other buses, but once it has done so can control the windows, lighting, instrument cluster, etc.
When I was younger, I loved physics because it bridged the gap between pure maths ("when are we ever going to need this stuff?") and the physical world. I didn't pursue it beyond high school
Now, as a full-time software engineer and part time jack of all trades, I appreciate stuff like this experiment and the work of Space X and others much as I appreciate good engineering. It's a difficult problem to solve. So many disciplines had to cooperate to grant us some small insight into the inner workings of our universe. It's marvelous, and makes me feel like a kid again.
I recently phone banked for my party for the first time and found that the voter rolls the party was working from had age, gender, race, and how many out of the previous 3 elections the voter had actually shown up to the polls.
It is no wonder that politicians ignore issues affecting of young people: they know for a fact that they don't vote in large numbers.
I voted today. I have always voted since I was of age. I get discouraged living in Florida and having my vote drowned in a sea of geriatrics. But I recognize that participating is the only way to affect change.
I can identify. I don't watch that much TV, but my SO does, and the hard drive on our DVR is so small that if I queue something it can delete something of hers.
I have a Playstation 3 which I used for Netflix, and went through the same hassles during the PSN outage. Then it seemed like every time I went to watch a movie there was a new software update I had to have. On top of that, the repeated software updates seem to have introduced a bug where it has to resync the resolution with the TV every time I power it on or boot Netflix. I'm never buying a Sony product ever again.
Netflix streaming is a great product, but has limited selection. Again, it's not that I consume that much media, but that what I want to watch usually isn't available.
At the end of the day, I have a cable TV + internet subscription, as well as Netflix, and I just torrent everything. I have a server that runs deluge, and I just drop a torrent in a folder and a couple of hours later it's ready to watch.
I though the piece was spot on.
edit: Just thought I'd add that since the legal music stores have popped up (iTunes, Beatport, Amazon) I hardly pirate music anymore. I spend hundreds of dollars a year on music. This is why the premise of the article jives with my experience.
So 30 megawatts of computing, plus cooling and other supporting services. How do you power something like this? Does ORNL have their own power station (given they have reactor(s) on site)? If power comes from an external station do they coordinate with the station operator when bringing a system like this online?