In the past year, natural language models
have become dramatically better at the expense of getting
dramatically bigger. In October of last year, for example, Google
released a model called BERT that passed a long-held
sentence-completion benchmark in the field. The larger version of the model
had 340 million parameters, or characteristics learned from the training
data, and cost enough electricity to power a US household for 50 days just
to train one time through.
Four months later, OpenAI quickly topped it with its model
GPT-2. The model demonstrated an impressive knack for constructing
convincing prose; it also used 1.5 billion parameters. Now, MegatronLM, the
latest and largest model from Nvidia, has 8.3
billion parameters. (Yes, things are getting out of hand.)
AI researchers have grown increasingly worried—and rightly so—about the
consequences of this trend. In June, we wrote about a research
paper from the University of Massachusetts, Amherst that showed
the climate toll of these large scale models. Training BERT, the
researchers calculated, emitted nearly as much carbon as a roundtrip flight
between New York and San Francisco; GPT-2 and MegatronLM, by extrapolation,
would likely emit a whole lot more.
The trend could also accelerate the concentration of AI research into the
hands of a few tech giants. Under-resourced labs in academia or countries
with fewer resources simply don’t have the means to use or develop such
In response, many researchers are now focused on shrinking the size of
existing models without losing their capabilities. Now two new papers,
released within a day of one another, successfully did that to the smaller
version of BERT, with 100 million parameters.
paper from researchers at Huawei produces a model called TinyBERT
that is 7.5 times smaller and nearly 10 times faster than the original. It
also reaches nearly the same language understanding performance as the
original. The second from
researchers at Google produces another that’s more than 60 times smaller,
but its language understanding is slightly worse than the Huawei
version. Read more here.
For more on tiny natural language
·Huawei’s full paper, “TinyBERT: Distilling
BERT for Natural Language Understanding”
·Google’s full paper, “Extreme Language
World’s 1st remote brain surgery via 5G network performed in China Published time: 17 Mar, 2019 13:12 · A Chinese surgeon has performed the world’s first remote brain surgery using 5G technology, with the patient 3,000km away from the operating doctor. Dr. Ling Zhipei remotely implanted a neurostimulator into his patient’s brain on Saturday, Chinese state-run media reports . The surgeon manipulated the instruments in the Beijing-based PLAGH hospital from a clinic subsidiary on the southern Hainan island, located 3,000km away. The surgery is said to have lasted three hours and ended successfully. The patient, suffering from Parkinson’s disease, is said to be feeling well after the pioneering operation. The doctor used a computer connected to the next-generation 5G network developed by Chinese tech giant Huawei. The new device enabled a near real-time connection, according to Dr. Ling. “You barely feel that the patient is 3,000 kilometers away,” he said.
Visualizing The Power Of The World's Supercomputers BY TYLER DURDEN FRIDAY, JAN 21, 2022 - 04:15 AM A supercomputer is a machine that is built to handle billions, if not trillions of calculations at once. Each supercomputer is actually made up of many individual computers (known as nodes) that work together in parallel. A common metric for measuring the performance of these machines is flops , or floating point operations per second . In this visualization, Visual Capitalist's Marcus Lu uses November 2021 data from TOP500 to visualize the computing power of the world’s top five supercomputers. For added context, a number of modern consumer devices were included in the comparison. Ranking by Teraflops Because supercomputers can achieve over one quadrillion flops, and consumer devices are much less powerful, we’ve used teraflops as our comparison metric. 1 teraflop = 1,000,000,000,000 (1 trillion) flops. Supercomputer Fugaku was completed in March 202
Too Much Power to the People? A Food Safety Site Tests the Limits Several national chain restaurants have been the target of complaints on IWasPoisoned.com since the site began in 2009. By KEVIN ROOSE FEB. 13, 2018 Dan Laptev, an electronics analyst, was making his way through the Charlotte, N.C., airport this month when he stopped at Starbucks for a light dinner — a ham-and-cheese sandwich and a cup of hot chocolate. He ate, drank, boarded his flight and got home. And that’s when the trouble started. Mr. Laptev spent much of that night hunched over the toilet with a violently upset stomach. Suspecting his Starbucks meal as the source of his ills, he sent a complaint through the company’s website, but got only an automated form email back. So he did the next best thing: He logged on to his computer and went to IWasPoisoned.com, a website that allows users to post reports of food poisoning, and submitted his saga. “I wanted to let people know to stop eating at Star