Lost amid the mounting evidence pointing to Chinese research laboratories as the source of the Pandemic, is the continuing saga of the deterministic spying and supply chain corruption campaign the Chinese PRC has been conducting over the past several years.
The same story that magically disappeared from Bloomberg News once the media mogul’s attention shifted to national politics. So, here’s a reminder.
Three years ago, Amazon began due diligence on a startup called Elemental Technologies, as they were considering them as an acquisition to boost their streaming video service (now Amazon Prime Video). Elemental made software for compressing large video files and they had several national security contracts that made Amazon comfortable with cybersecurity considerations and the integration with AWS, as Amazon has many large contracts with agencies like the CIA.
A third-party contractor discovered some troubling issues on their first diligence pass which prompted Amazon to take a closer look at the hardware servers that Elemental customers installed in their networks. It turns out that these servers were assembled by Super Micro Computer Inc., a San Jose-based company (aka, Supermicro) that’s also the world’s biggest independent supplier of server motherboards.
At the evaluation site, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design. Amazon realized that these servers were probably installed at Federal agencies as well, and after notifying the Federal government, discovered that Elemental’s servers were networked throughout the Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships.
And Elemental was of course just one of hundreds of Supermicro customers.
During the ensuing probe, investigators discovered that the chips were designed to allow attackers to create a stealth doorway into any network connected to the tricked-out servers. Not surprisingly, investigators have traced the motherboards back to factories run by manufacturing subcontractors in (gasp!) China.
This attack scenario is far deadlier than the software-based back-doors that we all have grown accustomed to seeing. In fact, a well-done, nation-state-level hardware implant would be like wandering badge-less into the Pentagon unchallenged. Yet here we are with maybe 80% of the world’s servers sporting a motherboard with the additional chip. Completely beneath the radar.
There are two ways for spies to alter the guts of computer equipment. One, known as interdiction, involves manipulating devices as they’re in transit from manufacturer to customer. This approach is favored by our own U.S. spy agencies. The second method involves seeding modifications into the core processors at birth.
China makes 75 percent of the world’s mobile phones and 90 percent of its PCs. To pull off a seeding attack would mean that the perpetrators would have to develop a deep understanding of a product’s design, then be able to manipulate components at the factory, and finally be able to ensure that the doctored devices would make it through the global logistics chain to the desired destination. All of which is as difficult as it sounds.
Yet, that is exactly what was found here. The chips had been inserted during the manufacturing process, by operatives from a unit of the People’s Liberation Army and their goal was clearly to gain long-term access to high-value corporate secrets and sensitive government networks. Which they have done. The chips were designed to be as inconspicuous as possible, appearing more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment. Depending on the board model, the chips varied slightly in size, suggesting that the attackers had supplied different factories with different batches. Many different factories with many different batches.
The primary role of the chips was to manipulate the core operating instructions that tell the server what to do as data moves across a motherboard, and at a crucial moment, allow it to effectively edit the information queue, inject its own code, or alter the order of the instructions the CPU was meant to follow.
The chips were then able to tell the device to communicate with one of several anonymous computers elsewhere on the Internet that were loaded with advanced code, and to prepare the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
This system could then let the attackers alter how the device functioned, line by line, however, they wanted, leaving no one the wiser. As one example, consider the Linux operating system where code resides that validates a user by verifying a typed password against a stored encrypted one. An implanted chip can alter part of that code, so the server won’t check for a password anymore, and voila, the secure machine is open to all users, authorized or otherwise. That same chip can also steal encryption keys for secure communications, block security updates that would neutralize the attack, and open new pathways to the Internet.
So far, the investigation has concluded that over 30 companies have been affected, including a major bank, large government contractors, and one of the world’s most valuable companies, Apple. Apple has been one of the top Supermicro customers and had planned to order an additional 30,000 of its servers for its new global network of data centers. But in the summer of 2015, Apple conducted their own investigation and discovered the rogue chips which led to an immediate cancellation of that order and severance of the business relationship. The extent to which Apple’s existing servers are affected and how those networks have been compromised is unclear. Apple’s official statement is that no networks or servers have been compromised and nothing is at risk.
In 2009, Elemental announced a development partnership with In-Q-Tel Inc., the CIA’s investment arm, along with a deal that enabled Elemental servers to be used in national security missions across the U.S. government, including the Department of Defense data centers, to process drone and surveillance-camera footage, on Navy warships to transmit feeds of airborne missions, and inside government buildings to enable secure videoconferencing. NASA, both houses of Congress, and the Department of Homeland Security have also been and are still, customers.
Supermicro, founded in 1993 by Charles Liang, a Taiwanese engineer was an early beneficiary of the contract manufacturing outsourcing trend in the Valley and began by providing engineering design services along with access to supply chain factories in Taiwan and China. Today, Supermicro sells more server motherboards than almost anyone else on the planet. It also dominates the $1 billion market for boards used in special-purpose computers, from MRI machines to weapons systems. Its motherboards can be found in made-to-order servers for banks, hedge funds, cloud computing providers, and web-hosting services, throughout the world.
Its core product, the motherboards, are all manufactured by contractors in China.
Hundreds of full-time Taiwanese or Chinese engineers with overseas ties work at the San Jose facility where Mandarin is the chosen spoken language supplemented by the written Hanzi which makes it easier for visiting Chinese partners to gain an understanding of their operations and designs. The current investigation is still trying to determine whether active spies were planted inside Supermicro and/or any of its suppliers.
The easiest way to imagine Supermicro is to think of them as the Microsoft of the hardware world, with more than 900 customers in 100 countries by 2015, Supermicro potentially offers inroads to a massive collection of sensitive targets. Attacking their motherboards is like attacking Windows.
The reality is in fact that the security of the entire global technology supply chain has been compromised, though consumers and most companies don’t know it yet
Way back in 2014, U.S. intelligence officials had specific and concrete evidence that China’s military was preparing to insert these rogue chips into Supermicro motherboards bound for U.S. companies. But, issuing a broad warning to Supermicro’s customers could have crippled the company, (still a major American hardware maker), and it wasn’t clear from the intelligence exactly whom the operation was targeting or what its ultimate aims were. Plus, without confirmation that anyone had been attacked, the FBI was limited in how it could respond. So, we did nothing.
At last count, Supermicro had three primary manufacturers constructing its motherboards, two headquartered in Taiwan, one in Shanghai, and subcontract manufacturers scattered throughout China. Four of those subcontractors are now known to have been building out Supermicro motherboards for at least the last two years. Those subs operate the way all subs operate in China. Plant managers at the subs are approached by people who claim to represent Supermicro or who hold positions connected to the Chinese government. These middlemen would request changes to the motherboards’ original designs, offering bribes where they encountered resistance or suspicion and even threatening factory managers with inspections that would surely shut down their plants. Once these “arrangements” were finalized, they would organize the rest of the supply chain back to Supermicro.
And we know all of this, yet we happily do business with them anyway.
Ignoring China’s Ministry of Foreign Affairs’ ridiculous statement about China being “a resolute defender of cybersecurity”, we should congratulate instead the Trump administration’s recent decision to pull all off-shore manufacturing back to the States. China comically wants the world to believe that their 2011 proposed international guarantees on hardware security were somehow meaningful and that their plea to “parties” making “gratuitous accusations and suspicions” must end, so that we can all Kumbaya and “conduct more constructive talk and collaboration together in building a peaceful, safe, open, cooperative and orderly cyberspace.” Right.
As far as Amazon is concerned, their security team, conducting its own investigation into AWS’s Beijing facilities found altered motherboards throughout those servers, including more sophisticated designs than they’d previously seen. In one case, the rogue chips were thin enough so that they’d been able to be embedded between the layers of fiberglass onto which the other components were attached.
While China has long been known to monitor banks, manufacturers, and ordinary citizens on its own soil, and the main customers of AWS’s China cloud were domestic companies or foreign entities with operations in China, the fact that the country was conducting those operations inside Amazon’s cloud presented a Gordian knot. The dilemma was that it would be difficult to remove the equipment without alerting interested parties and that, even if they could devise a way, doing so would certainly alert the attackers that the chips had been discovered. So, they instead developed a method of monitoring the chips and in so doing, could not identify any traffic among the entry points and the attackers. Which only told them that the bad guys were saving the chips to mount a delayed attack or that they’d infiltrated other parts of the network before they started monitoring, neither of which was particularly reassuring. The point is that this is all going on right now as the world ponders what to do about this invasion.
When two of its major customers, Apple and Amazon bailed out on Supermicro in 2016, its CEO, Liang vaguely blamed it on pricing with the statement, “When customers asked for a lower price, our people did not respond quickly enough.” Whether that sort of dissembling raises the question of complicity, I will leave it to you, but it seems to me that Supermicro is making it harder to take seriously the company’s case for innocence.
Because concurrent with the malicious chips’ discovery in 2015 and the investigation unfolding around it, Supermicro has suspiciously missed two deadlines to file quarterly and annual reports required by regulators and was delisted from the Nasdaq this summer. This seems like extraordinary behavior for a company growing at the breathtaking rate of 100% over the last four years ($871 million in the second quarter of fiscal year 2020).
In 2015, then-President Obama and Chinese President Xi Jinping appeared together at the White House for a joint press conference touting a landmark deal on cybersecurity. China had agreed that it would no longer support the theft by hackers of U.S. intellectual property to benefit Chinese companies. But anyone who had been paying attention, including senior White House officials knew that China was happy to offer this concession because it had already developed advanced forms of cyber-theft supported by its monopoly of the Asian technology supply chain.
U.S. government officials and U.S. technology executives subsequently met in a Pentagon-sponsored event to determine ways in which the U.S. could develop commercial products that could detect hardware implants. Everyone present knew that the hardware on the topic was being produced by Supermicro. Why, in the name of national security did we not simply shut the company down?
I am acutely aware that it is unbelievably difficult to make decisions and execute any sort of rational-sounding plan within the labyrinth we call the Federal government. There are hundreds of thousands of people in roles where they have been since the last several administrations, continuing to do the work that they are chartered to do regardless of who is in charge now or what their political agenda happens to be. There are tons of moving parts and hundreds of shadow agencies and agreements dependent on any directional movement, one way or the other.
In short, it is not easy to get “easy” things done.
Decades ago, we had made the decision to send advanced production work to Southeast Asia and low-cost Chinese manufacturing had come to underpin the business models of many of America’s largest technology companies. Apple, for instance, made many of its most sophisticated electronics domestically up until 1992 when it closed its state-of-the-art plant for motherboard and computer assembly in Fremont, Calif., and sent that work to China.
The assumption that China would surely not do anything like spying in its factories to jeopardize its position as the workshop to the world became the guiding principle for continuing the folly. We seem to be intoxicated with this classic devil’s bargain as our somatic capitalism continues to opt for the supply chain we desire, even with all of the known global risks of accepting a lower-capacity supply chain with greater security. Well, COVID-19 just declared that this dog no longer hunts.
The cheap chip ship has finally sailed, though it leaves behind more than 80% of the world’s servers infected with an incurable cyber disease.
What we do about this next matters more than anything we have done in the past.