Until this week, SolarWinds was a little known IT software group from Texas. Its deserted lobby has a framed magazine article from a few years ago when it was on a list of America’s “Best Small Companies”.
Now the Austin-based company is at the heart of one of the biggest and most startling cyber hacks in recent history, with ramifications that extend into the fields of geopolitics, espionage and national security.
For nine months, sophisticated state-backed hackers have exploited a ubiquitous SolarWinds software product in order to spy on government and business networks around the world, including in the US, UK, Israel and Canada. Wielding innovative tools and tradecraft, the cyber spies lurked in email services, and posed as legitimate staffers to tap confidential information stored in the cloud.
The bombshell revelations have sent 18,000 exposed SolarWinds customers scrambling to assess whether outsiders did indeed enter their systems, what the damage was and how to fix it.
The sprawling operation targeted some of the US government’s most sensitive data. The commerce and energy departments both admitted they had been compromised although the latter said it has no evidence of intrusions into its nuclear weapons management networks “so far”. Numerous other federal agencies have acknowledged that they are inspecting for fallout.
But the true scale of the ongoing campaign and its motivations are not yet — and may never be — known. There are indicators that it may be part of an even broader campaign that extends beyond the SolarWinds software. Experts have been swift to point a finger at Russia, which has wielded similar tactics in past cyber operations, though officials have refused to confirm a suspected culprit.
The massive hack has shone a light on the vulnerability of US government agencies and many of the world’s biggest companies to cyber intrusions via the long tail of vendors they rely on for IT services. SolarWinds is one of hundreds of relatively unknown companies that provide software to governments and business for their networks.
“This is the most consequential cyber espionage campaign to date,” says Dmitri Alperovitch, co-founder of security group CrowdStrike who now runs the Silverado Policy Accelerator think-tank.
“It is going to take months to ascertain the full impact and actually be successful at ejecting the adversaries,” he adds. “And there’s going to be phase 2 which is understanding how we have failed to understand that this intelligence operation was taking place . . . but also [to] figure out how we’re gonna rebuild our cyber security in government.”
A ‘silent cold war’
The adversaries first broke through their victims’ defences by injecting malicious code into the patches of SolarWinds’ Orion product between March and June of this year. This meant that as some 18,000 SolarWinds clients updated their software, they unwittingly introduced a hidden backdoor for attackers to come in.
Once inside, the hackers were able to move around at will, undetected, going to great lengths to cover their tracks and identity.
John Hultquist, director of intelligence analysis at FireEye, the cyber security company which was itself a casualty of the attack, says the perpetrators painstakingly “compartmentalised” their actions, making it harder to connect one intrusion to another. The hackers did not want to exploit every opportunity for fear of raising suspicion. “This is about quality over quantity. Every organisation they access endangers their access — which risks the entire operation,” he says.
One western security official says there is already evidence that the hackers conducted detailed reconnaissance on the organisations they had breached, and depending on what they found, would then decide which victims to prioritise. Microsoft, also a victim of the hackers, said on Thursday that it had identified 40 customers that had been “targeted more precisely and compromised through additional and sophisticated measures”, largely IT and security companies as well as government agencies.
Michael Chertoff, chairman of the Chertoff Group, a security and risk management consultancy, who served as secretary of homeland security in the Bush administration, says that “our adversaries’ hacking skills have also gotten better and they have become more aggressive.” He adds: “There is a bit of a silent cold war in the cyber space domain.”
The hackers leveraged other novel techniques to impersonate trusted users and access highly sensitive information, according to a rare advisory published by the US National Security Agency on Thursday.
“If you have unfettered access, you can create your own administrator’s [control], user IDs and passwords and credentials that look like normal employees’,” says Theresa Payton, former White House chief information officer and chief executive of cyber security consultancy Fortalice Solutions, who dubs this level of access the “God’s door”.
“You can hijack dormant accounts, you can inject documents, you can change things.”
The Cybersecurity and Infrastructure Security Agency warned that the hackers also used other undisclosed “vectors” as part of their campaign, and that it will be “highly complex and challenging” for victims to actually eject the perpetrators from their systems.
“Can you imagine if you found out that six months ago somebody was in your house and now you’re trying to figure that out?” says Ms Payton. “The forensic evidence gets damaged and destroyed.”
“If it is [Russian foreign intelligence], they will not run away once detected,” says Suzanne Spaulding, security expert at the Center for Strategic and International Studies. “If you think they’re out of your system, they may have just gone deeper into hiding. They have in the past been combative — we may have a battle on our hands.”
US officials have been evasive when it comes to attributing the attacks. Only Richard Blumenthal, Democratic senator from Connecticut, has publicly singled out Russia as the main culprit, after he and other members of Congress received a classified briefing from intelligence officials.
“Today’s classified briefing on Russia’s cyber attack left me deeply alarmed, in fact downright scared,” Mr Blumenthal wrote on Twitter on Wednesday.
Many cyber experts believe the attack bears the hallmarks of a Russia-backed campaign.
One person who had been briefed on the investigation says there were clues buried in the hackers’ language and coding that pointed to Russian perpetrators.
Some have pointed specifically at APT29, a prolific hacker group backed by the SVR, Russia’s Foreign Intelligence Service, which has previously been linked to the theft of emails from the Democratic National Committee ahead of the 2016 US election. One person with knowledge of the hack suggested it could also be a sister unit to APT29.
Supply chain risk
The SolarWinds hack is the latest in a long line of increasingly advanced cyber attacks over a period of more than a decade since China first penetrated Pentagon and White House networks. Washington received a big wake-up call in 2015 after it discovered that China had obtained sensitive data on several million government employees by hacking the Office of Personnel Management.
But the severity of the SolarWinds attack and the wide net of victims have prompted soul-searching among the cyber security community, US government and corporations.
“The main implication for me is to underline the weakness of much of the west’s cyber defences and in that respect it’s a bit discouraging, morale-sapping, it’s frankly a bit embarrassing,” says Ciaran Martin, who stepped down earlier this year as head of the UK’s National Cyber Security Centre, the defensive arm of signals intelligence agency GCHQ, and now a professor at the University of Oxford’s Blavatnik School.
One key lesson from this attack, say cyber experts, is that defences among the majority of western institutions are simply not strong enough. In particular, organisations have not paid enough attention to the security of software suppliers — such as SolarWinds — in their supply chain.
Prof Martin says securing the supply chain is the “hardest nut to crack” because there is neither a globally-recognised set of software security standards, nor any form of enforcement if these are not met.
“If you’re the chief information security officer in a company or US government and you need to buy software how do you know what’s good?” asks Prof Martin. “We have to accelerate on the long hard road to fixing [our supply chain defences] and if this doesn’t prompt us to, I don’t know what will.”
Others apportion the blame in part on inaction from the government and weakness in their own systems. “I don’t think the security measures taken after the OPM hack were at all sufficient, or at all helpful,” says Mr Alperovitch. “We had spent literally hundreds of millions of dollars on systems that did nothing to protect us here.”
Thomas Bossert, former homeland security adviser to President Donald Trump and president of Trinity Cyber, a security consultancy, says the government needs better tools to carry out “deep inspection of network traffic” to detect suspicious activity.
Many questions remain unanswered. For example, there is no clarity on how SolarWinds, whose shares have fallen by more than 25 per cent since last Friday, was hacked in the first instance.
Dick Durbin, Democratic senator for Illinois, described the hack as “virtually a declaration of war by Russia on the United States” — a suggestion which has been widely shot down by cyber experts, who argue that hacking for espionage purposes is entirely different from an offensive cyber campaign which is intended to cause harm, for instance by targeting critical infrastructure.
US officials and cyber experts also privately admit that American spy agencies — most notably the NSA — are constantly engaged in exactly the same kind of hacking of overseas governments that they publicly rail against back in Washington.
James Lewis, a cyber security expert at the Center for Strategic and International Studies think-tank, argues that hacks have become inevitable and that it is critical for the US government to think more about how it could change the risk calculation in a way that makes Russia and China less likely to conduct attacks on the US. This should be a priority for the incoming Biden administration, he adds.
“We have to stop thinking of cyber as somehow unique. This is part of a larger conflict with Russia and China. We have two giant espionage campaigns aimed at the US. One [Russia] is looking for political effect, and the other [China] is looking to steal technology.”
He adds: “[But] we have no strategy or leadership. Every president has failed to deal with this.”
Many experts call for international accords around responses to global cyber attacks, as a preventive measure.
Google chief executive Sundar Pichai argues that governments need to draw up a cyber framework that is “the equivalent of internet disarmament”. He adds: “I’m not saying it’s going to be easy, but it has to be on the agenda of the G20, given how important digital infrastructure is becoming.”
Additional reporting by Miles Kruppa in Texas and Richard Waters in San Francisco