WASHINGTON -- U.S. government agencies and private companies rushed Monday to secure their computer networks following the disclosure of a sophisticated and long-running cyber-espionage intrusion suspected of being carried out by Russian hackers.
The full extent of the damage is not yet clear. But the potential threat was significant enough the Department of Homeland Security's cybersecurity unit directed all federal agencies to remove compromised network management software and thousands of companies were expected to do the same.
What was striking about the operation was its potential scope as well as the manner in which the perpetrators managed to pierce cyber defenses and gain access to email and internal files at the Treasury and Commerce departments and potentially elsewhere.
The intrusion was stark evidence of the vulnerability of even supposedly secure government networks, even after well-known previous attacks.
"It's a reminder that offense is easier than defense and we still have a lot of work to do," said Suzanne Spaulding, a former U.S. cybersecurity official who is now a senior adviser at the Center for Strategic and International Studies.
The identity of the perpetrator remained unclear. A U.S. official, speaking on condition of anonymity because of an ongoing investigation, told The Associated Press on Monday Russian hackers are suspected.
The Washington Post, citing unnamed sources, said the attack was carried out by Russian government hackers who go by the nicknames APT29 or Cozy Bear and are part of that nation's foreign intelligence service.
The intrusion came to light after a prominent cybersecurity firm, FireEye, learned it had been breached and alerted foreign governments and major corporations were also compromised. The company did not say who it suspected, though many experts believed Russia was responsible given the level of skill involved.
U.S. authorities acknowledged federal agencies were affected by the breach on Sunday, providing few details. The Cybersecurity and Infrastructure Security Agency, known as CISA, said in an unusual directive that the widely used network software SolarWinds had been compromised and should be removed from any system using it.
The national cybersecurity agencies of Britain and Ireland issued similar alerts.
SolarWinds is used by hundreds of thousands of organizations around the world, including most Fortune 500 companies and multiple U.S. federal agencies. The perpetrators were able to embed malware in a security update issued by the company, based in Austin, Texas. Once inside, they could impersonate system administrators and have total access to the infected networks, experts said.
"Quite honestly, my heart sank when I saw some of the details, just the amount of information they could potentially have if they are reading everyone's emails and they are accessing sensitive files within places like Treasury or Commerce," said Ben Johnson, a former National Security Agency cyber-engineer who is now chief technology officer of software security firm Obsidian.
National Security Council spokesman John Ullyot said Monday the Trump administration was working with CISA, U.S. intelligence agencies, the FBI and government departments affected by the intrusion to coordinate a response.
"It's obviously incredibly significant and widespread," said Chris Painter, who coordinated cyber-policy at the State Department during the Obama administration. "How much was compromised? How much was exfiltrated? There are lots of open questions now."
Kremlin spokesman Dmitry Peskov said Monday Russia had "nothing to do with" the hack.
"Once again, I can reject these accusations," Peskov told reporters. "If for many months the Americans couldn't do anything about it, then, probably, one shouldn't unfoundedly blame the Russians for everything."
Federal agencies have long been attractive targets for foreign hackers looking to gain insight into American government personnel and policymaking.
Hackers linked to Russia, for instance, were able to break into the State Department's email system in 2014, infecting it so thoroughly it had to be cut off from the internet while experts worked to eliminate the infestation. A year later, a hack at the U.S. government's personnel office blamed on China compromised the personal information of some 22 million current, former and prospective federal employees, including highly sensitive data such as background investigations.
Cybersecurity experts said the nature and level of tradecraft involved in this latest effort suggest a foreign nation. Many have pointed out the goal of the months-long effort appeared to be espionage and not information that could be quickly used for profit or to simply inflict damage. Russia was the most likely culprit, though China and perhaps others are potential candidates.
In terms of scale alone, the operation seems similar to the Office of Personnel Management hack authorities suspect was carried out by the Chinese government, said Ben Buchanan, a Georgetown University cyber-espionage expert.
"The operational tradecraft -- how the hack was carried out -- seems to have been extremely good. These operators are experienced and capable, adept at finding a systemic weakness and then exploiting it quietly for months," said Buchanan, author of "The Hacker and The State."
Members of Congress were pressing the government for more information. "If reports are true and state-sponsored hackers successfully snuck malware-riddled software into scores of federal government systems, our country has suffered a massive national security failure that could have ramifications for years to come," said Sen. Ron Wyden, an Oregon Democrat who is a prominent voice on cyber issues.
If it was carried out by a foreign government, and the U.S. has the proof, then it becomes a question of what to do about it.
Some obvious options would include expelling diplomats of the offending country, imposing sanctions or filing criminal charges for cyber-espionage, steps Washington and the European Union have taken against Russia in the past.
"I'm sure that the departments like NSA and Cyber Command are coming up with options, that the Treasury Department is looking at sanction options, that the State Department is looking for how they will send a strong signal," Spaulding said. "Whether they will get approval for all these things from the White House remains to be seen."
In the meantime, SolarWinds and its many private-sector clients were working to close any breaches and repair the damage.
The company said in a financial filing it believed an unknown number of customers, though fewer than 18,000, had installed the compromised product update earlier this year. SolarWinds has said its customers include all five branches of the U.S. military, the Pentagon, the State Department, NASA, the National Security Agency, the Department of Justice and the White House, along with the top U.S. telecommunications and accounting firms, though it hasn't identified which of its customers were using the compromised product.
"We anticipate this will be a very large event when all the information comes to light," said John Hultquist, director of threat analysis at FireEye.
Connect with the Southeast Missourian Newsroom:
For corrections to this story or other insights for the editor, click here. To submit a letter to the editor, click here. To learn about the Southeast Missourian’s AI Policy, click here.