Inside the Shadowserver crisis

This week hasn’t been the best for the Shadowserver Foundation. The nonprofit is fighting for its life after its main US sponsor pulled the plug. How did we get here, what does this mean for the internet, and what’s next?

Shadowserver began in 2004 as a purely voluntary initiative started by Nicholas Albright. Furious to discover that cybercriminals had infected his recently-deceased father’s computer with botnet malware, he worked with ISPs to shut the criminal network down, and Shadowserver was born.

Now, it’s a multinational non-profit with full-time staff doing the same thing that Albright did when he first began – gathering information and delivering it to organizations that can make a difference. That includes 107 national CERTs in 136 countries, and over 4,600 network owners ranging from ISPs to hosting companies, universities, and banks.

It’s involved in an estimated nine in ten takedowns. Sometimes it’s an active participant, sinkholing domains, finding infected machines, and analyzing malware. Sometimes, it harbours the internet’s toxic waste. It runs a system called the registrar of last resort (ROLR), which registers known malicious domains so that their original criminal owners can’t get at them. Most recently, when Microsoft led the takedown of the infamous Necurs botnet, ROLR took millions of criminal domains registered randomly by the domain’s algorithm.

Now, Shadowserver faces an existential threat after Cisco, which is its largest US sponsor, pulled its funding. This is a big deal, because Cisco contributes 95% of Shadowserver’s US money.

In an announcement asking for public support, Shadowserver warned that it lost four in seven donated full-time US staff – all donated by Cisco – that handled systems administration and development. It will lose the other three on May 26. That’s when it also has to move its US data center infrastructure to another location. That’s going to cost US$400,000, which it must raise by May 15.

One of those Cisco employees is Richard Perlotto, the director of Shadowserver, whom Cisco has laid off along with the rest of his team. This week, he has been hitting the phones at 7am each morning, trying to raise funds to keep Shadowserver alive.

“I don’t have to have the money in my pocket tomorrow, but I need a commitment that someone’s going to give me the money in May. So I can sign those contract,” he told us. “we have hundreds of terabytes of memory, we have 10s of thousands of CPU cores, and petabytes of data.” Around 12 petabytes, in fact.

Couldn’t a Google or a Microsoft step in to help by donating datacenter space? It isn’t that easy, Perlotto says. Its existing infrastructure runs on bare-metal hardware. It isn’t designed for cloud environments, and migrating is the kind of intensive redesign project that takes well-funded companies months of effort. He also says that a cloud environment would be far more expensive than a colocation arrangement. He needs another colo, and a big truck for all that data. He also needs to re-employ at least four team members to keep the whole thing running.

That’s only the beginning. It’ll cost $1.7m to run the nonprofit’s US operations this year, excluding the data center move. That’s assuming the organization doesn’t upgrade any infrastructure or replace any of the drives holding its 1.2 petabytes of data.

Why would Cisco pull its support like that? The company isn’t saying much, other than this boilerplate comment:

“Cisco supports the evolution of Shadowserver to an industry alliance enabling many organizations to contribute and grow the capabilities of this important organization. Cisco is proud of its long history as a Shadowserver supporter and will explore future involvement as the alliance takes shape.”

However, there’s a smoking gun: the corporate restructuring that the company announced this week. David Goeckeler, EVP and general manager of Cisco’s network and security business group, is gone. It also merged its security and applications groups and is seeking a new leader for the combined group. The company told Shadowserver that it would pull the plug back in September, explains Perlotto, but he couldn’t reveal his crisis until Cisco officially announced its own news. That’s partly what’s left him scrabbling for short-term support.

Perlotto doesn’t blame Cisco at all.

“Cisco done a great job,” he says. “Everything we do while we provide our services for free, there’s still a cost associated with it. And Cisco’s been paying that bill for the whole internet for 15 years”

In fact, he blames himself.

“Because the funding was there, it just wasn’t a worry,” he explains. “We said we’ll think about that next year.” Next year never came. There was always another takedown to concentrate on. Another malware strain to unpack.

When we spoke to him, three days after Shadowserver sent out its distress call, roughly 300individuals had already donated around US$14,000. It’s a strong show of love for Shadowserver, but it needs to add some zeros to that number. That will have to come from institutional support. Perlotto has launched the Shadowserver Alliance, which is an attempt to get funding from a broader base of companies.

This raises a couple of connected questions for us. The first is how we stop this kind of situation happening again, where a small nonprofit becomes vital to the health of the internet without evolving its governance at the same time. This creates unacceptable single points of failure.

Perlotto will be addressing that problem by bringing in board members who “aren’t Shadowserver people”. However, prevention is better than cure, and something this important to the internet shouldn’t be scrabbling around for cash. How can we ensure that other nonprofit linchpins for internet security don’t suffer the same fate?

The second question is related: how much support should the government be providing here? At the moment, public sector funding is minimal and comes entirely from the EU for Shadowserver’s operations on that side of the Atlantic. “We have a small amount of EU funding from a few of the National CERTs for project work as well as the Horizon 2020 funding, again for projects,” Perlotto says. “None of that pays for the daily operations of Shadowserver.”

The EU funding is locked into that region. The US can’t use any of it, and yet it’s the US that runs all the tools, handles all the data, and operates the servers. There is no Shadowserver data center in the EU. Without the US operation, there is no Shadowserver.

The internet is a part of the critical national infrastructure, and nowhere is that more clear than during a health crisis like the one we face now, where people rely on it for critical information and services while they self-isolate. The US government refers to the internet explicitly in its list of CNI sectors, which include the communications and IT industries.

Government support hasn’t been a priority thus far, explained Perlotto, but that might be about to change. “We have been acting as the CDC for the Internet for many years and just did not need to seriously look for outside funding, but obviously times have changed for us,” he said.

The problem now is that COVID-19 is the only issue on the US government’s mind, and it’s likely to stay that way for some time. The US government also thinks in 12-18-month budgetary cycles, he says – and it’s an election year.

Perlotto is optimistic that he’ll get the funding he needs. We hope so. Without it, disaster looms. ROLR controls some toxic domains perpetually, and its disappearance means they’d be released back into the wild. That means the creeps that used them as C2 domains could re-register them and revive slews of zombie machines. Just as no one wants to see the re-emergence of a vanquished disease, no one wants to see a dead botnet spring back to life.

If you’d like to donate to Shadowserver, you can find their blog post about the crisis here and there are some links at the bottom to help fund the data center move and join its Alliance.


Bookmark and Share