A security researcher who has previously exposed GSM and SS7 security issues has warned that telcos’ adoption of cloud-based operations for computing and network functions exposes them to a host of new security vulnerabilities.
Karsten Nohl of Security Research Labs, speaking to the May contain pirates conference, said his team of researchers had found ways to go from an initial penetration of a cloud instance, obtaining enough credentials to spy on user communications, extract user data, obtain the system administrator status and ultimately delete a network.
Although Nohl’s speech was titled as exposing Open RAN vulnerabilities, it wasn’t really, by Nohl’s own admission, really about telecommunications protocols, let alone Open RAN itself. There is nothing specific about the Open RAN architecture, protocols or interfaces its researchers called. Instead, Nohl focused much more on generic vulnerabilities that the increased use of cloud software and automated processes can cause in any environment and as such the vulnerabilities found would apply to other private cloud instances, he added.
However, with telcos increasingly adopting containerized functions deployed on Kubernetes and automating processes through APIs, the attack surface resulting from cloud adoption increases and needs to be addressed, Nohl said. Open RAN as it plays a role does so because it is another driver for the adoption of cloud function and automated processes. He also described a potential exploit that could target the Smart Radio Controller, through its cloud host platform.
Nohl described how his “red team” hackers had, over days and weeks, been able to find credentials to access part of the network’s cloud stack and then go from there to extend their position of control over the network.
The exploits showed that the security risks of moving to a cloud ops, CI/CD pipeline are social and technical.
One of them is the deployment of container configurations that do not separate physical resources between functional components. This means that if something that is considered less secure is hacked, the hacker can break out of that environment and influence something underneath (Kubernetes) or in other nearby containers.
“It turns out there are multiple avenues to perform this type of container evasion,” Nohl said.
Most commonly, this is a configuration that assigns a Privileged or sys_admin capability to a container that is not considered security-critical, but can then be exploited to access other containers or the system. Kubernetes host.
Another vector was to exploit access to the host_PID namespace that was set for containers and use it to kill processes on the host machine. “If you combine that with another capability, ptrace, you can also inject code into the process,” Nohl said. “It’s basically root-level access, where two seemingly benign things are combined for a complete exploit.”
A final hack was to use network access so that a guest could access the localhost. “At the very least, given a shared interface between guest and host, the guest can transfer everything from the localhost, unless it uses SSL. But Kubernetes clusters are built for everything in a cluster to be reliable, so admins assume they don’t need SSL: giving network-level access to a guest usually means the host is hacked.
All of these, Nohl said, are feats his team has seen in real-life evaluations, and aren’t just theoretical.
A second risk for telcos moving to cloud operations relates to the nature of the people and processes involved. There are simply “a lot more developers involved”, and with the increased use of automation and software pushed through CI/CD tools, some threats are magnified.
“Instead of five Linux sysadmins, you can now phish hundreds of people at various companies, all contributing code in some way and if you get to one of them, there’s a good change that finally allows you to influence a mobile network There is an ecosystem of software development tools that are now part of developing a network: someone commits something to Github, it’s packaged somewhere, imaged and the image deployed. If any part of this chain is hacked, you are at risk.”
Nohl described how his team found sensitive code posted in a developer query to Stackoverflow, old crackable passwords left detectable, old dev sites left online.
One trip was to find an old development website that was itself isolated from the production network. But it was running in a Docker container on Kubernetes, and those access capabilities were assigned to the container. “So we’re moving out of the container on Kubernetes, and now we’re not constrained.”
In this case, Nohl’s team “very slowly” started searching the internal network, finding hundreds of services and APIs connecting the network’s microservices. “Then if you send ‘bad stuff’ over an API, you get debug info back and the is a cred from one of the developers. This then allows us to access the data lake systems and there we find text messages from customers. It was a weeks-long hacking journey to do what took us a minute on 2G, but now we have an entire country’s text messages.
“It’s a journey and it doesn’t target telecommunications standards. What matters is that it’s a virtualized network with many bits of automation floating around.
A third hack saw researchers enter the field of telecommunications, specifically the RAN intelligent controller. “The RIC is optimization software in each of the hundreds of dockers in hundreds of Kubernetes and of course again they haven’t configured the Docker enough – so we can break into all of these K8 environments and bring down the network. You can see how many steps are involved – and again, none of this is telco specific,” he said.
Nohl said his team’s findings showed that security is still not being done “by design” within telecommunications teams. Patching and curing had to be done at an absolute minimum on a near-continuous basis. Netflix, he said, has a 72-hour self-destruct on its Docker containers, after which they are rebuilt from the CI/CD pipeline that contains all new patches built in.
He also urged hackers to take telecommunications security seriously as well, as networks migrate to the cloud. “We invite you to think about it. We rely on them [telco networks] and is it important to keep them safe.