In recent years, there’s been a number of attacks targeting the software supply chain, which is made up of the code, people, systems and processes in the software development lifecycle. The SolarWinds hack of 2020 was one of the most notorious cyber attacks ever since it was highly evasive, innovative, and targeted U.S. federal agencies. To deal with software supply chain security, the U.S. government responded with section 4 of Executive Order 14028. After that, there have been a few additional efforts, namely:
- NIST’s SSDF
- Google and OpenSSF’s SLSA framework
- CIS’s Software Supply Chain Security Guide
- CISA/NSA’s Security the Software Supply Chain: Recommended Practices
- Code to Cloud and CBDR from Palo Alto Networks
At Palo Alto Networks IT, we also began a journey to harden our CICD pipeline all the way from our developers, dependencies, various scans, to CSPM in our cloud and runtime monitoring in the workload to stronger protection against hackers.
In this blog, I would like to share two recent projects to give you an idea of how to monitor different aspects of many Git repositories with Prisma Cloud. This approach gives us a way to see the vulnerability risks in our repositories, allows us to block builds and cosign/check images using Prisma Cloud, and as a result, improve our software supply chain security.
In my previous post, we talked about our OPA Notary integration. Today, I’m going to switch the software to Cosign instead of Notary since Cosign is much easier to work with.
CICD Pipeline at Palo Alto Networks IT
We have 100+ code repositories in our GitHub Enterprise for microservices. Figure 1 shows a general view of the pipeline. Here are the highlights:
- We authenticate our developers to GitHub by using Okta and SSH keys. This is much more secure than relying only on a pair of username/password or a SSH key with a passphrase that can be brute forced.
- Currently, GitGuardian is used to scan secrets in our repos. In the future, we will look at our Prisma Cloud Code Security to scan secrets as well.
- Drone CI (along with CircleCI, GitLab CI, etc.) is one of the popular choices when you are addressing the ephemeral environment aspect of the build pipeline. In fact, the famous SolarWinds SUNBURST was injected by the SUNSPOTS malware that persisted in the build servers for months and the malware was monitoring msbuild.exe process during the builds.
- We are using multiple static code analysis tools complementing each other to harden our code base.
- We have secure base container images (e.g., JDK, node, Python) from InfoSec to build our microservices images.
- Our container image scanner is provided by Prisma Cloud.
- We sign our images using Cosign with a private key.
To elaborate on highlight #6, Prisma Cloud Container image scanner is very flexible. You can have different rules for different sets of the images. You can fail the builds at different CVEs severity threshold, add a grace period condition, check to see if the CVE is fixable by vendors, and allowlist individual CVEs with IDs.
Projects!
Cloud native deployment with Kubernetes and container images is also our official way to deploy services. Therefore, we prioritized a few projects like reducing the number of CVEs of the container images and cosign image signing.
Git Repository Dashboard
With hundreds of Git repositories, it’s important to know how many Git repositories (from GitHub Enterprise) and how many critical CVEs we need to address in the corresponding repositories. While you have all the information from GitHub Enterprise and Prisma Cloud, there is no bridge between these two systems in a single view. That’s why we developed a dashboard to show the CVEs number with each repository – see example below:
All the data is coming from the GitHub Enterprise and Prisma Cloud APIs. KPIs are being shown at the top of the dashboard for a quick update. The last column is also interesting as you can flip the button to enable/disable the build failure if there’s a critical CVE found in the corresponding container image during the builds. This all goes back to the flexible Prisma Cloud container image scanner configuration and an API available to change the scope of the rule dynamically.
Container Image Signing with Cosign
From my last blog post, you should be familiar with the number of phrases the Kubernetes API requests need to go through. However, notary v1 was challenging to operationalize since you need to maintain the service and the backend database. Notary v2 (notation) is still in beta and it’s pretty similar to what the other more mature software Cosign can do.
I think part of Cosign’s popularity comes from:
Starting from 1.24, Kubernetes release artifacts are signed using cosign signatures, i.e., Distroless base images are signed by cosign.
Also, cosign signatures are co-located with the container images that provide benefits such as:
- You do not need to maintain another service, database and its backup.
- As long as you have the authentication and authorization to pull/push the image, you already have the same permissions to pull/push the cosign signature from/to the same location.
The architecture of using Cosign is very similar to the notary one. In order to reach the “cosign verify” step, we need to run a k8s cosign wrapper for the OPA to send requests to.
A few months ago, we open sourced our k8s-cosign-wrapper while helping our customers. It’s a simple starting point if you need to write your own cosign wrapper to work with OPA. For example, sometimes you have to rebuild your wrapper image using certain base images in your environment or have to use the latest cosign binary. Another case is you may want to add certain features like:
- mTLS between the defenders/OPA gatekeeper and the cosign wrapper
- An UI to show the stats in the wrapper
- Multiple public keys support
It’s Just the Beginning…
Supply chain security is a big deal that is affecting many areas of the CICD pipeline. Security teams should identify and focus on the important components in a supply chain – in a practical way. For example, the build process must be fully automated to reduce human interaction. Anything related to people needs to have an MFA to authenticate, and their work must be reviewed by multiple reviewers. Source code, build and deployment servers need to be well secured too.
I hope this post has provided you with some useful insights on software supply chain operationalization at scale. As we look ahead, there will be new cybersecurity threats to identify, prevent or contain. At Palo Alto Networks IT, we will soon be doing more Software Composition Analysis (SCA) and secret scanning with Prisma Cloud code security and the capabilities provided by Cider Security to take on new challenges.
