In Day 14, we learned what container image scanning is and why it's important. We also learned about tools like Grype and Trivy that help us scan our container images.
However, in modern SDLCs, a DevSecOps engineer would rarely scan container images by hand, e.g., they would not be running Grype and Trivy locally and looking at every single vulnerability. Instead, they would have the image scanning configured as part of the CI/CD pipeline. This way, they would be sure that all the images that are being built by the pipelines are also scanned by the image scanner. These results could then be sent by another system, where the DevSecOps engineers could look at them and take some action depending on the result.
A sample CI/CD pipeline could look like this:
- Developer pushes code
- Lint the code
- Build the code
- Test the code
- Build the artifacts (container images, helm charts, etc.)
- Scan the artifacts
- (Optional) Send the scan results somewhere
- (Optional) Verify the scan results and fail the pipeline if the verification fails
- Push the artifacts to a repository
A failure in the scan or verify steps (steps 6 and 7) would mean that our container will not be pushed to our repository, and we cannot use the code we submitted.
Today, we are going to take a look at how we can set up such a pipeline and what would be a sensible configuration for one.
Let's take a look at the Grype scanner. Grype is an open-source scanner maintained by the company Anchore.
Scanning a container image with Grype is as simple as running:
grype <IMAGE>For example, if we want to scan the ubuntu:20.04 image, we can run:
$ grype ubuntu:20.04
✔ Vulnerability DB [no update available]
✔ Pulled image
✔ Loaded image
✔ Parsed image
✔ Cataloged packages [92 packages]
✔ Scanned image [19 vulnerabilities]
NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY
coreutils 8.30-3ubuntu2 deb CVE-2016-2781 Low
gpgv 2.2.19-3ubuntu2.2 deb CVE-2022-3219 Low
libc-bin 2.31-0ubuntu9.9 deb CVE-2016-20013 Negligible
libc6 2.31-0ubuntu9.9 deb CVE-2016-20013 Negligible
libncurses6 6.2-0ubuntu2 deb CVE-2021-39537 Negligible
libncurses6 6.2-0ubuntu2 deb CVE-2022-29458 Negligible
libncursesw6 6.2-0ubuntu2 deb CVE-2021-39537 Negligible
libncursesw6 6.2-0ubuntu2 deb CVE-2022-29458 Negligible
libpcre3 2:8.39-12ubuntu0.1 deb CVE-2017-11164 Negligible
libsystemd0 245.4-4ubuntu3.19 deb CVE-2022-3821 Medium
libtinfo6 6.2-0ubuntu2 deb CVE-2021-39537 Negligible
libtinfo6 6.2-0ubuntu2 deb CVE-2022-29458 Negligible
libudev1 245.4-4ubuntu3.19 deb CVE-2022-3821 Medium
login 1:4.8.1-1ubuntu5.20.04.4 deb CVE-2013-4235 Low
ncurses-base 6.2-0ubuntu2 deb CVE-2021-39537 Negligible
ncurses-base 6.2-0ubuntu2 deb CVE-2022-29458 Negligible
ncurses-bin 6.2-0ubuntu2 deb CVE-2021-39537 Negligible
ncurses-bin 6.2-0ubuntu2 deb CVE-2022-29458 Negligible
passwd 1:4.8.1-1ubuntu5.20.04.4 deb CVE-2013-4235 LowOf course, you already know that because we did it on Day 14.
However, this command will only output the vulnerabilities and exit with a success code. So if this were in a CI/CD pipeline, the pipeline would be successful even if we have many vulnerabilities.
The person running the pipeline would have to open it, see the logs and manually determine whether the results are OK. This is tedious and error prone.
Let's see how we can enforce some rules for the results that come out of the scan.
As we already established, just scanning the image does not do much except for giving us visibility into the number of vulnerabilities we have inside the image. But what if we want to enforce a set of rules for our container images?
For example, a good rule would be "an image should not have critical vulnerabilities" or "an image should not have vulnerabilities with available fixes."
Fortunately for us, this is also something that Grype supports out of the box.
We can use the --fail-on <SEVERITY> flag to tell Grype to exit with a non-zero exit code if, during the scan, it found vulnerabilities with a severity higher or equal to the one we specified.
This will fail our pipeline, and the engineer would have to look at the results and fix something in order to make it pass.
Let's tried it out.
We are going to use the springio/petclinic:latest image, which we already found has many vulnerabilities.
You can go back to Day 14 or scan it yourself to see how much exactly.
We want to fail the pipeline if the image has CRITICAL vulnerabilities.
We are going to run the can like this:
$ grype springio/petclinic:latest --fail-on critical
✔ Vulnerability DB [no update available]
✔ Loaded image
✔ Parsed image
✔ Cataloged packages [212 packages]
✔ Scanned image [168 vulnerabilities]
NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY
spring-core 5.3.6 java-archive CVE-2016-1000027 Critical
spring-core 5.3.6 java-archive CVE-2022-22965 Critical
...
1 error occurred:
* discovered vulnerabilities at or above the severity threshold
$ echo $?
1We see two things here:
- apart from the results, Grype also outputted an error that is telling us that this scan violated the rule we had defined (no CRITICAL vulnerabilities)
- Grype exited with exit code 1, which indicates failure. If this were a CI pipeline, it would have failed.
When this happens, we will be blocked from merging our code and pushing our container to the registry. This means that we need to take some action to fix the failure so that we can finish our task and push our change.
Let's see what our options are.
Once we encounter a vulnerability that is preventing us from publishing our container, we have a few ways we can go depending on the vulnerability.
The best-case scenario is when this vulnerability is already fixed in a newer version of the library we depend on.
One such vulnerability is this one:
NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY
snakeyaml 1.27 1.31 java-archive GHSA-3mc7-4q67-w48m High
This is a High severity vulnerability.
It's coming from the Java package snakeyaml, version 1.27.
Grype is telling us that this vulnerability is fixed in version 1.31 of the same library.
In this case, we can just upgrade the version of this library in our pom.xml or build.gradle file,
test our code to make sure nothing breaks with the new version,
and submit the code again.
This will build a new version of our container, re-scan it, and hopefully, this time, the vulnerability will not come up, and our scan will be successful.
Sometimes a vulnerability we encounter will not have a fix available. These are so-called zero-day vulnerabilities that are disclosed before a fix is available.
We can see two of those in the initial scan results:
NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY
spring-core 5.3.6 java-archive CVE-2016-1000027 Critical
spring-core 5.3.6 java-archive CVE-2022-22965 Critical
When we encounter such a vulnerability, we need to evaluate how severe it is and calculate the risk of releasing our software with that vulnerability in it.
We can determine that the vulnerability does not constitute any danger to our software and its consumers. One such case might be when a vulnerability requires physical access to the servers to be exploited. If we are sure that our physical servers are secure enough and an attacker cannot get access to them, we can safely ignore this vulnerability.
In this case, we can tell Grype to ignore this vulnerability and not fail the scan because of it.
We can do this via the grype.yaml configuration file, where we can list vulnerabilities we want to ignore:
ignore:
# This is the full set of supported rule fields:
- vulnerability: CVE-2016-1000027
fix-state: unknown
package:
name: spring-core
version: 5.3.6
type: java-archive
# We can list as many of these as we want
- vulnerability: CVE-2022-22965
# Or list whole packages which we want to ignore
- package:
type: gemPutting this in our configuration file and re-running the scan will make our pipeline green.
However, it is crucial that we keep track of this file and not ignore vulnerabilities that have a fix. For example, when a fix for this vulnerability is released, it's best we upgrade our dependency and remove this vulnerability from our application.
That way, we will ensure that our application is as secure as possible and there are no vulnerabilities that can turn out to be more severe than we initially thought.
The worst-case scenario is if we encounter a vulnerability that does not have a fix, and it is indeed dangerous, and there is a possibility to be exploited.
In that case, there is no right move. The best thing we can do is sit down with our security team and come up with an action plan.
We might decide it's best to do nothing while the vulnerability is fixed. We might decide to manually patch some stuff so that we remove at least some part of the danger. It really depends on the situation.
Sometimes, a zero-day vulnerability is already in your application that is deployed. In that case, freezing deploys won't help because your app is already vulnerable.
That was the case with the Log4Shell vulnerability that was discovered in late 2021 but has been present in Log4j since 2013. Luckily, there was a fix available within hours, but next time we might not be this lucky.
As we already learned in Day 14, scanning your container images for vulnerabilities is important as it can give you valuable insights about the security posture of your images.
Today we learned that it's even better to have it as part of your CI/CD pipeline and to enforce some basic rules about what vulnerabilities you have inside your images.
Finally, we discussed the steps we can take when we find a vulnerability.
Tomorrow we are going to take a look at container registries that enable this scanning out of the box and also at scanning other types of artifacts. See you on Day 22.