DevOps Blog

Stay in the know with in-depth articles about DevOps, micro-services and cloud native topics, delivered to you weekly.

DevOps Security: Secrets from the Trenches

Written by Marius Rimkus
on February 18, 2020

The underlying principles of DevOps have transformed how organizations develop, operate, and maintain IT infrastructure and applications. DevOps united two worlds that were historically independent—software development and IT operations, and introduced new ways of defining requirements and specifications, new development models, new testing practices, a strong focus on automation and customized development tooling.

 

DevOps created a culture of openness and collaboration, but the DevOps model also raises security issues, which are frequently left out of the collaborative process. Early DevOps teams did not include security staff, and security remained a separate silo and a separate stage of the development cycle, added at the end as an afterthought. Read on to learn how to solve DevOps security challenges.

 

DevOps Security Challenges

Below, you’ll find a review of the most current and crucial DevOps security challenges. Keep these in mind when you assess your security posture.

DevOps and Cloud Environments

DevOps teams commonly rely on cloud deployment for automation and agility, which raises concerns about cloud security. They run a large number of servers and containers which are difficult to monitor and secure. Many DevOps cloud environments operate at such a large scale and such a fast pace, that even a small oversight in security or configuration can lead to catastrophic production issues.

Reliance on Open Source

DevOps teams often use many open source components, both in their tooling and as part of the software they develop, which are not necessarily vetted for security issues or tested for vulnerabilities. Even if a component was valid when it was chosen for a project, it will quickly be reused across multiple projects and teams, and teams may be unaware that it has become out of date or a new vulnerability has been discovered.

Cultural Resistance to Security

DevOps moves fast, and in the past many practitioners felt security only holds back their velocity. In reality, the effort required to recognize a security vulnerability early in the development process and fix it is much lower than trying to fix it late in the development cycle, or worse, after it has already reached production.

Security Teams Left Behind

Traditionally, security teams required weeks or months to fully review code and configuration for vulnerabilities and security issues. However, DevOps teams deploy new code over a period of days or hours. The same security teams find it difficult to receive a release candidate and fully analyze them for security issues at this rapid pace.

 

DevOps Secrets From the Trenches

Here are a few secrets we learned from seasoned DevOps practitioners that can help you solve some of the challenges of DevOps security, and make it work in your organization.

Add Security into the Agile Planning Process

Make sure your user stories are sized properly, and that security teams are able to process a story and provide security insights fast enough. Security tasks should be an inseparable part of sprint planning. Define acceptance criteria that include security, to avoid pushing components with security issues out to production. Ensure that security, like testing, is part of the definition of done. 

Manage All Software Artifacts in Source Control

Ensure you have a version control system that covers all aspects of your applications—not just source code but also schemas, UI, access control definitions and static resources. Developers must always make changes in a separate environment, and security staff should be able to immediately see, across all types of software artifacts, what has changed from version to version.

Lock Down Microservices

If you are using a microservices architecture, think of security from the get go. Shift security left, and design a holistic security plan that defines how new microservices the team builds should be secured. The burden on teams should be shifted from fixing security issues in existing microservices, to complying with the initial security requirements for new microservices. 

 

Leverage authorization protocols like OAuth to restrict access in a predictable, centralized way. Consider using a distributed firewall packaged which each microservice, to get granular control over internal and external network connections. Adopt monitoring solutions like Prometheus which provide visibility in a containerized environment.

Code Quality

Define a standard coding style and enforce its use, ensuring that all developers on the team use the same code quality best practices. Take the following measures to enforce code quality:

 

  • Create unit tests with coverage of at least 80%, to ensure you are checking for basic coding errors and functional gaps
  • Leverage static and dynamic code analysis to catch problems with code and security issues developers may have missed
  • After automated tests pass, use peer review to ensure the most experienced members of your team see important changes and suggest opportunities for optimization

 

Taking these measures will ensure that when code reaches your security experts, it will already be in good shape, allowing them to focus on non-trivial security issues that automated tools and non-security experts are likely to miss.

Add Strict Quality Gates to Your Pipeline

Do everything you can to catch changes to code that might represent a quality or security issue, and prevent them from moving into the next stage of your CD pipeline. Define a branch for each phase of development, and make incremental adjustments, identifying merge conflicts via the version control system.

 

Changes should be released using strict quality gates. Promotion to more advanced stages of the dev/test/production pipeline should be automated, but add a manual review gate for sensitive changes. Ensure that anything that has a critical security impact is reviewed by security or testing staff, who can push a button to continue the pipeline’s automatic process.

Pay Attention to Container Security

Containers are the achilles heel of many DevOps environments. Use the following best practices to ensure your containers are secure:

 

  • Use trusted, signed base images—remember that container images contain software that may be exposed to vulnerabilities, and those vulnerabilities carry over to every container created from the image.
  • Scan images for vulnerabilities—even trusted images can have new or unknown vulnerabilities. Use one of the many free tools available to ensure you never deploy a container before running a security scan.
  • Never provide root access on a container—root access on containers gives an attacker root access on the Docker host, and may allow them to compromise an entire Kubernetes cluster.
  • Use secrets correctly—work closely with security staff to ensure you only store sensitive data like user credentials using Kubernetes secrets, in a way that is not exposed to attackers.
  • Monitor containers for anomalous behavior—you should have tools in place to monitor containers at all times, not only in the production environment but also in dev and test, for behavior that might represent a cyber attack.

Conclusion

DevOps prioritizes fast product release over everything. However, that does not mean DevOps teams should give up on cybersecurity altogether. If you add security into your agile planning process, you’ll be able to integrate security without losing speed. 

You can also introduce security automation into the pipeline. You can utilize testing tools, as well as automated policies and response. Hopefully, these tips can help you secure your apps while still retaining a fast rate of releases.