cyber-security-resources/SCOR/devops_devsecops.md

Integrating Development, Operations, and Security: A Guide to DevOps, CI/CD, and DevSecOps

DevOps: Bridging Development and Operations

DevOps is a set of practices that automates and integrates the processes between software development and IT operations teams. The goal is to shorten the system development lifecycle while delivering features, fixes, and updates frequently in close alignment with business objectives. DevOps encourages a culture of collaboration, where developers and operations teams work closely across the entire software application life cycle, from development and test through deployment to operations.

Key Principles of DevOps:

• Continuous Integration and Continuous Deployment (CI/CD): Automating the integration of code changes from multiple contributors and ensuring the software can be reliably released at any time.
• Automation: Automating repetitive tasks to increase efficiency, reduce errors, and allow team members to focus on more strategic work.
• Monitoring and Feedback: Implementing robust monitoring of applications and infrastructure to quickly identify and address issues.

CI/CD: The Backbone of Modern Software Deployment

CI/CD stands for Continuous Integration and Continuous Deployment, a cornerstone practice within DevOps that enables software development teams to focus on meeting business requirements, code quality, and security because deployments are automated.

Continuous Integration (CI):

CI involves automatically integrating code changes from multiple contributors into a main repository multiple times a day. This is followed by automated builds and tests, which ensures that the codebase is always in a deployable state, helping to detect and fix integration errors quickly.

Continuous Deployment (CD):

CD extends CI by automatically deploying all code changes to a testing or production environment after the build stage. This allows for faster feedback and more rapid release cycles, with the goal of providing a seamless, automated path to bring new features, configurations, and bug fixes to customers.

DevSecOps: Incorporating Security into the Mix

DevSecOps integrates security practices within the DevOps process. The aim is to bake security into the early stages of the software development lifecycle, making it a shared responsibility among developers, operations, and security teams. This approach is designed to eliminate silos between teams, encourage proactive security measures, and embed security controls and testing deeply into the CI/CD pipeline.

Key Aspects of DevSecOps:

• Security as Code: Implementing security practices as code to automate security policies, configurations, and testing throughout the development lifecycle.
• Early and Continuous Security: Integrating security tools and processes early in the software development cycle, allowing for continuous security assessments and feedback.
• Collaboration and Culture Shift: Fostering a culture where security is everyone's responsibility, promoting collaboration, and ensuring that security considerations are integrated into decision-making processes.

The Synergy of DevOps, CI/CD, and DevSecOps

Integrating DevOps, CI/CD, and DevSecOps creates a synergistic effect that enhances software quality, security, and delivery speed. This integration facilitates a collaborative environment where continuous feedback, automation, and early security measures are ingrained in the development process. As a result, organizations can achieve faster time to market, improved product quality, and a stronger security posture, all while maintaining alignment with business objectives.

• Faster Time to Market: By automating the deployment process and integrating security early on, organizations can reduce the time from development to deployment.
• Improved Quality and Reliability: Continuous testing and monitoring ensure that quality issues and vulnerabilities are identified and addressed early, leading to more reliable releases.
• Enhanced Security: Integrating security into the CI/CD pipeline enables proactive identification and mitigation of vulnerabilities, reducing the risk of security breaches.

In conclusion, the adoption of DevOps, CI/CD, and DevSecOps is transformative, enabling organizations to navigate the complexities of modern software development with agility and security. By fostering collaboration, automating processes, and integrating security from the outset, businesses can not only accelerate their digital transformation efforts but also ensure that their applications are secure, resilient, and aligned with their strategic goals.

Automation & Orchestration

Tools for automation, orchestration, deployment, provisioning and configuration management.

• Ansible - Simple IT automation platform that makes your applications and systems easier to deploy.
• Salt - Automate the management and configuration of any infrastructure or application at scale.
• Puppet - Unparalleled infrastructure automation and delivery.
• Chef - Automate infrastructure and applications.
• Juju - Simplifies how you configure, scale and operate today's complex software.
• Rundeck - Runbook Automation For Modernizing Your Operations.
• StackStorm - Connects all your apps, services, and workflows. Automate DevOps your way.
• Bosh - Release engineering, deployment, and lifecycle management of complex distributed systems.
• Cloudify - Connect, Control, & Automate from core to edge: unlimited locations, clouds and devices.
• Tsuru - An extensible and open source Platform as a Service software.
• Fabric - High level Python library designed to execute shell commands remotely over SSH.
• Capistrano - A remote server automation and deployment tool.
• Mina - Really fast deployer and server automation tool.
• Terraform - use Infrastructure as Code to provision and manage any cloud, infrastructure, or service.
• Pulumi - Modern infrastructure as code platform that allows you to use familiar programming languages and tools to build, deploy, and manage cloud infrastructure.
• Packer - Build Automated Machine Images.
• Vagrant - Development Environments Made Easy.
• Foreman - Complete lifecycle management tool for physical and virtual servers.
• Nomad - Deploy and Manage Any Containerized, Legacy, or Batch Application.
• Marathon - A production-grade container orchestration platform for DC/OS and Apache Mesos.
• OctoDNS - Managing DNS across multiple providers. DNS as code.
• ManageIQ - Manage containers, virtual machines, networks, and storage from a single platform.
• Ignite - Open Source Virtual Machine (VM) manager with a container UX and built-in GitOps management.
• Selefra - An open-source policy-as-code software that provides analytics for multi-cloud and SaaS.
• Spacelift - Flexible orchestration solution for IaC development.
• Atlantis - Terraform Pull Request Automation
• KubeVela - Modern application delivery platform that makes deploying and operating applications across today's hybrid, multi-cloud environments easier, faster and more reliable.
• Stacktape - Developer-friendly Infrastructure as a Code framework built on top of AWS.
• Score - Open Source developer-centric and platform-agnostic workload specification.
• Meshery - An open source, cloud native manager that enables the design and management of all Kubernetes-based infrastructure and applications.
• Digger - Open Source Infrastructure as Code management tool that runs within your CI/CD system.

Continuous Integration & Delivery

Continuous Integration, Continuous Delivery and Continuous Delivery. GitOps.

• On premises
  • Buildbot - automate all aspects of the software development cycle.
  • Gitlab CI - pipelines build, test, deploy, and monitor your code as part of a single, integrated workflow.
  • Jenkins - automation server for building, deploying and automating any project.
  • Drone - a Container-Native, Continuous Delivery Platform.
  • Concourse - pipeline-based continuous thing-doer.
  • Spinnaker - fast, safe, repeatable deployments for every Enterprise.
  • goCD - Delivery and Release Automation server.
  • Teamcity - enterprise-level CI and CD.
  • Bamboo - tie automated builds, tests, and releases together in a single workflow.
  • Integrity - Continuous Integration server.
  • Zuul - drives continuous integration, delivery, and deployment systems with a focus on project gating.
  • Argo - Open Source Kubernetes native workflows, events, CI and CD.
  • Strider - Continuous Deployment/Continuous Integration platform.
  • Evergreen - A Distributed Continuous Integration System from MongoDB.
  • werf - Open Source CI/CD tool for building Docker images & deploying them to Kubernetes using a GitOps approach.
  • Flux - automatically ensures that the state of your Kubernetes cluster matches the configuration you’ve supplied in Git.
  • Flagger - progressive delivery Kubernetes operator (Canary, A/B Testing and Blue/Green deployments).
  • Tekton - powerful and flexible open-source framework for creating CI/CD systems.
  • PipeCD - Continuous Delivery for Declarative Kubernetes, Serverless and Infrastructure Applications.
  • Gitploy - Build the deployment system around GitHub in minutes.
  • Dagger - CI/CD as Code that Runs Anywhere.
• Public Services
  • Travis CI - easily sync your projects, you’ll be testing your code in minutes.
  • Circle CI - powerful CI/CD pipelines that keep code moving.
  • Bitrise - CI/CD for mobile applications.
  • Buildkite - run fast, secure, and scalable continuous integration pipelines on your own infrastructure.
  • Cirrus CI - continuous integration system built for the era of cloud computing.
  • Codefresh - GitOps automation platform for Kubernetes apps.
  • Github actions - GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD.
  • Kraken CI - Modern CI/CD, open-source, on-premise system that is highly scalable and focused on testing.
  • Earthly - Develop CI/CD pipelines locally and run them anywhere.

Source Code Management

Source Code management, Git-repository manager, Version Control. Some of them are included in Code review section.

• GitHub - Helps developers store and manage their code, as well as track and control changes to their code.
• Gitlab - Entire DevOps lifecycle in one application.
• Bitbucket - Gives teams one place to plan projects, collaborate on code, test, and deploy
• Phabricator - A collection of web applications which help software companies build better software.
• Gogs - A painless self-hosted Git service.
• Gitea - A painless self-hosted Git service.
• Gitblit - Pure Java Git solution for managing, viewing, and serving Git repositories.
• RhodeCode - Centralized control for distributed repositories. Mercurial, Git, and Subversion under a single roof.
• Radicle - Radicle is a sovereign peer-to-peer network for code collaboration, built on top of Git.

Web Servers

Web servers and reverse proxy.

• Nginx - High performance load balancer, web server and reverse proxy.
• Apache - Web server and reverse proxy.
• Caddy - Web server with automatic HTTPS.
• Cherokee - Highly concurrent secured web applications.
• Lighttpd - Optimized for speed-critical environments while remaining standards-compliant, secure and flexible.
• Uwsgi - Application server container.

SSL

Tools for automating the management of SSL certificates.

• Certbot - Automate using Let’s Encrypt certificates on manually-managed websites to enable HTTPS.
• Let’s Encrypt - Free, automated, and open Certificate Authority.
• Cert Manager - K8S add-on to automate the management and issuance of TLS certificates from various issuing sources.

Databases

Relational (SQL) and non-relational (NoSQL) databases.

• Relational (SQL)
  • PostgreSQL - Powerful, open source object-relational database system.
  • MySQL - Open-source relational database management system.
  • MariaDB - Fast, scalable and robust, with a rich ecosystem of storage engines, plugins and many other tools.
  • SQLite - Small, fast, self-contained, high-reliability, full-featured, SQL database engine.
• Non-relational (NoSQL)
  • Cassandra - Manage massive amounts of data, fast, without losing sleep.
  • ScyllaDB - NoSQL data store using the seastar framework, compatible with Apache Cassandra
  • Apache HBase - Distributed, versioned, non-relational database.
  • Couchdb - Database that completely embraces the web.
  • Elasticsearch - Distributed, RESTful search and analytics engine capable of addressing a growing number of use cases.
  • MongoDB - General purpose, document-based, distributed database built for modern applications.
  • Rethinkdb - Open-source database for the realtime web.
  • Key-Value
    • Couchbase - Distributed multi-model NoSQL document-oriented database that is optimized for interactive applications.
    • Leveldb - Fast key-value storage library.
    • Redis - In-memory data structure store, used as a database, cache and message broker.
    • RocksDB - A library that provides an embeddable, persistent key-value store for fast storage.
    • Etcd - Distributed reliable key-value store for the most critical data of a distributed system.

Observability & Monitoring

Observability, Monitoring, Metrics/Metrics collection and Alerting tools.

• Steampipe - The universal SQL interface for any cloud API, & cloud intelligence dashboards extensible w/ HCL+SQL.
• Sensu - Simple. Scalable. Multi-cloud monitoring.
• Alerta - Scalable, minimal configuration and visualization monitoring system.
• Cabot - Self-hosted, easily-deployable monitoring and alerts service.
• Amon - Modern server monitoring platform.
• Icinga - Monitors availability and performance, gives you simple access to relevant data and raises alerts.
• Monit - Managing and monitoring Unix systems.
• Naemon - Fast, stable and innovative while giving you a clear view of the state of your network and applications.
• Nagios - Computer-software application that monitors systems, networks and infrastructure.
• Sentry - Error monitoring that helps all software teams discover, triage, and prioritize errors in real-time.
• Shinken - Monitoring framework.
• Zabbix - Mature and effortless monitoring solution for network monitoring and application monitoring.
• Glances - Monitoring information through a curses or Web based interface.
• Healthchecks - Cron monitoring tool.
• Bolo - Building distributed, scalable monitoring systems.
• cAdvisor - Analyzes resource usage and performance characteristics of running containers.
• ElastiFlow - Network flow monitoring (Netflow, sFlow and IPFIX) with the Elastic Stack.
• Co-Pilot - System performance analysis toolkit.
• Keep - Open source alerting CLI for developers.
• Globalping CLI - Run network commands like ping, traceroute and mtr from hundreds of global locations.
• Grai - Open source observability integrating data impact analysis into CI.
• Canary Checker - Open source health check platform.
• Metrics/Metrics collection
  • Thundra Foresight - Visibility into CI pipeline by spotting test failures in no time.
  • Prometheus - Power your metrics and alerting with a leading open-source monitoring solution.
  • Collectd - The system statistics collection daemon.
  • Facette - Time series data visualization software.
  • Grafana - Analytics & monitoring solution for every database.
  • Graphite - Store numeric time-series data and render graphs of this data on demand.
  • Influxdata - Time series database.
  • Netdata - Instantly diagnose slowdowns and anomalies in your infrastructure.
  • Freeboard - Real-time dashboard builder for IOT and other web mashups.
  • Autometrics - An open source micro framework for observability.
• Logs Management
  • Anthracite - An event/change logging/management app.
  • Graylog - Free and open source log management.
  • Logstash - Collect, parse, transform logs.
  • Fluentd - Data collector for unified logging layer.
  • Flume - Distributed, reliable, and available service for efficiently collecting, aggregating, and moving logs.
  • Heka - Stream processing software system.
  • Kibana - Explore, visualize, discover data.
  • Loki - Horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus.
• Status
  • Cachet - Beautiful and powerful open source status page system.
  • StatusPal - Communicate incidents and maintenance effectively with a beautiful hosted status page.
  • Instatus - Quick and beautiful status page.

Service Discovery & Service Mesh

Service Discovery, Service Mesh and Failure detection tools.

• Consul - Connect and secure any service.
• Serf - Decentralized cluster membership, failure detection, and orchestration.
• Doozerd - A consistent distributed data store.
• Zookeeper - Centralized service for configuration, naming, providing distributed synchronization, and more.
• Etcd - Distributed, reliable key-value store for the most critical data of a distributed system.
• Istio - Connect, secure, control, and observe services.
• Kong - Deliver performance needed for microservices, service mesh, and cloud native deployments.
• Linkerd - Service mesh for Kubernetes and beyond.

Chaos Engineering

The discipline of experimenting on a distributed system in order to build confidence in the system's capability to withstand turbulent conditions in production.

• Chaos Toolkit - The Open Source Platform for Chaos Engineering.
• Chaos Monkey - A resiliency tool that helps applications tolerate random instance failures.
• Toxiproxy - Simulate network and system conditions for chaos and resiliency testing.
• Pumba - Chaos testing, network emulation and stress testing tool for containers.
• Chaos Mesh - A Chaos Engineering Platform for Kubernetes.
• Litmus - Litmus enables teams to identify weaknesses in infrastructures.

API Gateway

API Gateway, Service Proxy and Service Management tools.

• API Umbrella - Proxy that sits in front of your APIs, API management platform.
• Ambassador - Kubernetes-Native API Gateway built on the Envoy Proxy.
• Kong - Connect all your microservices and APIs with the industry’s most performant, scalable and flexible API platform.
• Tyk - API and service management platform.
• Cilium - API aware networking and security using BPF and XDP.
• Gloo - Feature-rich, Kubernetes-native ingress controller, and next-generation API gateway.
• Envoy - Cloud-native high-performance edge/middle/service proxy.
• Traefik - Reverse proxy and load balancer for HTTP and TCP-based applications.

Code review

Code review. A few of the Source Code Management tools have built-in code review features.

• Gerrit - Web-based team code collaboration tool.
• Review Board - Web-based collaborative code review tool.
• MeshMap - World’s only visual designer for Kubernetes and cloud native applications. Design, deploy, and manage your Kubernetes-based, cloud native deployments allowing you to speed up infrastructure configuration.

Distributed messaging

Distributed messaging platforms and Queues software.

• Rabbitmq - Message broker.
• Kafka - Building real-time data pipelines and streaming apps.
• Activemq - Multi-Protocol messaging.
• Beanstalkd - Simple, fast work queue.
• NSQ - Realtime distributed messaging platform.
• Celery - Asynchronous task queue/job queue based on distributed message passing.
• Faktory - Repository for background jobs within your application.
• Nats - Simple, secure and high performance open source messaging system.
• RestMQ - Message queue which uses HTTP as transport.
• Dkron - Distributed, fault tolerant job scheduling system.
• KubeMQ - Kubernetes-native messaging platform.

Programming Languages

Programming languages.

• Python - Programming language that lets you work quickly and integrate systems more effectively.
• Ruby - A dynamic, open source programming language with a focus on simplicity and productivity.
• Go - An open source programming language that makes it easy to build simple, reliable, and efficient software.

Chat and ChatOps

Chat and ChatOps.

• Rocket - Open source team communication.
• Mattermost - Messaging platform that enables secure team collaboration.
• Zulip - Real-time chat with an email threading model.
• Riot - A universal secure chat app entirely under your control.
• ChatOps:
  • CloudBot - Simple, fast, expandable, open-source Python IRC Bot.
  • Hubot - A customizable life embetterment robot.

Secret Management

Security as code, sensitive credentials and secrets need to be managed, security, maintained and rotated using automation.

• Sops - Simple and flexible tool for managing secrets.
• Vault - Manage secrets and protect sensitive data.
• Keybase - End-to-end encrypted chat and cloud storage system.
• Vault Secrets Operator - Create Kubernetes secrets from Vault for a secure GitOps based workflow.
• Git Secret - A bash-tool to store your private data inside a git repository.
• Infisical - Open source end-to-end encrypted secrets sync for teams and infrastructure.
• Lade - Automatically load secrets from your preferred vault as environment variables.

Sharing

A collection of tools to help with sharing knowledge and telling the story.

• Gitbook - Modern documentation format and toolchain using Git and Markdown.
• Docusaurus - Easy to maintain open source documentation websites.
• Docsify - A magical documentation site generator.
• MkDocs - Project documentation with Markdown.

Conferences

• DevOpsCon
• AWS re:Invent
• DevSecOps
• ADDO
• DevOpsConnect
• @Scale
• devopsdays
• DevOps Enterprise Summit