Streamline Docker Compose with the Include Directive

Docker Compose is an essential tool for managing multi-container Docker applications. By defining services, networks, and volumes in a single YAML file, it simplifies the process of orchestrating containerized applications. This simplicity is particularly valuable for developers. They can quickly spin up development environments or deploy applications without dealing with complex orchestration tools.

However, as projects grow, so too does the complexity of the docker-compose.yml file. Adding more services, dependencies, and configurations can make the file bloated, difficult to read, and prone to errors. The increased size creates challenges in understanding the configuration. Maintaining the setup gets complicated, especially when multiple developers or teams are involved. Debugging and updating such files can become time-consuming and error-prone, reducing the efficiency that Docker Compose is meant to provide.

Managing sprawling configurations becomes a critical concern in large-scale projects. Compose files might include dozens of services. Each service has specific environment variables, ports, and dependencies. In such cases, a monolithic Compose file can act as a bottleneck for productivity and collaboration. Developers might find it hard to navigate, and small changes could inadvertently introduce issues in unrelated parts of the configuration.

Thankfully, Docker Compose now supports the include keyword. This feature is designed to address the challenges of managing large and complex configurations.

How the include directive works

The include directive allows developers to break down monolithic Compose files into smaller, reusable, and modular pieces. By including external Compose files, you can achieve better organization, improve readability, and streamline team collaboration.

In short, this directive allows you to split your Docker Compose file into separate ones and the syntax looks like this:

include:
  - db.yaml
  - frontend.yaml
  - backend.yaml

Analyzing Keitaro’s CKAN setup with Docker Compose

For reference we will use our docker-ckan repository which is used to add custom environment files. Usually, you split the Docker Compose file into separate compose services. This is done in folders where the compose files are. For the service configuration to be decoupled and independent of each other, best practice is to add a .env file for each compose file.

compose
├── config
│   ├── backend-services
│   │   ├── backend-services.yaml
│   │   ├── .datapusher-env
│   │   ├── .postgres-env
│   │   └── .solr-env
│   ├── ckan
│   │   ├── ckan.yaml
│   │   └── .env
│   └── .global-env
├── docker-compose.yml
├── makefile
├── postgresql
│   ├── docker-entrypoint-initdb.d
│   │   ├── 00_create_datastore.sh
│   │   └── 20_postgis_permissions.sql
│   └── Dockerfile
└── solr
   └── Dockerfile

If there is an interdependency between services where there is one service that uses an environment variable and another service uses it in the compose we can merge these in a single file:

include:
  - path: config/ckan/ckan.yaml
    env_file: 
      - config/ckan/.env
      - config/.global-env
    project_directory: .
  - path: config/backend-services/backend-services.yaml
    env_file: 
      - config/backend-services/.postgres-env
      - config/.global-env
    project_directory: .

networks:
  frontend:
  backend:

For our CKAN repository, we’ve created separate paths. We’ve split the actual CKAN application and its necessary back-end services into two distinct compose files. As shown above by the code sample you can see the file .global-env file. This file mainly defines the image versions of the applications. It also specifies other variables used inside some of the containers that are going to be created. The front-end and the back-end networks are in the root compose file. Both of the services use these networks in their network map.

From the ckan.yaml file, we have much less code. This way, we can expand this Compose file if necessary without making it look bloated.

services:
  ckan:
      container_name: ckan
      image: ghcr.io/keitaroinc/ckan:${CKAN_VERSION}
      networks:
        - frontend
        - backend
      depends_on:
        db:
          condition: service_healthy
          restart: true
        solr:
          condition: service_healthy
          restart: true
      ports:
        - "0.0.0.0:${CKAN_PORT}:5000"
      env_file:
        - ${PWD}/config/backend-services/.postgres-env
        - ${PWD}/config/ckan/.env
      volumes:
        - ckan_data:/srv/app/data

volumes:
  ckan_data:

Conclusion

As Docker Compose projects grow in scale and complexity, maintaining a clear and manageable configuration becomes crucial.

With this approach, you can focus on building robust, scalable applications. You won’t be bogged down by the challenges of managing a single YAML file. You can maintain clarity and efficiency using include in your projects. This is true whether you’re in a small development setup or a large-scale production environment. The use of include supports project efficiency throughout the lifecycle.

Adopt best practices for modularizing Docker Compose files. You’ll improve the maintainability of your configurations. This will also empower your team to work more effectively.