Configuration management is a common problem that needs to be solved by every company dealing with large scale deployments across many environments. Provisioning environments, deploying applications, and managing infrastructure service(s) is dependant on ensuring the correct configuration is available at runtime. Configuration files are required for nearly all software that we run, whether it’s a simple Cron job; a RESTful microservice, or infrastructure services (MySQL, Syslog etc). Typically we need to deploy the same set of software or services to different environments which are functionally identical with nearly the same configuration, but with slight variation to the configuration files. Having separate configuration files for each, across all environments, can result in duplication and large maintenance overhead.

Configuration management is often a challenging task since there are multiple facets associated with it.

  • There are many different formats used (YAML, INI , JSON, XML etc). This creates a lot of overhead, as developers need to be familiar with each of the defined standards and formats.
  • It is challenging to monitor the configuration drift that happens due to live debugging.
  • Sharing common configuration values among different services is hard without creating a lot of duplication.
  • The lack of standard practices for sharing the configurations might add delays to the development cycle since it involves interaction between app owners.
  • Configuration files sometimes need to contain secrets, passwords, and keys. Protecting them in the right place and retrieving them at right time is another important aspect of configuration management system.
  • When creating immutable deployments of a service, separating the service version from the configuration file version is useful to allow the two to evolve independently of one another.

All the above aspects are addressed through the Configuration Management Framework at WePay. These facets are stitched together to create an automated pipeline that generates configuration files dynamically and vends them to a service securely via a ‘Config Vending Service’. This allows us to treat configuration more like code, and deploy changes quickly and reliably.

Implementation

At WePay, we use YAML as the standard to configure any service. A centralized git repository is used to store the configuration files of all services. Below is the layout of the repository.

configuration repository structure

This repository has different folders, and each folder is dedicated to a service. The configuration in ‘config.yml’ is not the ultimate structure of the configuration file for a service, but rather instructions to ‘Config Management Framework’ to decorate and generate the actual configuration files which are vended for a service at runtime. The values in the file ‘config.yml’ can be overwritten per environment (stage, test or production). The actual configuration files are Jinja2 templates. Shared values are stored as tokens per environment as key values pairs.

By hiding most of the complex configuration details as templates, this abstraction makes the configuration platform agnostic. This standard (both YAML and file hierarchy) is then enforced across the organization for defining configuration files for any service.

The service configuration repository is versioned based on the timestamp when a user modifies any of the content in this repository. Changes to this repository can be submitted by developers as diffs or merge requests and be subject to code review before approval. Versioning configuration in git also gives us a complete history of every change that’s been made, and which developer made the change.

Abstraction

Each service may require a number of configuration files to function (service configuration, log file configuration, etc). These dependencies are identified and grouped to form a ‘type’ in an abstracted layer. Below is the structure of a microservice in the configuration repository.

Syntax

Every service that needs configuration management is categorized to follow a specific type. A type is basically a set of configuration files. Each configuration file is a Jinja2 template and is termed as subtype. The values specified under the subtype are decoration inputs to be applied to the template.

In the above example, the type ‘microservice’ has two different configuration files: application and test. A service that adheres to the type ‘microservice’ would get test and application configuration files. The key here is, users could overwrite parameters which are unique to the service, without having to manage the entire configuration file on their own. This abstraction avoids duplication of configuration files across the organization.

Framework

The diagram below shows how abstracted configs in a version control system are processed to vend usable files for a specific service at various stages.

configuration managament framework

Along with this, the service might also maintain its application configuration as a file (e.g. log configuration, db configuration, etc.). The service will need to have access to this file at start up time. All these configurations could be managed at a single place.

Our configuration repository is centralized on a Version control system (git, subversion). The repository is tied with a Continuous integration system (Teamcity, Jenkins, Bamboo, etc.) to process the raw configs to usable configuration files. The processed files are stored as an artifact in a binary store (Artifactory, Google-cloud-storage, etc.) as a tar file with a corresponding version. Secret values (passwords, certificates and keys) are managed in a secret store (Ansible Vault, Hashicorp Vault, etc). The entire system is tied together using the following components.

Config Baker

Config Baker is a binary application which is responsible for transforming raw configuration files (YAML) in the configuration repository to usable configuration files (.ini, *.conf,.py etc). This is tied with a continuous integration system. The config baker squishes default user defined configuration and environment specific configuration into a single file. This single squished configuration file is then digested to create multiple application configuration files (e.g. application config, log file config) based on the subtypes defined (by identifying right tokens and templates). At this stage, configuration files have the right structure and parameters except for secrets. This is a continuous process. On any change in the repository, a new releasable package of the configuration repository without secrets is created, and tagged with an appropriate version.

Config Baker

Config-Vending Service

This is a RESTful microservice which has access to the secret store and the binary store. This service is responsible for vending the final configuration files by combining the configuration release package created by the Config Baker with any secrets associated with them. Configs are encoded as JSON strings and returned back to the client.

Config-Client

This binary application is responsible for communicating with the Config-Vending-Service and fetching the right configs based on a service name, version, and environment.

Conclusion

Configuration management systems are essential for any fast growing organization that needs to manage large deployment footprints without much manual work. The system helps to enforce a standard pattern and enables cooperation between developers and site reliability engineers. All of this lets us move faster while still maintaining a high degree of confidence in our deployments.