Salt ❖ Experimenting with templating

Wrangling confinguration files with config management

James Booth

7 minute read

featured-image

While Salt is the swiss army knife of configuration management, I had a somewhat trivial task to accomplish - deployment of config files. Using a centralised point such as Pillar or the Salt file system along with Salt’s templating tools, it’s simple to distrbute dynamic config files and get that warm & fuzzy feeling of never having to connect to a remote host again.

The most prominent method is using file.managed with the template: jinja. This is practically a direct extension to Salt’s default jinja templating that occurs in .sls files (This can, of-course, be changed to use other templating languages), and will template any source files before shipping them to the minion.

However, I found as configuration files became more complicated and the data inside them became increasingly dynamic, it was getting increasingly more frustrating to merge the data structures, all the while maintaining the rigid syntax required by the configuration language. To this end, I wanted to explore what else Salt had to offer. The issue I knew I’d face wasn’t a lack of option, but actually figuring out which combination of options work best for each use-case.

In the below examples, I will use a real example of the Sensu monitoring system’s client.json file, which is what provides the basic configuration to a client, as well as any other variables or data we wish for the Sensu client to have. These can then be re-used in check results later on, so having extra data here can help.

Attempt 1 - Jinja variables in a template file with file.managed

Overview

This method is largely the most approachable and basic method, and it involves creating a template file manually and simply injecting values with jinja variables.

  • Create a regular json config file (/srv/salt/templates/myjsontemplate.json)
  • Add in jinja variables. These can refer to grains, pillars, etc.

    {
      "client": {
        "name":    "{{ salt['grains.get']('host') }}",
        "address": "{{ salt['grains.get']('fqdn_ip4')[0] }}", 
        "subscriptions": {{ salt['pillar.get']('nodegroups') | json }},
        "socket": {
          "bind": "127.0.0.1",
          "port": 3030
        },
        "fdqn": "{{ salt['grains.get']('fqdn') }}",
        "domain": "{{ salt['grains.get']('windowsdomain') }}",
        "os": {
          "family": "{{ salt['grains.get']('os_family') }}",
          "name": "{{ salt['grains.get']('osfullname') }}"
        },
        "specs": {
          "cpu_model": "{{ salt['grains.get']('cpu_model') }}",
          "cpu_num": "{{ salt['grains.get']('num_cpus') }}",
          "mem": "{{ salt['grains.get']('mem_total') }}"
        },
        "vars": {{ salt['pillar.get']('sensu:vars') | json }},
        "win_features": {{ salt['grains.get']('win_features') | json }}
      }
    }
    
  • Use file.managed to create & template the file on the minion.

    C:\\test.json:
      file.managed:
        - source: salt://templates/myjsontemplate.json
        - template: jinja
    

Pros

  • Easy enough to slot into existing configs and make them more dynamic without any major rewrites.
  • Flexible enough to work with any config file syntax (json, yaml, toml, etc.) or even non-config files (odt documents, etc.)
  • Clever usage of some jinja filters (eg. |tojson) allows embedding of any Salt data structure (dict, list, etc.) - but don’t expect it to look pretty!
  • Allows the use of advanced jinja templating features - logic (if, then, else), loops, mapping, filters, etc. - meaning that the config files can get really dynamic and intricate.

Cons

  • It is essentially glorified ‘find and replace’ with jinja templating, and can get funky when injecting complicated data structures, such as embedding a dict with the |tojson(indent=2) filter. This won’t look native unless a lot of effort is taken to preserve white-spacing, etc. but the end result is still valid, machine-readable configuration for our app.
  • If the templates get really intricate, with a lot of jinja logic, it may become difficult to support and it may be worthwhile using a pure python approach, such as using pydsl instead of jinja/yaml, or move logic over to a Python module.

Verdict

Best used basic config files or config files that just want to inject a few variables (Not complicated data structures). Logic is a nice feature, but don’t go too far into the weeds with loops and nested logic or you’ll find yourself programming in a templating language.

Attempt 2 - Structured data in pillar and templated with file.seralize

Overview

I liked the idea of this as it keeps with the core principal of keeping all config data inside Salt’s Pillar. However, I found I quickly ran into limitations.

  • Create pillar file (/srv/pillar/sensu/client_config.sls).
  • Add data, eg.

    sensu_client_config:
      client:
        name: {{ salt['grains.get']('host') }}
        address: {{ salt['grains.get']('fqdn_ip4')[0] }}
        subscriptions: {{ salt['pillar.get']('nodegroups') }}
        socket:
          bind: 127.0.0.1
          port: 3030
        fdqn: {{ salt['grains.get']('fqdn') }}
        domain: {{ salt['grains.get']('windowsdomain') }}
        os:
          family: {{ salt['grains.get']('os_family') }}
          name: {{ salt['grains.get']('osfullname') }}
        specs:
          cpu_model: {{ salt['grains.get']('cpu_model') }}
          cpu_num: {{ salt['grains.get']('num_cpus') }}
          mem: {{ salt['grains.get']('mem_total') }}
        vars: {{ salt['pillar.get']('sensu:vars') | pprint }}
        win_features: {{ salt['grains.get']('win_features') }}
    
  • Apply to node(s) in /srv/pillar/top.sls:

    base:
      'some_minion':
        - sensu.client_config
    
  • It should now be accessible to minions after a saltutil.refresh_pillar, demonstratable with a pillar.get 'sensu_client_config'

  • Create a state file to apply this to a minion:

    C:\\test.json:
      file.serialize:
        - dataset_pillar: sensu_client_config
        - formatter: json
        - show_changes: True
    

Pros

  • All data is held in Pillar, which is re-usable by other states.
  • Keeps data in a single location.
  • Can also be extended by any number of external pillars (file_tree, mongodb) and the syntax in the state remains the same.

Cons

  • The dealbreaker here - You cannot embed other Pillar data into this data! This means you can’t data from other sources, such as nodegroups.

Verdict

Usable only for data that doesn’t include other Pillar data. In my use case, I need to cross-reference Pillar data, but this won’t always be the case. And when it’s not the case, all of the data can be stored under a single Pillar key for ease of use.

Attempt 3 - Passing data straight into file.serialize

This method involves storing the data in one form, loading it in (there are a number of means) and then seralizing the data straight to the host.

  • Create the data one of many ways (Examples from the Salt docs because of laziness):

    • Create raw data with python and convert to json or yaml:
    data = {
        'foo': True,
        'bar': 42,
        'baz': [1, 2, 3],
        'qux': 2.0
    }
           
    yaml = {{ data | yaml }}
    json = {{ data | json }}
    # Or to  be saved with indented data
    pretty_json {{ data | tojson(indent=2) }}
    pretty_yaml = {{ data | yaml(False) }}
    
    • Import json or yaml directly:
    {%- set yaml_src = "{foo: it works}" | load_yaml %}
    {% load_yaml as yaml_src %}
      foo: it works
    {% % endload}
    {% load_json as json_src %}
        {
            "bar": "for real"
        }
    {% endload %}
    
    • Import json or yaml from file:
    {% import_yaml "myfile.yml" as myfile %}
    {% import_json "defaults.json" as defaults %}
    
  • In my case, I used load_yaml and added the data directly into the .sls file for demo purposes. For actual usage, I would keep my template yaml files in a standardised location, separate from the state logic.

    {% load_yaml as yaml_src %}
    client:
      name: {{ salt['grains.get']('host') }}
      address: {{ salt['grains.get']('fqdn_ip4')[0] }}
      subscriptions: {{ salt['pillar.get']('nodegroups') }}
      socket:
        bind: 127.0.0.1
        port: 3030
      fdqn: {{ salt['grains.get']('fqdn') }}
      domain: {{ salt['grains.get']('windowsdomain') }}
      os:
        family: {{ salt['grains.get']('os_family') }}
        name: {{ salt['grains.get']('osfullname') }}
      specs:
        cpu_model: {{ salt['grains.get']('cpu_model') }}
        cpu_num: {{ salt['grains.get']('num_cpus') }}
        mem: {{ salt['grains.get']('mem_total') }}
      vars: {{ salt['pillar.get']('sensu:vars') }}
      win_features: {{ salt['grains.get']('win_features') }}
    {% endload %}
            
    C:\\opt\\test.json:
      file.serialize:
        - dataset: {{ yaml_src }}
        - formatter: json
        - show_changes: True
        - encoding: unicode
    

Pros

  • Allows source data to be stored in a number of formats and output in an number of formats (eg. yaml -> json, python -> yaml, json -> json with dynamic values thanks to grains/pillars/etc).
  • Data can still be retrieved a number of ways, and due to the Salt file system (salt://) and flexibility of jinja, there are no restrictions as to where the data can be saved, what source format, etc.
  • Still benefits from being able to apply jinja logic if needed, or the data can simply be slurped in from one format and shot straight out as the output data without much interference.

Cons

  • Not so much an issue directly with this implementation, but it can spit out unicode strings as u'Whatever' as Python does. I had this happen with my custom win_features grains until I resolved the unicode issue in the grains.

Verdict

My favourite method of the bunch. The flexibility provided is pretty great and I’ve not seen it trip up on anything just yet.

The best part about this method is that I can store my config data as human-readable yaml and then serialize to json, which is more machine-readable. Nobody likes writing json.

comments powered by Disqus