Transform

Config DB allows you to transform the configuration after they've been scraped from the target. This is supported by all the scrapers.

Transformation can be useful when you want to

• hide/remove sensitive data from the scraped configuration (e.g. passwords, tokens, etc.)
• transform the scraped configuration using Javascript and Go templates.
• remove certain fields from the scraped configuration

Field	Description	Scheme	Required
script	Script to transform scrape result.	Script
exclude	Specify fields to remove from the config. It is useful for removing sensitive data and fields.	[]Filter	false
mask	Specify configurations to replace sensitive fields with hash functions or static string.	[]Mask	false

Script

Script allows you to transform the scraped configuration using Javascript and Go templates.

Field	Description	Scheme	Required
gotemplate	Specify Go template for use in script	string	false
javascript	Specify javascript syntax for script	string	false

Note

Unlike other transformation functions, script is ran before the attributes (id, name, type, ...) are extracted. So please make sure your transformation scripts are inline with the JSONPath selectors for the attributes.

JavaScript

You can supply a JavaScript code to transform the scraped configuration. Your JS code will have access to the special config variable which will contain the scraped config. Your script is expected to return a stringified JSON object which will be the new configuration.

Example: The following Config DB configuration specifies a transformation that'll add a new field "hello" with the value "world" to all the scraped configurations.

file:
  - type: Config
    id: $[0].id
    name: $[0].name
    transform:
      script:
        javascript: |+
          for (var i = 0; i < config.length; i++) {
            config[i].hello = "world"
          }
          JSON.stringify(config)
    paths:
      - fixtures/data/multiple-configs.json

Considering that the fixtures/data/multiple-configs.json file contains the following configuration

[
  {
    "name": "Config1",
    "id": 1,
    "password": "p1",
    "secret": "secret_1"
  },
  {
    "name": "Config2",
    "id": 2,
    "password": "p2",
    "secret": "secret_2"
  }
]

The JS transformation will result in two new config items

{"id": 1, "name": "Config1", "added": "a", "secret": "secret_1", "password": "p1"}
{"id": 2, "name": "Config2", "added": "a", "secret": "secret_2", "password": "p2"}

Go Templates

Go template is another powerful way to transform the scraped configuration. Just as you provide a javascript code, you can also provide a Go template. The Go template will have access to the special config variable which will contain the scraped config.

file:
  - type: Config
    id: '$.id'
    name: 'scraped'
    transform:
      script:
        gotemplate: |+
          [{
            {{range .config}}
              "name-{{.id}}": "hi {{.name}}",
            {{end}}
            "id": "what"
          }]
    paths:
      - fixtures/data/multiple-configs.json

The above transformation will result in the following config

{ "id": "what", "name-1": "hi Config1", "name-2": "hi Config2" }