Mitto has the ability to store source system data as JSON on the file system of the Mitto server.

The data is stored in a RocksDB key-value store.

Benefits of a Mitto Store

  1. Seeing the "raw" data from the source system converted to JSON (even if the source system's data was not in JSON originally)
  2. Querying the source system ONCE per endpoint and allowing Mitto to query the JSON store for nested lists, rather than re-querying the source system again

Configuring a Mitto Job to use a Store

Mitto input/output (IO) jobs can be optionally configured to store raw source data as JSON.

A normal Mitto IO job's JSON config has at least 3 key/value pairs: input, output, and steps. You might also see an optional sdl key/value pair.

A Mitto IO job that uses a store has an additional store key/value pair.

Example Mitto CSV Job with Store

Example source CSV file:

A Mitto job is created using this CSV file and the CSV plugin wizard.

Job Title: [CSV] store_example.csv

Job Name: csv_store_example_csv (The job name corresponds to the name of the store.)

Here's the job config of the newly created job:

{
    "input": {
        "delimiter": ",",
        "encoding": "ASCII",
        "includes_header": true,
        "source": "/var/mitto/data/store_example.csv",
        "use": "xsv.iov2#XsvInput2"
    },
    "output": {
        "dbo": "postgresql://localhost/analytics",
        "schema": "csv",
        "tablename": "store_example",
        "use": "call:mitto.iov2.db#todb"
    },
    "sdl": {
        "columns": [
            {
                "__header__": "id",
                "name": "id",
                "type": "Integer"
            },
            {
                "__header__": "name",
                "__jpath__": "$.'name'",
                "length": 6,
                "name": "name",
                "type": "String"
            }
        ],
        "name": null
    },
    "steps": [
        {
            "transforms": [
                {
                    "use": "mitto.iov2.transform#ExtraColumnsTransform"
                },
                {
                    "use": "mitto.iov2.transform#ColumnsTransform"
                }
            ],
            "use": "mitto.iov2.steps#Input"
        },
        {
            "use": "mitto.iov2.steps#CreateTable"
        },
        {
            "transforms": [
                {
                    "use": "mitto.iov2.transform#FlattenTransform"
                }
            ],
            "use": "mitto.iov2.steps#Output"
        },
        {
            "use": "mitto.iov2.steps#CollectMeta"
        }
    ]
}

When this job is run, the data from the CSV is piped into a database table. In this case a table named store_example is created in Mitto's internal PostgreSQL database in the csv schema.

The job can be edited and configured to use a store:

{
    "input": {
        ...
    },
    "output": {
        ...
    },
    "sdl": {
        ...
    }
    "steps": [
        ...
    ],
    "store": {
        "key": [
            "id"
        ]
    }
}

Note the addition of the store key/value pair. The store requires the source data has at least a primary key. In this case the primary key is id.

Querying the Mitto Store API

This section relates to benefit #1 of using a Mitto store.

The store section contains the key which references a source field.

You can use the name of the job and the key value in the resulting database table, to see the results of the JSON store in a browser.

Navigate to this URL in the browser:

https://{mitto_url}/api/store/{job_name}/{key_value}

Using our example job above:

https://{mitto_url}/api/store/csv_store_example_csv/1

This is the record in the csv_store_example_csv job's store with the primary key 1.

Stores with multiple keys

Some Mitto stores can have multiple keys.

For example:

"store": {
    "key": [
        "date_start",
        "ad_id",
        "country"
    ]
}

In order to see the Mitto store in the browser, you concatenate the key values with double underscore ("__").

https://{mitto_url}/api/store/{job_name}/{key_value_1}__{key_value_2}__{key_value_3}

Using the Mitto Store as a Source

This section relates to benefit #2 of using a Mitto store.

Read more about piping data from a Mitto store here.

Deleting a Mitto Store

If necessary, a job's Mitto store can be deleted using a Mitto command line job.

When a job is configured to use a store, the store is located on the Mitto server at this location: /var/mitto/store/{name of job}.

So using our CSV example from above, we can run a command to delete the store:

rm -rf /var/mitto/store/csv_store_example_csv