SDL stands for schema definition language. SDL plays a large part in how Mitto IO jobs process data and how Mitto learns and defines the structure of that data.

Mitto IO jobs pipe date from various inputs (APIs, databases, and files). Fields from inputs don't always have data types, however data types are needed when outputting data to relational databases.

SDL is mostly handled by Mitto internally through specific IO job steps and transforms (e.g. ExtraColumnsTransform, ColumnsTransform, and ReflectTransform), however there are a few use cases where specifying SDL is useful:

  • Forcing specific data types when outputting to a relational database
  • Adding new columns to the data piped from an input

How to Force Data Types with SDL

As data from an input is piped through Mitto, Mitto is constantly learning the data and assigning data types to it. Learn more about how Mitto samples data.

In some situations, you may want to force a specific data type.

This example will have Mitto force the data type of the id column to a String.

  sdl: {
    columns: [
      {
        name: id
        type: String
      }
    ]
  }

How to Add New Columns with SDL

Typically, IO jobs should extract data from an input and load it as is into the output. Transformations can be done after the fact with Mitto SQL jobs.

However, there are cases where you may want to do some transformation on the input data as it's being piped to the output.

This SDL example creates a new column in the output with the current UTC date and time.

  sdl: {
    columns: [
      {
        name: mitto_update_at
        type: DateTime
        __eval__: column["__type__"].python_type.utcnow()
      }
    ]
  }