Zuar has released the latest version of Mitto - version 2.7!

Below is a summary of the new features, new plugins, and improvements. Click to jump to a specific section.

We are incredibly excited about this release. If you are currently using Mitto and would like to schedule your upgrade, or if you are interested in seeing some of these features in action, please contact us!

New Features

Below are the major new features in Mitto 2.7.

Snowflake

Mitto now supports Snowflake as both a source and destination for data. Snowflake can be used as the source database for a Query job, the destination database for any input/output (IO) job (Salesforce, Netsuite, Query, etc), and the database for any SQL queries.

Mitto automatically handles creating new schemas and tables in Snowflake. Mitto also handles changing destination tables when source data changes (e.g. data types, new columns, etc).

We will be writing more content on the Snowflake integration in future blog posts.

Webhooks

Mitto has always had notifications built into the platform, both in the web UI and via email notifications. Recently, we had users request that we add other forms of notifications (e.g. Slack) to Mitto. We thought this was a great idea and generalized the idea by adding webhooks to Mitto!

What are webhooks?

In a nutshell, webhooks are events that allow applications to interact with each other. Specifically in Mitto, webhooks can request or send data to any URL when a job starts, completes, succeeds, or fails.

In the example below, we set up a Mitto webhook that sends a notification in Slack when the job is started. You can see the Slack notification in the top right.

Mitto webhooks can be initiated from four different job event types:

  • JOB_START - initiate the webhook when the job starts
  • JOB_COMPLETE - initiate the webhook when the job completes
  • JOB_SUCCESS - initiate the webhook if the job succeeds
  • JOB_FAILURE - initiate the webhook if the job fails

Mitto webhooks can either request data (GET) or send data (POST). Typically, you want to send data to another application via a POST.

Multiple webhooks can be added to any job in Mitto, enabling complex use cases.

To learn more about webhooks, see Mitto documentation here.

Use cases for Webhooks:

There are many use cases for webhooks in Mitto:

  • trigger an action in another platform (e.g. Slack, Twilio, etc)
  • trigger an action in Mitto (e.g. one Mitto job starting another Mitto job)

We will be writing more blog content in the future about these use cases.

Named Credentials

In 2.7, we added a credentials section to Mitto.

This allows Mitto users to store credentials in a single, secure location. These "named credentials" can be referenced in Mitto jobs.

Over time, credentials for the various applications and databases Mitto interacts with can change, and this feature simplifies the management of such changes.

Depending on the nature of the credentials (username, password, api key, database URL, etc), different information can be stored.

Mitto connectors/plugins create the available credential types. Here's an example using a "Basic" credentials type:

To learn more about named credentials, see Mitto documentation here.

Parameterized SQL

Transformation is an entire step of the Data Value Chain. Therefore, running SQL queries on automated schedules is a big use case for most Mitto users.

In Mitto 2.7, we took the first step in making some aspects of SQL easier for Mitto users by adding parameterized SQL. This was a direct request from some our of Mitto users.

To show a simple example of parameterized SQL, let's look at a simple SQL job in Mitto.

This job drops and recreates a table named normal_sql in the schema test.

Here's what the resulting data looks like:

number
1

Now let's say we want to make the value of 1 a parameter. Here's the same job tweaked to use parameterized SQL:

Looking at the job's JSON config, we can see where we can change the value of the :number parameter:

Here's the resulting data:

number
5

This is a very simple example, but the concept becomes very powerful with more complex SQL queries. Mitto users with less SQL knowledge can interact with SQL jobs by changing parameters rather than editing SQL.

To learn more about SQL jobs, see Mitto documentation here.

Backup / Restore

Mitto runs as a virtual appliance that can be hosted anywhere (on-premise, private/public clouds). Depending on the hosting environment, there are many external ways to backup and restore images of Mitto.

In Mitto 2.7, we have added two new commands to Mitto's command line interface to help with these backup and restore use cases using Mitto itself:

  • backup - Creates a backup file containing all Mitto application data
  • restore - Restores a Mitto system from backup

Using a Mitto command line job, you can create a job to back up Mitto on a schedule.

To learn more about backup and restore, see Mitto documentation here.

New Plugins

RegEx

Many Mitto users have large numbers of flat files (CSV, Excel, JSON, etc) they want to analyze.

Mitto has always had plugins that convert individual flat files into individual tables in a database.

The Regex plugin creates a single “union” table in a database from any files matching a regular expression by leveraging ANY of the flat file connectors (CSV, Excel, JSON, etc).

The resulting database table has two additional columns:

  • __source__ - the source file the rows originate from
  • __mtime__ - the modified time of the source file

Upsert is supported for RegEx jobs by using the __source__ as the primary key and __mtime__ as the last modified time.

To learn more about the RegEx plugin, see Mitto documentation here.

Email (Outgoing - SMTP)

Mitto users have data all over the place. Surprisingly, one of the many ways Mitto users receive data from vendors, partners, and even their customers is via email. This is a standard data staging use case that Mitto has been solving for a while by pulling files from emails with the Email (Incoming - IMAP) job.

Mitto users also have a need to distribute data via email. In Mitto 2.7, there is a new job type to solve this data distribution use case: Email (Outgoing - SMTP) job.

The Email (Outgoing - SMTP) plugin allows Mitto users to create custom formatted emails with or without attachments.

Here's an example of a Mitto job that sends an email with two image attachments and a custom HTML email body with another image.

This example also leverages the new Mitto 2.7 feature named credentials.

To learn more about the Email (Outgoing - SMTP) plugin, see Mitto documentation here.

Improvements

Below are some of the improvements we made to Mitto in 2.7.

v2 API - Swagger

Every Mitto instance has Swagger documentation at /docs. You can see all the API endpoints and test them using the Swagger UI.

In Mitto 2.7, we added many new v2 endpoints to replace earlier versions of API endpoints.

Swimlanes - New Summary Metrics

The Swimlanes page now has summary metrics across the top for the selecting time range:

  • Number of individual jobs run
  • Number of total job runs (jobs can be run more than once during a time range)
  • Successful job runs as a percentage of total job runs

To learn more about swimlanes, see Mitto documentation here.

Job Duration

We added a Duration column for jobs. Now Mitto users can see how long a job has been running in addition to how long it took to run historically.

Job History Pagination

We added pagination to a Job's history section. Now Mitto users can see every historical run for any given job.

Tableau Extract Refresh

We updated the wizard for the Tableau Extract Refresh plugin.

We also made changes to the plugin itself.

Previously, the resulting job stored the Tableau extract's task_id. Changes to extract refreshes on the Tableau side can create a new task_id, and therefore the Mitto job would be out of sync.

Now, the job stores the schedule_id and the target_id from Tableau. These values always tie back to the correct Tableau extract.

To learn more about the Tableau Extract Refresh plugin, see Mitto documentation here.

Generic Job

There are many custom job types in Mitto that do not currently have a wizard interface, so we added in a Generic job to make the process of adding those custom jobs easier.

Logs

We reversed the order of job logs (most recent entries at the bottom) and added auto pagination as users scroll up through the log's history.

Upgrade or take a test drive!

We are incredibly excited about this release. If you are currently using Mitto and would like to schedule your upgrade, or if you are interested in seeing some of these features in action, please contact us!