Automate Bash, R, and Python Scripts With Mitto
In this blog post we will discuss the Mitto command line job, and then walk through examples of automating scripts from various languages:
- Bash
- R
- Python
With Zuar's Mitto platform, you can automate your ELT/ETL processes and have data flowing from hundreds of potential sources into a single destination. Transport, warehouse, transform, model, report, and monitor: it's all managed by Mitto.
Related how-to video:
Mitto Command Line Jobs
A powerful feature in Mitto is the ability to automate commands and scripts with command line jobs.
These "cmd" jobs allow Mitto users to run any command on the shell of their Mitto instance.
An example might be a job that looks like the following:
{
"cmd": "echo 'hello world' > /var/mitto/data/hello_world.txt",
"cmd_env": {},
"exec": false,
"shell": true
}Instructions for creating a command line job in Mitto can be found here.
In our example case, title the job [cmd] Hello World!, add the JSON configuration above, and click "Done". On the newly created job's page, click the little pencil to edit the job type. Select cmd from the select menu.
Running this job will create a text file at: /var/mitto/data/hello_world.txt.
You can verify it's there by looking on the Files page in Mitto (clicking on Files in the menu on the left).
Here are the contents of the hello_world.txt:
Bash Scripts With Mitto
Alternatively, you can upload and run a bash script in your Mitto command jobs.
Bash, the most commonly available unix shell, is
the shell, or command language interpreter, for the GNU operating system.
Bash commands can be saved in a file and executed as a script.
The following Bash script (test_bash_script.sh) creates a CSV file: written_by_bash.csv
#!/bin/bash
echo "id, name" > /var/mitto/data/written_by_bash.csv
echo "1, Allen" >> /var/mitto/data/written_by_bash.csv
echo "2, Steve" >> /var/mitto/data/written_by_bash.csvUpload the Bash script to Mitto via the file manager.
Then create and run the following command job. Once again, the job type needs to be set to cmd.
{
"cmd": "bash /var/mitto/data/test_bash_script.sh",
"cmd_env": {},
"exec": false,
"shell": true
}The resulting file should look like this:
R Scripts With Mitto
You can also automate R scripts with Mitto.
R is a language and environment for statistical computing and graphics.
The following script (test_r_script) will create a tab separated file (written_by_r.tsv) with the numbers 1-5 on the first line and the numbers 6-10 on the second line.
x <- matrix(1:10, ncol = 5)
fil <- tempfile("data")
write(t(x), fil)
if(interactive()) file.show(fil)
unlink(fil)
write(x, file = "/var/mitto/data/written_by_r.tsv", sep = "\t")Upload the R script to Mitto via the file manager.
Create another cmd job (HINT: you can duplicate an existing job) and edit the job's config:
{
"cmd": "Rscript /var/mitto/data/test_r_script.R",
"cmd_env": {},
"exec": false,
"shell": true
}Finally, on the main job page, change the job type from io to cmd.
Running the job will create a file at: /var/mitto/data/written_by_r.tsv. You can verify this file exists in the file editor. The resulting file should look like this:
Python Scripts With Mitto
Automating a Python script is a similar process in Mitto with a command job.
"Python is an interpreted, high-level, general-purpose programming language."
The following script (test_python_script.py) will create a CSV: written_by_python.csv.
#!/usr/bin/env python3
"""NOTE: You must use python3"""
f = open("/var/mitto/data/written_by_python.csv", "w")
f.write("id, name\n1, Allen\n2, Steve\n")
f.close()One thing to note is that #!/usr/bin/env python won't work, as Mitto has python3 installed.
Upload the Python script to Mitto via the file manager.
Again, we'll create a simple job with the following JSON config, and we'll be sure to change the job type from io to cmd.
{
"cmd": "python3 /var/mitto/data/test_python_script.py",
"cmd_env": {},
"exec": false,
"shell": true
}The resulting file should look like this:
Learn more about automating Python scripts with Mitto.
Conclusion
The script examples covered above were extremely simple, however they show the power of automating scripts in Mitto with the command line job.
While the examples above cover a lot of use cases, they only touch the surface of the Mitto command line job.
Some other command line interface utilities Mitto users use include:
While Mitto has an extensive (and growing!) plugin library to facilitate many automation tasks out of the box, command line jobs enable many more advanced automation use cases.
To close, here are a few helpful Mitto-related links:
