Release Notes - Mitto [MIT] - 2.8.0

Epic

issue summary
MIT-1662 Make using Mitto with Snowflake more efficient
MIT-1432 Allow non-admin users
MIT-1000 Flat File Connector

Improvement

issue summary
MIT-1728 Add support to Subscriptions for CSV/Excel attachments
MIT-1725 Subscriptions - add support for relative dates in subscription value filters
MIT-1724 Subscriptions - add support for concurrent Tableau exports
MIT-1708 Add SMTP timeout parameter for Subscriptions / Tableau Request
MIT-1691 TRIAL RELATED - SQL Server TIMESTAMP data type - binary data - upsert in query job
MIT-1668 add multi select on files in file manager to download/delete
MIT-1629 Add ability to create a sitemap to mitto/docs
MIT-1628 Users API
MIT-1625 Modify job_tableau_export.py to eliminate the use of 'tableau login'
MIT-1623 implement COPY INTO for Snowflake
MIT-1622 Change publish destination for Mitto documentation
MIT-1610 Wizard - update css for dynamic json-editor elements
MIT-1601 Wizard - add ability to select all/none items in scrolled list
MIT-1600 Wizard - add ability to link to external help from wizard pages
MIT-1575 MariaDB support in Mitto
MIT-1567 Add enable/disable support to the web hooks API
MIT-1559 Extract job improvements
MIT-1553 MaxTimestamp updates from LeafLogix connector
MIT-1544 Mitto Feature Request : Delete File Dialog should list file name to be deleted
MIT-1528 Ability to HIDE or OBSCURE Credentials in Tableau Extract Job
MIT-1523 apply documentation updates to the cookicutter for plugins
MIT-1522 named credentials - add "generic" named credential
MIT-1519 Webhooks - Extend DB with last_triggered info
MIT-1515 Additional frontend functionality for v2 wizards
MIT-1498 generic job - add Type dropdown
MIT-1485 jobs in sequence show their job name rather than their job title - show job title
MIT-1484 sorting in a job's JSON config - does this have to be alphabetical or can it be user sorted?
MIT-1482 Improvements to dependency specification for Mitto plugins
MIT-1474 job history - add status filter
MIT-1454 command job - create wizard
MIT-1449 Move share/bin/* cmds to cli
MIT-1440 Create an IMAP wizard
MIT-1431 Convert builtins to use new wizard
MIT-1426 webhooks - add enable/disable button
MIT-1422 Do not return credentials password or have a "View password" icon
MIT-1383 Remove the XSV sampling page
MIT-1371 Mitto Feature Request - Browser Title Bar
MIT-1346 Mitto Feature Request: Sequence Job Names
MIT-1309 "bulk" delete of files from the Mitto file manager
MIT-1307 add /inventory/delta endpoint
MIT-1289 In Swimlane graphic tooltip include total run time
MIT-1286 Swimlanes - add "search"
MIT-1285 Swimlanes - add "status" filter
MIT-1260 show Title instead of Name for jobs in sequence list
MIT-1250 add "Type" filter to all jobs and history pages
MIT-1242 Swimlanes - add status filter
MIT-1234 Create a missing property filter
MIT-1229 add filters (status, time range) to the Graph page - reduce data size and improve performance
MIT-1211 Excel wizard
MIT-1179 add wizard for other flat file types beyond just CSV (Excel, JSON, etc)
MIT-1176 Improvements for Job API
MIT-1152 Use TOKEN_EXPIRY and create test cases
MIT-1143 Validate SQL queries for graph API
MIT-1120 sort the crosstab columns in the database -> schema -> table page
MIT-1089 show the same "job information" that is on the jobs/history page when looking at the jobs in a sequence on the sequence page
MIT-1087 make all crosstab columns sortable
MIT-1046 add "type" as a parameter to /jobs endpoint
MIT-1041 Convert notifier package to FastAPI
MIT-1032 Create an updated_at mixin in mitto.iov2.mixins
MIT-951 Add basic Job stats to Job list View in Sequence Details Screens
MIT-942 database controls - table viewer - make "drop" button gray/red.
MIT-913 database viz - let the user know they can click to drill in
MIT-890 add indexes on Mitto tables
MIT-831 autotag jobs with their connector for builtins
MIT-813 add pagination anywhere we show a list of jobs (sequence editor for example)
MIT-785 Ensure valid table names
MIT-780 add Redshift section (S3 etc) to output of query job
MIT-748 when downloading a file from the file manager put the name of the file in the dialog box
MIT-689 Job Search - case sensitive?
MIT-681 Response time on SCHEDULE on job/sequence detail screen
MIT-636 multi select "status filter"
MIT-622 [CUSTOMER REQUEST] notifications use case via bash script
MIT-25 Implement the hamburger menu on small/medium screens
MIT-15 Test for changes on the job page.

Sub-task

issue summary
MIT-1587 Drag and Drop UI element for json-schema

Task

issue summary
MIT-1615 Tests - Fix tests that use CREATE USER
MIT-1366 Upgrade to pydantic 1.0
MIT-737 500 error trying to remove a sequence created via a python script

Bug

issue summary
MIT-1776 Microsoft Access connector jobs failing because of column header case
MIT-1773 MSSQL - Query with Reflect - RuntimeError: Invalid reflection class: sqlalchemy.dialects.mssql.base.UNIQUEIDENTIFIER{noformat}
MIT-1765 credentials - credential (basic and generic) is empty after creating it
MIT-1764 Excel wizard - add indicator that Mitto is doing something when clicking next - long pauses
MIT-1763 sequence - when adding jobs to sequence, jobs aren't enabled by default
MIT-1762 sequence - creating new sequence, jobs end up in job id order (ascending) instead of user defined order
MIT-1761 Ownership of named credentials in /var/mitto/credentials prevents editing
MIT-1752 Job sequence - info about jobs in sequence missing after updating
MIT-1751 Job sequence - disabled jobs sometimes disappear
MIT-1750 icon colors for jobs in sequence on sequence page
MIT-1743 Sequence with no jobs hangs.
MIT-1742 running empty sequence - sequence runs indefinitely
MIT-1737 Google BigQuery - Create dataset if it doesn't exist
MIT-1732 Google BigQuery - google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS
MIT-1731 Incorrect 'showing' message on stage.
MIT-1729 Wizard - hidden fields no longer hidden after cutover to Pydantic 1.4
MIT-1722 query - wizard - 400 Bad Request 400 Bad Request 'tables_meta'
MIT-1716 CircleCI build deprecation warnings
MIT-1714 Missing X-REMOTE-USER header
MIT-1713 Snowflake column naming convention doesn't match PostgreSQL naming convention
MIT-1711 Logs Page - Large exceptions are not shown
MIT-1707 database page - meta schema does not show up on the list of schemas
MIT-1706 MySQL - job fails if database isn't already created
MIT-1705 Snowflake - Fails to adapt from boolean to number
MIT-1704 long job names in sequence editor - can't see full name or hover to see text
MIT-1702 Cred.io - SQL Syntax error (ALTER TABLE plt_wealth_user_plans_view ALTER COLUMN amount TYPE NUMERIC)
MIT-1694 Google BigQuery (as output) - 400 DELETE must have a WHERE clause at [1:1]
MIT-1693 Google BigQuery (as output) - 404 Not found: Dataset mitto-183418:meta was not found in location US
MIT-1692 Google BigQuery (as job output) - 400 Name meta not found inside meta at [3:14]
MIT-1686 Mitto wizards (CSV, Query, ??) when using Redshift as output are not adding extra Redshift info to the resulting job config
MIT-1679 file download - title and name don't sync when using wizard
MIT-1678 csv with no headers - front-end support
MIT-1676 Cannot Duplicate Job in Stage
MIT-1670 Wizard fails on csv
MIT-1667 job edit (name, title, etc) button doesn't save when user clicks edit again - user has to press enter to save a change
MIT-1666 PostgreSQL to Snowflake - query job with upsert and ExtraColumnsTransform rename_columns FALSE
MIT-1665 ReflectTransform with upsert on Snowflake - fails on first upsert attempt - KeyError: 'id'
MIT-1664 ReflectTransform - Snowflake to Snowflake - table DDL is correct, but all values are NULL
MIT-1663 logs - copy paste isn't possible with the way the lines are selected
MIT-1658 ReflectTransform - PostgreSQL to Snowflake REFLECT job fails on the second run
MIT-1653 ReflectTransform - SQL Server error - TypeError: '>' not supported between instances of 'NoneType' and 'int'
MIT-1650 column of zeros creates bool on first run and converts to numeric on second run (fails)
MIT-1648 data editor primary key is string with spaces
MIT-1647 UI not showing schemas starting with "pg" (no underscore!)
MIT-1640 Snowflake - can't convert data type from BOOLEAN to INT/FLOAT
MIT-1639 Snowflake - KeyError: 'check'
MIT-1638 Excel plugin - int and boolean being converted to float
MIT-1635 running 24 concurrent sequences - FATAL: sorry, too many clients already
MIT-1632 Migration script fails
MIT-1630 SQL wizard is blank (2.7.5)
MIT-1627 GTI Mitto Duplication Error
MIT-1626 When duplicating Job unusual error
MIT-1619 csv with no headers - wizard, sdl, and resulting table bugs
MIT-1618 500 /api/config
MIT-1614 "Upload Timed Out - Please Try Again"
MIT-1609 history - able to click next page when there is no next page
MIT-1606 pip install fails on fresh virtualenv
MIT-1603 jscs errors
MIT-1597 multiselect jobs -> actions -> create sequence - no jobs enabled
MIT-1577 /key/me 500 error
MIT-1576 MySQL - MySQL ALTER TABLE test ALTER COLUMN stuff TYPE VARCHAR(19)
MIT-1574 XSV plugin - bad file format (XLSX extension converted to CSV) returns 500 error in wizard
MIT-1568 CSV plugin - can't upload TSV file - You can't upload files of this type
MIT-1558 [CUSTOMER] Excel - how to deal with invalid date values? xlrd.xldate.XLDateNegative: -621438.0
MIT-1556 searching for "site" in sequence editor shows failed jobs list instead of just jobs that match "site"
MIT-1549 sequence - searching after selecting a job removes the selection from the job
MIT-1518 "Logs" button shows no log info for failed job
MIT-1513 search input - searches too fast for user to type in box
MIT-1494 Handle websocket connections after server shutdown/page close
MIT-1493 clean up SQL wizard text
MIT-1477 errors on upgrade to 2.7
MIT-1470 Exception info missing from log
MIT-1469 error in validation of config provided to generic job wizard
MIT-1466 Mitto Left Sidebar Inaccessible when browser is too thin
MIT-1446 Job Details page - Redundant API calls
MIT-1385 Monitor Job returncode set to query result
MIT-1184 Database API offset and limit error
MIT-1178 rename CSV plugin to Flat File
MIT-1146 Credential manager UI bug
MIT-1059 columns missing (that have data in source table) from query job SDL
MIT-1024 mariadb BOOL(EAN) error
MIT-970 PostgreSQL has a limit of 63 characters for column name length - how do we deal with this?
MIT-969 seems to be a Mitto job filename character length maximum (63 chars?) - some jobs require longer names
MIT-935 XSVre fails due to runtime table errors
MIT-919 CSV job creates table with all nulls except for index column
MIT-905 Query - MSSQL to MSSQL error - Progea
MIT-904 MSSQL - varchar > 8000 characters needs to be VARCHAR(MAX)
MIT-883 datetime shows up as time if it only has values from today
MIT-873 meta.meta table isn't updating based on subsequent job runs
MIT-848 Query job converted "timestamp with time zone" to "time"
MIT-826 Downloading twbx from file manager tries to render in browser instead of actually downloading
MIT-811 handle new columns being added to source object and backfilling all rows
MIT-807 Missing error message after sampling
MIT-805 [Xcel] RuntimeError: Job 'xsvre' returned: 1
MIT-804 SDL-based jobs have null for name
MIT-803 xsv not working on blackline
MIT-801 New CSV File job fails
MIT-798 Creating CSV job - data type is Date/Datetime but database is actually varchar
MIT-795 samples dates as datetime always
MIT-776 time field being converted to datetime - should be string?
MIT-774 Tableau Data Extract Refresh - 500 error when creating job on playworks.zuarbase.net
MIT-770 MySQL - double precision column being converted to string
MIT-769 Mitto Sales Demo - TypeError: Decimal('106.42') is not JSON serializable
MIT-768 Rehrig - 42000 error when trying to use Query connector
MIT-766 Query job (with upsert) fails on 2nd run - looks like is mixing up the primary key and last updated columns
MIT-765 multiple primary keys does not work with query connector even after fixing the config
MIT-752 add an option to NOT use a transaction for SQL job
MIT-744 TypeError: Object of type 'UUID' is not JSON serializable
MIT-735 500 error when GET'ing database endpoints too often
MIT-723 UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead.
MIT-672 psycopg2.ProgrammingError: syntax error at or near "user" LINE 1: ALTER TABLE salesforce.cttmp__user RENAME TO user
MIT-656 Netsuite Mitto Demo credentials error
MIT-626 meta table counts not being recorded?
MIT-625 [Redshift] Timestamp format is invalid. See DATEFORMAT and TIMEFORMAT Strings for valid formats.
MIT-587 sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: sorry, too many clients already
MIT-436 timestamp of file manager is off
MIT-434 sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected
MIT-418 sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected
MIT-405 weird conversion of SQL on toughmudder.zuarbase.net after upgrade
MIT-371 [Redshift] psycopg2.NotSupportedError: ALTER COLUMN TYPE is not supported
MIT-343 [Marketo] AttributeError: 'Sampler' object has no attribute 'sample_text'
MIT-341 [Marketo] AttributeError: 'Sampler' object has no attribute 'sample_text'
MIT-224 location of uploaded files (CSVs, etc) has changed with new verison which breaks old jobs
MIT-167 TypeError: 'dict' object does not support indexing
MIT-150 Invalid JWT -> 500 Error
MIT-93 email alerts come from mitto@akiri.co even though jobs on separate mitto instance
MIT-2 Password save : Google chrome

New Feature

issue summary
MIT-1689 Add "Previous" button to first page of job create wizards
MIT-1685 Ace editor is shown above footer
MIT-1683 File list UX
MIT-1682 Better css for rounded buttons on environment variable editor
MIT-1681 Better card multi-selected state
MIT-1680 Show 'running' on job card when running
MIT-1646 preinstall R in Mitto
MIT-1641 Show successful output of cmd jobs in logs.
MIT-1452 system alerts - high RAM usage
MIT-1413 Duplicate button for sequences
MIT-1396 Merge the session pytest fixtures
MIT-1390 Test amCharts FDG in Mitto
MIT-1388 Add parameters to the MaxTimestamp step
MIT-1342 Amazon Vendor Central Meta XSVre Job
MIT-1115 Rclone job
MIT-1039 Change control record in Mitto
MIT-425 API for filtering history by date range
MIT-191 ability to define indexes on tables in the UI/job
MIT-41 Allow arbitrary job ordering