Platform Enhancements

Over the last 6 months, we have been working on platform stability as one of the core focus areas for the Daton platform engineering team. We set out to achieve three overarching goals by end of 2022 –

• Flexible pipelines with a high degree of user controls
• Identify and reduce all barriers to fast data transfers
• Improve visibility and transparency in the data pipeline

Our first major milestone as a part of this plan was the release of an upgraded multi-stage pipeline that was launched on December 21; we covered the details of that upgrade in our earlier update. The following enhancements to the platform are now in production.

Enhanced configurations

• Upgraded capability to reduce delay in propagating configuration setups and updates to the warehouse. 
• Improved visibility into any configuration errors in the UI. 
• Extension of “Sync Now” capability to also handle configuration changes without disrupting the pipeline.

Benefits

• Tables will be configured as soon as the integration is set up, and the first job will start immediately. 
• Updates to tables will no longer get stuck in the pipelines. This tended to happen when users made changes to the configuration when jobs were in progress. 
• Visibility into configuration errors, if any. 
• Users can also use “Sync Now” to push through configurations without needing to wait for the scheduler to kick in.

Table Level Sync

"Sync Now" enables users to run replication jobs in an integration on-demand. As useful as this feature is, the challenge is that it was implemented at the Integration level. When a user hits “Sync Now”, all the tables in that integration would be replicated again even when the user just wanted one of the tables to be re-synced. This problem would become more glaring when there are 100s of tables in an integration. 

With this release, we now support the “Sync Now” functionality at the table level.

Benefits

Better control on:

• data replication, a user who wants to sync only a single table in the integration can now do so without triggering all the tables in the integration.
• costs, users can now sync a single table instead of replicating data for other tables when they didn’t need to.

Re-use Integration Names

• It is common to create integrations to test the service and later delete the integration and start on a clean slate. 
• However, users couldn’t re-use the deleted integration names. 
• The functionality to allow the re-use of the integration name after deleting is now made available. 

Benefits

• Data models built of test integrations can be re-used by dropping and creating integrations with the same names.
• Column hashing: A new feature to handle PII data has been released as a core platform capability. 

The UI will be updated in the upcoming releases to allow users to select columns to hash. 

Until the UI changes are made, please reach out to our support team to discuss your column hashing needs and get the integrations set up accordingly.  

UI

• Improvements in the UI for multi-account selection in the source configuration page
• BigQuery Projects list size increased so that the list is visible to add a new destination.

Upcoming Enhancements

Daton connector- ETA 26 Feb 2022

We understand and appreciate the demands that the data teams are receiving. This results in data delays, putting pressure on those involved in building the data infrastructure. 

As a part of our ongoing effort to become a more transparent data pipeline, we plan to launch a new connector, as step 1, to sync job-related data that Daton is generating in your Data warehouse. Future releases will be built on this capability as we work towards achieving our aim of complete transparency.

The development of the Daton connector is ready, but we had to push this release to the last week of Feb 22. There are some prerequisites that we need to handle before exposing this connector to the users that we couldn’t cover in this release.

Connector Enhancements

S3 source connector 

Enhanced our S3 source and have added support for the following:

1. Folder and Sub-folder level processing
2. Regular expression-based file search
3. If you have files with different types of names in a folder, then you can use REGEX to highlight patterns of files to look for and process data
4. Added support for gz, zip, rar compressed file processing
5. Fix for clone integration regex field

Amazon SP-API

Added Turkey as a marketplace.

Amazon Sponsored Display Connector

Added support for the VCPM fields.

Amazon DSP Connector

Added support for some additional fields.

Amazon MWS Connector

• Added the GetMatchingProduct table.
• Fixed the errors in the ListInventorySupply table.
• Fixed the issue of not loading data with FBAStorageFeesReport.

Infusionsoft Connector

Added the 'tagIds' column in Contacts table.

Intercom Connector

Fixed the missing data in the conversations table.

Google Adwords Connector

Fixed the token renewal issues for some accounts.

Braintree Connector

Fixed the issue of data not loading till the present day.

Unicommerce Connector

Fixed the missing data in the sales order table

RECHARGE Connector

Fix for missing data in the orders table.

LinkedIn Connector

Fixed the issue with data replication.

Pinterest Connector

• Error fixed in the Campaign Ads table.
• Fixed the intermittent java.lang.Arithmetic exception in campaign ads table.

NEW Connectors Added!

• Daily Exchange Rates (Finance)
• Snapchat (Advertising)
• Pinterest (Advertising)
• Gorgias (Customer Support)

Alert!

Upcoming Downtime Notice:

There will be a planned maintenance being performed on 26 Feb. 

• During this time, the system will not be processing any new jobs. 
• All the processes will automatically resume once the maintenance is complete. 
• The exact time of the planned maintenance will be communicated via an email update.