Advanced Configurations
Daton supports a variety of loading modes across its connectors, and this documentation provides a brief overview of them.
Advanced Configurations
Daton supports a variety of loading modes across its connectors, and this documentation provides a brief overview of them.
Loading Mode Overview
The loading mode of an integration determines how data is structured, staged, and sized in the data warehouse. There are two primary loading modes, and choosing the appropriate one depends on your specific data needs.
1. Append Mode
In Append mode, data is added (appended) to the existing dataset in the warehouse without needing a primary key. This mode preserves the historical data and increases the size of the table over time.
- Key Benefit: It helps in tracking the history of changes. For example, different stages of an order can be tracked by appending each stage into the warehouse.
- Drawback: Over time, the table size grows significantly.
2. Upsert Mode
In Upsert mode, the data is inserted or updated based on a set of primary keys defined during the integration setup.
- If the primary key exists: The corresponding record is updated.
- If the primary key does not exist: A new record is inserted.
- Key Benefit: This mode ensures that only the latest information is stored, based on the defined primary keys.
- Drawback: You cannot preserve historical data since only the latest record is retained.
Unnesting Modes
Unnesting determines how RECORD-type columns (nested data) are stored in the warehouse. The choice of unnesting mode depends on your query and data structure needs.
1. Do Not Unnest
This mode stores nested data as a single RECORD column in the warehouse, keeping the data in its original nested format.
- Use Case: When you want to maintain the original structure of the nested data and access it directly as a single column.
2. Up to One Level
In this mode, the first level of the RECORD column is split into a separate child table, while deeper nested records remain in their original form.
- Use Case: When you want partial unnesting, splitting off the first level but keeping deeper levels of nesting intact.
3. Up to Two Levels
Here, the nested RECORD columns are split into child tables up to two levels of depth.
- Use Case: When you need more detailed unnesting, separating data across two levels.
