Data Sync Overview

Iterable Data Sync is a native, fully-managed ETL service that enables you to copy your most valuable Iterable-enriched customer data into your own cloud data warehouse or storage. It’s designed to give you full control and analysis in your environment while simplifying multi-project operations via organization-level configuration and monitoring in the Iterable UI.

Iterable Data Sync provides a reliable, scalable path to a single source of truth in your warehouse—without building and maintaining custom pipelines.

# In this article

# How it works

Iterable Data Sync connects to your destination and exports data to it based on your configuration. After providing the destination to Iterable, you will receive an Iterable link from the Iterable team to fill in details regarding your destination. During the first export of data, Iterable Data Sync sends all supported data to your selected destination–during subsequent exports, only new and updated Iterable data is exported, reducing the volume of data operations required, to provide optimal performance.

To enable Iterable Data Sync, use the Iterable-provided link to configure your export details, create a special purpose user to export data to the selected warehouse, allowlist an IP address associated with your organization’s region in Iterable (USDC or EDC) for the data transfer, and provide the selected destination required connection details and credentials. Details are provided in the following sections of this document.

# Supported data sources and destinations

With Data Sync, you can export:

• Iterable system events
• Campaign-related data (for example, name, projectId, type, etc.)
• User profile fields (including Iterable-generated UUID, customer generated userID, email address, and more)
• List and subscription-related data

At this time, you can export data to Databricks, BigQuery, Redshift, AWS S3, and SFTP. Additional destinations will be available in the future.

# Data formatting

Exported data is formatted the same as it is in Iterable. Timestamps for data transfers are provided for each table as follows:

• For Databricks and Redshift, the _transfer_status table includes transfer_last_updated_at.

• For BigQuery, the __TABLES_SUMMARY__ metatable includes last_updated.

• For AWS S3, data is transferred as Apache Parquet, by default, with the following file structure:

  <bucket_name>/<folder_name>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.parquet
  

  Where:

  • <bucket_name> and <folder_name> are set when you configure the destination.
  • <model_name> is the name of the data model being transferred (equivalent to a table name in relational data destinations).
  • <transfer_date> (formatted as 2006-01-01) and <transfer_timestamp> (formatted as 20060102150405) are generated when the transfer starts.
  • <file_part> is a monotonically increasing integer for a given timestamp, without special meaning.

You can also transfer data as CSV or JSON files in Apache Hive style partitions, but Parquet is preferred for performance.

• For SFTP, exports use Parquet (default), CSV, or JSON/JSONL in a Hive-style folder layout, plus manifest files under _manifests/ at the path root.

# Limitations

The following limitations apply when exporting data with Iterable Data Sync:

• It can take up to 6–12 hours for data to populate your data warehouse.

• Performance varies based on the size and complexity of your data model and the number of records you're exporting.

• Note that individual files for object storage destinations will not exceed ~4 GB in size. Rows transferred to object storage destinations are treated as "append only". This has the following effects for all object storage destinations:

  • Data is always written based on the current data model, so any schema changes or data model updates must be handled downstream from a transfer.

  • Deduplication, merging updated rows will not take place during transfer, all updates will be written as unique rows. These updates must also be made downstream from a transfer.

  • Full refresh transfers will append the full dataset without deleting existing data.

# Common tasks

Once you configure any destination type, you can perform the following common tasks from the Data Sync Index page (Integrations > Data Sync):

• Select an existing destination to view details like its current status, the number of rows and volume of data transferred, and a transfer log for each sync with details about any errors that may have occurred.

• Edit existing destinations.

• Archive existing destinations. Any inflight transfers will be stopped, and future transfers will not be allowed. You'll still be able to view data that was transferred before the destination was archived.

# Want to learn more?

For more information about some of the topics in this article, check out these resources. Iterable Academy is open to everyone — you don't need to be an Iterable customer!

Iterable Academy

• Product Bite: Data Sync