To drive impactful user experiences, marketers need up-to-date information so that users get the right message at just the right time. Customer data comes from many sources, and often gets stored in a data source. Now, you can import from your data source to Iterable and activate your data for cross-channel success.
Iterable's Smart Ingest, co-developed with Hightouch, enables you to quickly set up and manage your data source imports directly within your Iterable project. Smart Ingest empowers you to own your data activation strategy by eliminating the need to set up custom API solutions, and provides flexible data import options that enable you to activate data from data sources with minimal effort.
NOTE
Contact your Iterable customer success manager to discuss adding Smart Ingest to your plan.
In this article
How it works
Smart Ingest connects to your source and imports data to Iterable based on your schedule and configuration. You can set up multiple syncs to import different types of data. Each sync defines how data is matched between your data source and Iterable.
Smart Ingest detects changes in your source data using change data capture (CDC), and syncs only new and updated records to Iterable with each sync run. The first sync run initializes change data capture, and processes all rows from your data model.
Once the first sync is complete, subsequent syncs send only new or changed data to Iterable, reducing the volume of data operations required and providing streamlined performance.
Benefits of Smart Ingest
Here are some of the benefits of using Smart Ingest:
- Streamlined data activation: Import data from your data source to Iterable with minimal effort.
- Customizable syncs: Define how data is matched between your data source and Iterable.
- Flexible data import options: Import data from data sources with minimal effort.
- Easy setup and management: Set up and manage your data source imports directly within your Iterable project.
- No-code solutions: Eliminate the need to set up custom API workflows to import data from your source to Iterable.
Smart Ingest workflow
Here's an overview of the steps to setting up and using Smart Ingest.
Set up a sync
A sync defines how data is matched between your data source and Iterable. There are a few steps to setting up your first sync with Smart Ingest. Here's a high-level overview:
Connect Iterable to your data source. Smart Ingest uses this data source to import data to Iterable. To learn more, read Supported data sources.
Define a data model, which is a customizable table of data Smart Ingest imports from your source. This can be a table name, a view, or a custom SQL query that you enter into the sync configuration. To learn more, read Smart Ingest Data Models.
Select a sync type. The sync type determines what kind of data you sync to in Iterable (Users, Events, Catalog, etc). To learn more about sync types, read Sync Types.
Create field mappings to tell Smart Ingest where each field goes in Iterable. You can create field mappings to map fields from your data model to fields in Iterable. To learn more, read Smart Ingest Field Mapping Overview.
Configure sync settings to handle your data, your way.
Set the sync on a schedule. You can run the sync as often as every 15 minutes.
Import data to Iterable
Running a sync initializes a Smart Ingest pipeline between your data warehouse and Iterable. Smart Ingest detects changes in your source data and syncs only new and updated records to Iterable with each sync run.
Data is imported to Iterable in batches, and you can monitor the sync status and performance in the Iterable UI.
The first sync run initializes change data capture for subsequent syncs, and captures all rows from your data model. Once the first sync is complete, subsequent syncs send only new (changed) data syncs to Iterable, providing streamlined performance.
To learn more about sync runs and other factors that impact sync performance, read Optimizing Smart Ingest Sync Performance.
Monitor your sync runs
Keep an eye on your sync status and performance, so you can resolve problems or fine-tune your sync when you need to.
To learn more, read Monitoring Smart Ingest Syncs.
Make changes when you need to
You can update your sync to:
- Add new fields.
- Add, update, or remove field mappings.
- Change sync settings.
- Change or disable the sync schedule.
- Adjust performance with changes like batch size and sync engine.
Some changes can impact the performance of your sync, so it's important to understand how changes can affect your sync before you make them.
To learn more, read Editing and Testing a Smart Ingest Sync.
Use your data in Iterable
Once your sync begins importing to Iterable, that data is available to use for personalizations, segmentation, journeys, and more.
Supported data sources
To use Smart Ingest, your organization needs to have an established data source that contains data you want to use in Iterable.
Smart Ingest can import from these providers:
- Amazon Athena
- Amazon Redshift
- Amazon S3
- Azure Blob Storage
- Azure Synapse
- ClickHouse
- Databricks
- Elasticsearch
- Firebolt
- Google BigQuery
- Google Cloud Storage
- Microsoft SQL Server
- MongoDB
- MySQL and MariaDB
- Oracle
- PlanetScale
- PostgreSQL
- Rockset
- SFTP (CSV files only)
- SingleStore
- Snowflake
- Teradata Vantage
- Trino
Sync types
Smart Ingest can sync different types of data to Iterable. To learn more about each sync type, read the associated article:
- Users
- Events
- Catalog
- Static Lists
- Email Address Changes
- Deleting, Forgetting, and Unforgetting Users
Limitations
Smart Ingest has some limitations that you should be aware of.
Data limits
Smart Ingest doesn't have explicit data limits, but the performance of your sync can be affected by the size of your data model, the number of records, and the complexity of your data.
It is possible, for example, to sync millions of records in a single sync, but this can impact performance because high-volume syncs take longer to complete. This would limit how often you can run the sync.
To learn about the factors that impact Smart Ingest performance, read Optimizing Smart Ingest Sync Performance.
Number of syncs
By default, an Iterable organization can utilize up to five syncs across all projects in production environments. Once your account has five syncs, the ability to create a new sync is disabled.
Syncs that run at least once before they are deleted still count towards this limit until the next calendar month.
For additional capacity, talk to your customer success manager to discuss adding on more Smart Ingest syncs to your plan.
Syncs that run in test environments don't count towards the limit. Let your customer success manager know if you have a test project that needs to be excluded from this limit. Smart Ingest syncs in test projects are set up exactly the same as in production projects.
Schedule frequency
The highest frequency that you can schedule a sync is once every 15 minutes.
If you need to send data in real-time to Iterable for event-triggered messages, consider using Iterable’s API instead.
Sync performance and rate limits
The performance of Smart Ingest syncs can vary based on the size of your data model, the number of records, and the complexity of your data. Syncs are subject to rate limits, which can affect the speed of your syncs.
To learn about how Smart Ingest performs, read Optimizing Smart Ingest Sync Performance.
Terms to know
Here are some terms you should know when working with Smart Ingest:
Change data capture (CDC)
Change data capture (CDC) is a process that identifies and captures changes made to data in a database. Smart Ingest uses change-based CDC to detect changes in your data source and sync only new and updated records to Iterable.
Data model
A data model, also called just a model, is a customizable set of data that Smart Ingest imports from your source.
To learn more, read Smart Ingest Data Models Overview.
Data source
The data source is the location of data that you connect to Smart Ingest to import. Smart Ingest supports a variety of data sources, including data warehouses and SFTP.
Destination fields
Destination fields are the fields in Iterable that you map your data model fields to. You can use these fields for personalizations, segmentation, and more.
Field mapping
Field mappings tell Smart Ingest where each field goes in Iterable. You can create field mappings to map fields from your data model to fields in Iterable.
Smart Ingest offers several types of field mappings you can use to transform your data as it's imported to Iterable.
To learn more, read Smart Ingest Field Mapping Overview.
Sync
A sync defines how data is matched between your data source and Iterable, and can run on a set schedule. You can set up multiple syncs to import different types of data.
To learn more about the components of a Smart Ingest sync, read Creating a Smart Ingest Sync.
Sync engine
The sync engine is the part of Smart Ingest that manages the data import process. The sync engine is responsible for syncing data from your source to Iterable. You can configure the sync engine when you connect your source to Smart Ingest.
For some supported data sources, Smart Ingest offers two different sync engines: Basic and Lightning. The Basic sync engine is best for smaller data models and uses read-only access. The Lightning sync engine is optimized for performance and can handle large data models but requires read-write access to your data source.
To learn more, read Optimizing Smart Ingest Sync Performance.
Sync type
The sync type determines what kind of data you sync to in Iterable (Users, Events, Catalog, etc). Each sync type has different requirements and settings.
Want to learn more?
For more information about some of the topics in this article, check out this Iterable Academy course. Iterable Academy is open to everyone — you don't need to be an Iterable customer!