Creating a Smart Ingest Sync

A sync defines the data that’s being imported from your source data warehouse and where it goes in Iterable. Every sync includes these components:

• A connected source
• A data model
• A sync type and sync preferences that vary for each sync type
• Field mappings that define how columns from your source sync to fields in Iterable
• A schedule that determines how often the sync runs

Once you’ve connected Smart Ingest to a source, you can proceed to build your first sync. This article walks you through creating a new sync in Smart Ingest.

# In this article

# Step 1: Add a new sync

To create a new Smart Ingest sync:

1. Log in to Iterable as a user with the Manage Integrations project permission and open the project you’re working on.

2. Go to Integrations > Smart Ingest.

3. Find the connected source you’re creating a sync for. Click New Sync.

This opens the sync builder, where you can begin by creating a data model.

# Step 2: Create a data model

Regardless of the sync type you’re creating, you need to define a data model for the sync. A data model is a table, SQL query, or CSV file that contains the data you want to import to Iterable.

# Data model setup for database sources

To create a data model for a Smart Ingest sync source with a database connection:

1. Select a modeling method (Table Selector or SQL Query), then click Continue. 

   IMPORTANT

   You can change the sync’s modeling method while you’re working in the sync creation flow, however it can’t be changed after you create the sync—meaning you can’t switch an existing sync from the table selector to a SQL query, or vice-versa, after it is created.

2. Enter a Model Name to identify the data model. This is for display in Iterable.

3. Define your data model using the modeling method you selected:

   • Table Selector

     Select a table name from the available list. The tables shown are scoped to the role and database configured in your data warehouse source configuration.

     (Hint: If you don’t see your table names, click the refresh icon.) 

   • SQL Query

     Enter your SQL query. Smart Ingest's query builder detects your database schema and provides suggestions when you need them.

     (Hint: To format your query for readability, click Beautify.) 

4. Click Preview Results to view a few rows of data from your model.

5. Check the data preview to make sure the records (rows and columns) are the data you expect to see. Each row should represent a single record/entity you want to import to Iterable.

   The results should contain a column with unique values for each row (like user ID, event ID, or email) to be used as the primary key. This is what Smart Ingest uses for change data capture.

6. Select the table’s Primary Key from the drop-down menu

   (Hint: You must click Preview Results before selecting a primary key.)

7. Click Continue to proceed to the next step in the sync setup wizard.

# Data model setup for file-based sources

Data models for file-based sources are created using a CSV file. Smart Ingest treats the CSV file as a table, and your data model is defined by the columns in the CSV file.

To create a data model for a Smart Ingest sync source with a file-based data import:

1. Enter the Relative Path to the file you want to sync data from.

   The file path is relative to the root directory of the data source. For example, if the file is in a folder named data, the path would be data/filename.csv.

2. Select the File Type from the drop-down menu.

   Supported file formats vary based on the source you’re connecting to. Common file types include .CSV and .JSON.

   For a CSV file, also select the delimiter used in the file.

3. Click Preview Results to load a preview of your model's data.

4. Name your model and select its primary key.

   Smart Ingest uses the primary key to determine which rows have been added, changed, or removed since the last sync.

5. Click Continue to proceed to the next step in the sync setup wizard.

# Step 3: Set a sync type, field mappings, and preferences

After creating a data model, you can set the sync type, field mappings, and preferences for your sync.

1. Select the sync type you’re creating. The setup wizard changes dynamically to reflect the preferences and field mappings available for the sync type you selected.

2. Review the sync settings for the sync type you’re building, and make selections to fit your use case. Read each sync type's description to learn about its associated settings.

3. Set up field mappings to define how columns from your source sync to fields in Iterable. Unmapped fields do not sync data to Iterable. To learn more about field mapping, read Smart Ingest Field Mapping Overview.

4. Schedule the sync. Select from available options to define how often the sync should run.

5. Click Finish to save and close.

# Sync schedule options

When you create a sync, you can schedule how often the sync runs, and you can change a sync’s schedule at any time after the sync is created.

The highest frequency that you can schedule a sync is once every 15 minutes.

Syncs scheduled by cron expressions run based on UTC. All other schedules are set based on your local time.

Flexible scheduling options include the following methods:

• Manual (not scheduled) - Syncs run only when you manually trigger the sync from the Smart Ingest UI.

• Interval (basic) - Syncs run at a set interval (such as every 2 hours).

• Custom recurrence (intermediate) - Syncs run at a set interval. You can set precise sync times (such as at 9:00AM), and the schedule can be different each day of the week.

• Cron expression (advanced) - A fully customizable sync schedule using cron expressions.

  Example: 0 0 4 ? * MON,TUE,WED,THU,FRI *

# Next steps

After you create a sync, it’s active and runs as scheduled.

If you set a specific schedule for your sync, it becomes active as soon as you click Finish, and runs as scheduled.

If you set the sync to a Manual schedule, then you can enter the sync overflow menu (three dots) and open View History. From the new screen that pops up, click Run whenever you want to import data from your source.

The first sync run is always going to initialize change data capture and may take additional time. To learn more about sync performance, expectations, and ways to make adjustments, read Optimizing Smart Ingest Sync Performance.