Articles in this section

See more

Amazon S3 + Smart Ingest

Smart Ingest lets you pull data from CSV and JSON files stored in an Amazon S3 bucket and import to Iterable. To get started, you need an S3 bucket and AWS credentials.

NOTE

Smart Ingest can only import data from Amazon S3.

In this article

Connection requirements

To connect with Smart Ingest, you need the following details:

  • AWS credentials
  • Amazon Athena database details

AWS credentials

To access your data, Smart Ingest needs a few minimum permissions.

Your AWS administrator can configure AWS credentials for Smart Ingest. There are two ways to connect:

  • Cross-Account Role - Assign access to a role in another AWS account.

  • Access Key - Use a regular IAM user that has permission to access resources.

Work with your AWS administrator to choose an access method, generate AWS credentials, and then add those credentials in Smart Ingest.

For instructions and more information to add AWS credentials to Smart Ingest, read AWS Credentials for Smart Ingest.

Minimum permissions for Amazon S3

Smart Ingest needs the following IAM actions to import files from your S3 bucket:

  • s3:GetObject - Grants permission to retrieve objects from Amazon S3.
  • s3:ListBucket - Grants permission to list some or all the objects in an Amazon S3 bucket (up to 1000).

You can edit the following JSON sample to create your IAM policy for S3:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
              "s3:GetObject",
              "s3:ListBucket"
            ],
            "Resource": [
              "arn:aws:s3:::${bucketName}/*",
              "arn:aws:s3:::${bucketName}"
            ]
        }
    ]
}

Amazon S3 details

In addition to AWS credentials, you need the following details to connect to your Amazon S3 bucket:

  • Region - The AWS region where your S3 bucket is located (selected from a drop-down list).

  • Bucket Name - The name of the S3 bucket where your data files are stored. (Not a URL.)

  • Path - The relative path to the CSV or JSON file you want to sync data from, like path/prefix/file.csv. The path shouldn't contain the bucket name.

  • Use Last Modified - Check this box to sync the last modified file with a given prefix. Enter a prefix like path/prefix in the path field. To sync the last modified file in the entire bucket, check the box and leave the path field blank.

Connecting to Amazon S3

To connect Smart Ingest with Amazon S3:

  1. Log in to Iterable as a user with the Manage Integrations project permission and open the project you’re working on.

  2. Go to Integrations > Smart Ingest.

  3. Click Connect a New Source.

  4. Select Amazon S3 and click Continue.

  5. In Step 1, select the AWS Credentials you want to use to connect.

    If this is your first time connecting to AWS, you need to add AWS credentials. To learn how to generate these credentials and add them in Iterable, read AWS Credentials for Smart Ingest.

    If you already have AWS credentials set up, select them from the drop-down list.

  6. In Step 2, select the Region for your S3 Bucket.

  7. Enter the Bucket Name for the S3 bucket you want to connect. This should be the name of the bucket where your data files are stored, not a URL.

  8. Click Continue. Smart Ingest automatically tests the connection.

  9. When the connection test is successful, click Continue. (If there are problems connecting, click Back and review the connection details for accuracy.)

  10. Add a name for your data source to display in Iterable.

  11. Click Finish.

Next steps

You've now connected your storage bucket to Smart Ingest. The next thing you can do is create a sync.

Importing data from Amazon S3

Smart Ingest can sync data from a specific CSV or JSON file in your S3 bucket, as well as from the last modified file that matches a given prefix.

When setting up a new sync, you need to provide the following details:

  • Path - The relative path to the CSV or JSON file you want to sync data from, like path/prefix/file.csv. The path shouldn't contain the bucket name.

  • Use Last Modified - Check this box to sync the last modified file with a given prefix.

    To sync the last modified file with a given prefix, check the Use Last Modified box and enter a prefix like path/prefix in the path field. See Amazon's S3 docs on prefixes for examples of how prefixes match objects.

    To sync the last modified file in the entire bucket, check the Use Last Modified box and leave the path field blank.

CSV and JSON file requirements

Smart Ingest can import CSV and JSON files from Amazon S3. Make sure your files meet certain requirements, including:

  • Formatting for the supported file type (CSV or JSON). To learn about file requirements for Smart Ingest, read Smart Ingest Data Models Overview.

  • Field requirements vary depending on the sync type you're setting up. To learn more, read the Smart Ingest sync guide for the sync type you're setting up.

Related articles