Articles in this section

See more

Amazon Athena + Smart Ingest

Smart Ingest lets you pull data stored in your Amazon Athena data warehouse and import it to Iterable.

NOTE

Smart Ingest can only import data from Amazon Athena.

In this article

Connection requirements

To connect with Smart Ingest, you'll need the following details:

  • AWS credentials
  • Amazon Athena database details

AWS credentials

Your AWS administrator can configure AWS credentials for Smart Ingest. There are two ways to connect:

  • Cross-Account Role - Assign access to a role in another AWS account.

  • Access Key - Use a regular IAM user that has permission to access resources.

Work with your AWS administrator to choose an access method, generate AWS credentials, and then add those credentials in Smart Ingest.

When generating these credentials, Smart Ingest needs a few minimum permissions.

For more information and instructions on how to add AWS credentials to Smart Ingest, read AWS Credentials for Smart Ingest.

Minimum permissions for Amazon Athena

The AWS user that you use to connect to Athena must have the following permissions:

  • Read and list permissions for Athena resources and execute permissions for Athena queries:

    • ListWorkGroups
    • ListDataCatalogs
    • ListDatabases
    • ListTableMetadata
    • StartQueryExecution
    • GetQueryExecution
    • GetQueryResults
    • GetQueryResultsStream

  • Read and write permissions for the output location in S3 to both write the results to S3 and read them back:

    • PutObject
    • ListBucket
    • ListBucketMultipartUploads
    • ListMultipartUploadParts
    • GetBucketLocation
    • GetObject
    • AbortMultipartUpload
    • PutObject

See the AWS managed policy AmazonAthenaFullAccess for more information.

Amazon Athena database details

In addition to AWS credentials, you need the following details to connect to your Amazon Athena data warehouse:

  • Region - The AWS region where your Athena resources are located (selected from a dropdown list).

  • The Athena Workgroup you previously set up in AWS (selected from a drop-down list).

  • Query output location — Athena saves an output file to S3 for each query. You only need to enter this if your workgroup doesn't have an output location configured. This is an S3 path.

  • Data Catalog name (selected from a drop-down list).

  • Database name (selected from a drop-down list).

Connecting to Amazon Athena

To connect Smart Ingest with Amazon Athena:

  1. Log in to Iterable as a user with the Manage Integrations project permission and open the project you’re working on.

  2. Go to Integrations > Smart Ingest.

  3. Click Connect a New Source.

  4. Select Amazon Athena and click Continue.

  5. In Step 1, select the AWS Credentials you want to use to connect.

    If this is your first time connecting to AWS, you need to add AWS credentials. To learn how to generate these credentials and add them in Iterable, read AWS Credentials for Smart Ingest.

    If you already have AWS credentials set up, select them from the drop-down list.

  6. In Step 2, select the Region for your AWS account.

  7. Select the Workgroup you previously set up in AWS.

  8. Enter the Query output location (optional). This is the S3 path where Athena saves the output file for each query. You only need to enter this if your workgroup doesn't have an output location configured.

  9. Select the Data Catalog and Database names from the drop-down lists.

  10. Click Continue. Smart Ingest automatically tests the connection.

  11. When the connection test is successful, click Continue. (If there are problems connecting, click Back and review the connection details for accuracy.)

  12. Add a name for your data source to display in Iterable.

  13. Click Finish.

Next steps

Now that you've connected your data warehouse to Smart Ingest, you can create a sync.

Importing Athena data types

Smart Ingest parses most Athena data types into JavaScript types before sending them to Iterable.

The following Athena data types are imported to Iterable as the string data type:

  • BIGINT
  • ARRAY
  • MAP
  • BINARY
  • STRUCT

To learn more about Athena data types, see AWS's support article Data Types in Amazon Athena.

Troubleshooting

CredentialsError: Missing credentials in configuration

If you receive the error: CredentialsError: Missing credentials in config, it means Smart Ingest can't assume the role required.

This error is often due to an issue with the IAM trust policy. This policy defines which other AWS accounts can assume roles in your account. When you set up AWS credentials in Smart Ingest using a cross-account role, we provide you with Hightouch's Account ID and an additional external ID. You should include Hightouch's Account ID in your role trust policy, and can optionally include the external ID.

NOTE

Smart Ingest is co-developed by Iterable and Hightouch. Hightouch is a data processor for this feature. Smart Ingest data operations and schemas may contain the Hightouch name, but the feature is fully supported by Iterable.

To troubleshoot this error, follow these steps:

  • Ensure that the role exists.
  • Ensure that the role has a trust policy attached (for example, AmazonAthenaFullAccess, or a custom role with minimum permissions assigned).
  • Ensure that the role's trust policy includes Hightouch's AWS account ID (324528010823) as a principal.
  • If the role trust policy includes an external ID check, ensure it matches the external ID generated by Smart Ingest.

If you want to check the external ID but misplaced the external ID that Smart Ingest generated when you connected, you need to create new credentials in Smart Ingest. Smart Ingest only displays this value once during source setup.

Related articles