Articles in this section

Configuring Your SFTP Destination

# In this article

# Prerequisites

By default, SFTP uses keypair authentication for access. To configure your destination, you'll need a provided public key formatted as the algorithm, the key body (Base64-encoded, beginning with AAAA), and an optional comment (like an email address).

# Step 1: Create a user on the SFTP server

Log in to the SFTP server and complete the steps below.

  1. Create group sftpwriter:
sudo groupadd sftpwriter
  1. Create user sftpwriter:
sudo useradd -m -g sftpwriter sftpwriter
  1. Switch to the sftpwriter user:
sudo su - sftpwriter
  1. Create the .ssh directory:
mkdir ~/.ssh
  1. Set permissions:
chmod 700 ~/.ssh
  1. Navigate to the .ssh directory:
cd ~/.ssh
  1. Create the authorized_keys file:
touch authorized_keys
  1. Set permissions:
chmod 600 authorized_keys
  1. Add the public key to the authorized_keys file. The key (including the key and comment) should be all on one line in the file, without linebreaks.
echo "ssh-rsa <ssh_public_key_beginning_with_AAAA> sftpwriter-public-key" > authorized_keys

# Step 2: Add your destination

Use the following details to complete the connection setup: host name, folder name, username, port, and preferred delimiter character.

# Enabling write permissions at the SFTP root (required)

In addition to write access within your configured <folder>, this destination writes per-transfer manifest files under a _manifests/ directory created at the root of the SFTP home/path. These files allow downstream systems to detect when a transfer is complete. Ensure that the SFTP user can create and write to _manifests at that root (even if your data is in a subfolder). Refer to the FAQs in this article to learn more.

# Enabling PGP encryption (optional)

SFTP files are encrypted in-transit, by virtue of the SFTP protocol. Iterable offers an optional, additional layer of encryption at-rest for SFTP files using PGP encryption.

To enable PGP encryption, generate your own PGP public/private key pair, and provide the public key on a per-destination basis (one for each PGP-enabled destination).

The public key must be provided in ASCII armored format, beginning with the header line: -----BEGIN PGP PUBLIC KEY BLOCK----- and ending with the tail line: -----END PGP PUBLIC KEY BLOCK-----.

For security reasons, only RSA and ECC keys are supported, with the following requirements:

  • RSA keys must have a key size of 2,048 bits or more.
  • ECC keys must have a key size of 224 bits or more.

If PGP encryption is enabled, both files will be encrypted in the PGP binary format with a .pgp file extension appended to their filenames. For example, encrypted CSV files will have filenames like your_data.csv.pgp.

It's not possible to enable/disable the PGP encryption settings of an existing destination. Instead, you must create a new destination with the new PGP configuration.

# Permissions checklist

Verify that the following permissions are applied in your project:

  • SFTP user created with SSH key-based authentication.
  • Provided public key added to ~/.ssh/authorized_keys for the SFTP user.
  • SFTP user has write access to the configured folder.
  • SFTP user can create and write to _manifests/ at the SFTP home/path root.
  • Firewall or network perimeter allows the service's egress IP to connect on port 22.
  • (If using PGP encryption) PGP public key in ASCII armored format is ready to provide during destination configuration. RSA keys must be 2,048 bits or larger.

# Frequently Asked Questions

Q: How will the data appear in my SFTP server?

A: The data will be loaded with the configured file format (Parquet, CSV, or JSON/JSONL) in a predictable folder structure that can be easily parsed by downstream systems.

    sftpwriter_home_folder/
    ├─ some_provided_folder/
    │  ├─ some_table_a/
    │  │  ├─ dt=2024-01-01/
    │  │  │  ├─ 0_20240101181004.csv
    │  │  │  ├─ 1_20240101184002.csv
    │  │  ├─ dt=2024-01-02/
    │  │  │  ├─ 0_20240102180123.csv
    │  │  ├─ dt=2024-01-03/
    │  │  │  ├─ 0_20240103182145.csv
    │  ├─ some_table_b/
    │  │  ├─ dt=2024-01-01/
    │  │  │  ├─ 0_20240101186004.csv
    │  │  ├─ dt=2024-01-02/
    │  │  │  ├─ 0_20240102185123.csv
    │  │  ├─ dt=2024-01-03/
    │  │  │  ├─ 0_20240103187145.csv

Q: How is the SFTP connection secured?

A: Use SSH key-based authentication for a dedicated, least-privileged SFTP user. Restrict access to only the required directories (e.g., chroot), and allowlist the service's static egress IP at your network perimeter.

Q: What if I cannot provide write access at the root of my SFTP server?

A: Write privileges at the root are expected in order to write manifest files. Note that Iterable recommends using a dedicated SFTP endpoint for data exports for security and data isolation best practices. If write permissions cannot be provided at the root, you can chroot your writer user to your target folder, and instead configure the SFTP connection without a folder name. Both the manifest files and data will then be written at this target folder.

Q: What file formats are supported?

A: Parquet (default/recommended), CSV, and JSON/JSONL.

Q: How are large datasets written?

A: Files are automatically split; multiple files may be written per model per transfer.

Q: How do I know when a transfer completed?

A: Each transfer writes a manifest JSON file per model under _manifests/ at the root. Files follow the pattern: _manifests/<model_name>/dt=<transfer_date>/manifest_{transfer_id}.json. Use these manifests to trigger downstream processing.

Q: Why do I sometimes see duplicates?

A: File-based destinations are append-oriented. The change-detection process uses a lookback window to prevent missed records, which can create duplicates across adjacent transfers. Downstream pipelines can deduplicate by primary key prioritizing rows in the most recent transfer window.

Q: Can I provide my own SSH/SFTP public key? Where is the private key stored?

A: You can't provide your own public key for security reasons. The private key is securely generated and stored in Iterable's system and is never shared externally.

Q: Can I provide my own PGP public key?

A: Yes, you can optionally provide your own PGP public key for Iterable to encrypt the SFTP files with, on a per-destination basis (one for each destination). The public key must be provided in ASCII armored format.

Q: Is this PGP public key an alternative to the SSH/SFTP public key?

A: No, the PGP key is not an alternative to the mandatory SSH/SFTP key. The PGP key is optional, while the SFTP key is mandatory. The two kinds of keys serve functionally distinct purposes. The SSH/SFTP key is used for authentication and encryption in-transit. On the other hand, the PGP key is not used for authentication, and is instead used to add another layer of encryption of files in transit, and importantly, to ensure that the files remain encrypted at rest in the SFTP destination.

Q: When exactly does the PGP encryption occur, when enabled?

A: The PGP encryption is applied in Iterable, after the data is pulled from the source. The data remains PGP-encrypted in-transit between Iterable and the SFTP server, and stays PGP-encrypted at rest in the SFTP destination.

Q: Will the PGP-encrypted files be generated in the armored (ASCII) format, or in binary?

A: Files are encrypted in the binary format only.

Q: Are manifest files also encrypted, or only the landed data?

A: If PGP encryption is enabled, both the manifest files and landed data will be encrypted.

Q: Are there any constraints on the PGP public keys?

A: RSA key sizes must be 2,048 bits or more. ECC key sizes must be 224 bits or more.

Related articles