Migrating from Metadata Stores to Catalog

Catalog is an upgrade to Iterable's metadata stores. It provides a web interface for viewing and editing catalog data, various new APIs, collections for dynamic searching of catalog data and template customization, and a #catalog Handlebars helper for referencing specific catalog data in a template.

For a description of the differences between catalog and metadata stores, please read Key differences between catalog and metadata stores.

This document describes how to migrate from metadata stores to catalog.

# Table of contents

# Sunsetting metadata stores

Metadata store APIs and the #metadata Handlebars helper will sunset on March 10, 2020.

Please migrate from the metadata store APIs to the Catalog APIs and adjust your templates to use the #catalog Handlebars helper as soon as possible.

# No need to migrate data from metadata stores to Catalog

It is already possible to use the Catalog web interface, Catalog APIs, and #catalog Handlebars helper to work with data originally defined as part of a metadata store.

Metadata table names, item names, and field names remain the same when viewed as catalogs.

Unless you want to use collections to search metadata store data, there is no need to re-create or manually migrate your metadata store data to a catalog.

The metadata store APIs, Catalog APIs, Catalog web interface, #metadata Handlebars helper, and #catalog Handlebars helper all operate on the same underlying data. Because of this, using the metadata store APIs to modify metadata store data also impacts the analogous catalog data (and vice-versa).

# Using the Catalog web interface

To view your metadata store in Iterable's catalog web interface, navigate to Content > Catalogs.

For example, the following screenshots shows a UsedCars metadata store table that is now available as the UsedCars catalog:

To learn how to use the Catalog web interface, read Using Catalog, Catalog Items, and Defining Data Types for Catalog Item Fields.

# Using the Catalog APIs

Metadata store data is currently available using Catalog APIs, as described below.

The corresponding metadata store and Catalog APIs listed below have different request and response formats—they are not drop-in replacements for one another.

# List available metadata store tables / catalogs

• Metadata store API: GET /api/metadata
• Catalog API: GET /api/catalogs
  • For more information about this API, read Get catalog names

# Create or replace an item in a metadata store / catalog

• Metadata store API: PUT /api/metadata/{table}/{key}
• Catalog API: PUT /api/catalogs/{catalogName}/items/{itemId}
  • For more information about this API, read Create or replace a catalog item

# List the items in a metadata store table / catalog

• Metadata store API: GET /api/metadata/{table}
• Catalog API: GET /api/catalogs/{catalogName}/items
  • For more information about this API, read Get the catalog items for a catalog

# Get a single item from a metadata store / catalog

• Metadata store API: GET /api/metadata/{table}/{key}
• Catalog API: GET /api/catalogs/{catalogName}/items/{itemId}
  • For more information about this API, read Get a specific catalog item

# Delete an item from a metadata store table / catalog

• Metadata store API: DELETE /api/metadata/{table}/{key}
• Catalog API: DELETE /api/catalogs/{catalogName}/items/{itemId}
  • For more information about this API, read Delete a catalog item

# Delete a metadata store table / catalog

• Metadata store API: DELETE /api/metadata/{table}
• Catalog API: DELETE /api/catalogs/{catalogName}
  • For more information about this API, read Delete a catalog

# Using the catalog Handlebars helper

Templates can use a new Handlebars helper, #catalog, to reference previously existing metadata store data and new catalog data.

For example, the following text from an Iterable template uses the #metadata Handlebars helper:

{{#metadata "restaurants" favRestaurant as |restaurantInfo|}}
    The restaurant's best item is {{restaurantInfo.bestitem}}
{{/metadata}}

To use the new Handlebars helper, replace #metadata with #catalog (and change the closing tag, too):

{{#catalog "restaurants" favRestaurant as |restaurantInfo|}}
    The restaurant's best item is {{restaurantInfo.bestitem}}
{{/catalog}}

For more information, read Referencing catalog items directly.

# Using collections to search metadata store data

As a campaign is being sent, templates can use collections to search catalog data. The data returned by a collection can then be used to customize the message for the current recipient.

Collections can search catalogs for data related to the specific values stored on a recipient's Iterable profile, making it possible to find dynamic and specifically relevant results.

However, in order for a collection to search a catalog, the fields contained in that catalog's various items must be given explicit data types.

The fields in metadata store items do not have explicit data types. Because of this, their corresponding catalog item fields do not have explicit data types, and they cannot be searched by collections.

The following instructions describe how to create a new catalog, define its field types, and then fill it with data from an existing catalog (by re-uploading its data to the new catalog). Collections can then be used to search the data in this new catalog.

1. Download all the data from your existing metadata store / catalog.

   This example will download the data from a UsedCars catalog.

   To download a catalog's data, use the GET /api/catalogs/{catalogName}/items API. To download all the data, make multiple calls and use the page and pageSize parameters as necessary. For more information about this API, read Get the catalog items for a catalog.

   Example request URL:

   GET https://api.iterable.com/api/catalogs/UsedCars/items?api_key=YOUR_API_KEY

   Example response body:

   {
       "msg": "",
       "code": "Success",
       "params": {
           "catalogItemsWithProperties": [
               {
                   "catalogName": "UsedCars",
                   "itemId": "Hatchback",
                   "size": 193,
                   "lastModified": 1567726021000,
                   "value": {
                       "miles": 94222.9,
                       "year": 2005,
                       "color": "blue",
                       "options": [
                           "automatic",
                           "premium stereo"
                       ],
                       "available": true,
                       "previousOwner": {
                           "firstName": "Kate",
                           "lastName": "Jones",
                           "yearsOwned": 5
                       },
                       "is.convertible": false
                   }
               },
               {
                   "catalogName": "UsedCars",
                   "itemId": "Minivan",
                   "size": 200,
                   "lastModified": 1567726021000,
                   "value": {
                       "miles": 15000.1,
                       "year": 2015,
                       "color": "silver",
                       "options": [
                           "dvd player",
                           "two sliding doors"
                       ],
                       "available": true,
                       "previousOwner": {
                           "firstName": "Ken",
                           "lastName": "Johnson",
                           "yearsOwned": 4
                       },
                       "is.convertible": false
                   }
               }
           ],
           "totalItemsCount": 2
       }
   }
   

2. Save the downloaded data as a backup.

3. Create a new catalog.

   The new catalog will be used to store the data from the original catalog (modified a bit as necessary).

   To create the new catalog, call POST /api/catalogs/{catalogName}, For more information about this API, read Create a catalog. Alternatively, use Iterable's web interface.

   For example, the data above came from a UsedCars catalog, so the new catalog might be called UsedCars2.

   Example request URL:

   POST https://api.iterable.com/api/catalogs/UsedCars2?api_key=YOUR_API_KEY

   This request does not require a JSON body.

4. List the fields in the original metadata store / catalog.

   To do this, use the GET /api/catalogs/{catalogName}/fieldMappings API. For more information about this API, read Get field mappings (data types) for a catalog.

   Example request URL:

   GET https://api.iterable.com/api/catalogs/UsedCars/fieldMappings?api_key=YOUR_API_KEY

   Example response body:

   {
       "msg": "",
       "code": "Success",
       "params": {
           "definedMappings": {},
           "undefinedFields": [
               "is.convertible",
               "is",
               "miles",
               "year",
               "color",
               "options",
               "available",
               "previousOwner"
           ]
       }
   }
   

   NOTES

   • Fields with period in their names, such as is.convertible, cause an extra field to appear in the undefinedFields list returned by the above API call. In this case, the extra field is is (which comes before the period in the is.convertible field name).
   • Nested fields, such as the firstName, lastName, and yearsOwned fields under previousOwner, don't show up in this API call. However, you will need to define field mappings for these nested types, as shown in a subsequent step.

5. Choose a new name for any field that has a period.

   It is only possible to assign explicit data types to catalog fields that do not include periods.

   For example, each car in the UsedCars catalog above has an is.convertible field. This field will need a new name, such as convertible.

   You'll use these modified field names in the next step.

6. Assign fields and data types to the new catalog created above.

   Before creating any data in the new catalog, tell Iterable about the field names and types the new catalog will contain. To do this, use the PUT /api/catalogs/{catalogName}/fieldMappings API. For more information about this API, read Set a catalog's field mappings (data types)

   Example request URL:

   PUT https://api.iterable.com/api/catalogs/UsedCars2/fieldMappings?api_key=YOUR_API_KEY

   Example request body:

   {
       "mappingsUpdates": [
           {
               "fieldName": "convertible",
               "fieldType": "boolean"
           },
           {
               "fieldName": "miles",
               "fieldType": "double"
           },
           {
               "fieldName": "year",
               "fieldType": "long"
           },
           {
               "fieldName": "color",
               "fieldType": "string"
           },
           {
               "fieldName": "options",
               "fieldType": "string"
           },
           {
               "fieldName": "available",
               "fieldType": "boolean"
           },
           {
               "fieldName": "previousOwner",
               "fieldType": "object",
               "children": [
                   {
                       "fieldName": "firstName",
                       "fieldType": "string"
                   },
                   {
                       "fieldName": "lastName",
                       "fieldType": "string"
                   }
               ]
           }
       ]
   }
   

   NOTES

   • In this request, is.convertible has been changed to convertible.
   • The options field contains an array of strings, so this request gives it a string type. Any array field should be given the type of the items it contains.

7. Create a copy of the old catalog's data in the new catalog.

   Now that a new catalog has been created, fill it up with the data retrieved from the old catalog.

   Use the POST /api/catalogs/{catalogName}/items API to create up to 1,000 catalog items at a time. For more information about this API, read Bulk create catalog items

   IMPORTANT

   Be sure to use field names that don't contain periods, as described in previous steps (for example, the request below contains convertible fields instead of is.convertible fields).

   Example request URL:

   POST https://api.iterable.com/api/catalogs/UsedCars2/items?api_key=YOUR_API_KEY

   Example request body:

   {
       "documents": {
           "Minivan": {
               "convertible": false,
               "miles": 15000.1,
               "year": 2015,
               "color": "silver",
               "options": [
                   "dvd player",
                   "two sliding doors"
               ],
               "available": true,
               "previousOwner": {
                   "firstName": "Ken",
                   "lastName": "Johnson",
                   "yearsOwned": 4
               }
           },
           "Hatchback": {
               "convertible": false,
               "miles": 94222.9,
               "year": 2005,
               "color": "blue",
               "options": [
                   "automatic",
                   "premium stereo"
               ],
               "available": true,
               "previousOwner": {
                   "firstName": "Kate",
                   "lastName": "Jones",
                   "yearsOwned": 5
               }
           }
       }
   }
   

8. Adjust your templates to reference the new catalog and field names (if desired).

9. Finally, create collections to search the new catalog, and use them to customize templates.

# Further reading

• Catalog Overview
• Using Catalog
• Catalog Items
• Defining Data Types for Catalog Item Fields
• Collections
• Using Collections in Messages
• Catalog Issues and Limitations
• Catalog APIs
• API Overview and Sample Payloads
• Metadata Stores
• Metadata Store APIs