{"componentChunkName":"component---src-templates-markdown-template-jsx","path":"/guides/data-warehouse/warehouse-sync/setup/","result":{"data":{"pageMetadata":{"title":"Warehouse Sync User Guide","path":"/guides/data-warehouse/warehouse-sync/setup/","sourceCode":null,"metadataParent":{"title":"Warehouse Sync","path":"/guides/data-warehouse/warehouse-sync/","sourceCode":null,"hasVersionPicker":null,"metadataChildren":[{"title":"Warehouse Sync User Guide","path":"/guides/data-warehouse/warehouse-sync/setup/","order":1,"label":null},{"title":"Historical Data and Warehouse Sync","path":"/guides/data-warehouse/warehouse-sync/historical-data/","order":2,"label":null}],"metadataParent":{"title":"Data Warehouse","path":"/guides/data-warehouse/","metadataChildren":[{"title":"Composable Audiences","path":"/guides/data-warehouse/composable-audiences/","order":1,"label":null},{"title":"Warehouse Sync","path":"/guides/data-warehouse/warehouse-sync/","order":4,"label":null}],"metadataParent":{"title":"Guides","path":"/guides/","metadataChildren":[{"title":"Platform Overview","path":"/guides/platform-overview/","order":0.5,"label":null},{"title":"Getting Started","path":"/guides/getting-started/","order":1,"label":null},{"title":"mParticle Agent","path":"/guides/mparticle-agent/","order":1.5,"label":null},{"title":"Oversight","path":"/guides/oversight/","order":2,"label":null},{"title":"Data Platform","path":"/guides/data-platform/","order":3,"label":null},{"title":"Data Warehouse","path":"/guides/data-warehouse/","order":4,"label":null},{"title":"Customer 360","path":"/guides/customer-360/","order":5,"label":null},{"title":"Analytics","path":"/guides/analytics/","order":6,"label":null},{"title":"Audiences","path":"/guides/audiences/","order":7,"label":null},{"title":"Settings","path":"/guides/settings/","order":8,"label":null},{"title":"Troubleshooting mParticle","path":"/guides/troubleshooting/","order":10,"label":null},{"title":"Default Service Limits","path":"/guides/default-service-limits/","order":11,"label":null},{"title":"Glossary","path":"/guides/glossary/","order":12,"label":null}],"metadataParent":null}}}},"markdownRemark":{"html":"

Warehouse Sync is mParticle’s reverse-ETL solution, allowing you to use an external warehouse as a data source for your mParticle account. By ingesting data from your warehouse using either one-time, on-demand, or recurring syncs, you can benefit from mParticle’s data governance, personalization, and prediction features without having to modify your existing data infrastructure.

\n

\n \n \n \n \n \n \n \n Supported warehouse providers

\n

You can use Warehouse Sync to ingest both user and event from the following warehouse providers:

\n\n\n

\n \n \n \n \n \n \n \n Warehouse Sync setup overview

\n
    \n
  1. Prepare your data warehouse before connecting it to mParticle
    2. \n
  2. Create an input feed in your mParticle account for your warehouse
    3. \n
  3. Connect your warehouse to your new mParticle input feed
    4. \n
  4. Specify the data you want to ingest into mParticle by creating a SQL data model
    5. \n
  5. \n

    Map your warehouse data to fields in mParticle

    \n\n
    6. \n
  6. Configure when and how often data is ingested from your warehouse
    7. \n
\n

\n \n \n \n \n \n \n \n Observability for Warehouse Sync

\n

mParticle’s Observability offers support for tracing data ingested into your account through a Warehouse Sync pipeline. Simply select the pipeline feed you want to trace from the list of Feeds when creating a new trace configuration.

\n

Trace configurations for Warehouse Sync pipelines will generate a new unique trace for each event batch generated by your pipeline within the timeframe you specify in your configuration. Like other trace configurations, pipeline traces display spans for each processing stage your data passes through once it’s ingested into mParticle.

\n

Warehouse Sync traces do not currently provide row-specific details from your source database or details for diagnosing errors encountered during data ingestion. The current version of Warehouse Sync tracing is intended to help diagnose and troubleshoot issues encountered post-ingestion.

\n

\n \n \n \n \n \n \n \n Warehouse Sync API

\n

All functionality of Warehouse Sync is also available from an API. To learn how to create and manage syncs between mParticle and your data warehouse using the API, visit the mParticle developer documentation.

\n

\n \n \n \n \n \n \n \n Warehouse Sync setup tutorial

\n

\n \n \n \n \n \n \n \n Prerequisites

\n

Before beginning the warehouse sync setup, obtain the following information for your mParticle account. Depending on which warehouse you plan to use, you will need some or all of these values:

\n\n

\n \n \n \n \n \n \n \n How to find your mParticle Org, Account, Region, and Pod AWS Account IDs

\n

To find your Org ID, Account ID, Pod ID, and Pod AWS Account ID:

\n
    \n
  1. Log into the mParticle app at app.mparticle.com
    2. \n
  2. \n

    From your web browser’s toolbar, navigate to view the page source for the mParticle app.

    \n\n
    3. \n
\n

\n

\n \n \n \n \n \n \n \n 1 Prepare your data warehouse

\n

Work with your warehouse administrator or IT team to ensure your warehouse is reachable and accessible by mParticle.

\n
    \n
  1. Whitelist the mParticle IP address range so your warehouse will be able to accept inbound API requests from mParticle.
    2. \n
  2. Ask your database administrator to perform the following steps in your warehouse to create a new role that mParticle can use to access your database. Select the correct tab for your warehouse (Snowflake, Google BigQuery, Amazon Redshift, or Databricks) below.
    3. \n
\n
\n
\n

\n \n \n \n \n \n \n \n Authentication

\n\n

If you are creating a new Snowflake pipeline, you must use key pair authentication. If you have an existing Snowflake mParticle pipeline with password authentication, please migrate to key pair authentication.

\n

The process for setting up key pair authentication takes two steps:

\n
    \n
  1. Create a new key pair by following the instructions in Key Management.
    2. \n
  2. Assign the public key you created to your Snowflake user. You will do this when setting up Snowflake as described in the instructions below.
    3. \n
\n
\n \n \n \n \n \n \n \n Migrate to key pair authentication
\n

If you have an existing Snowflake configuration that you want to migrate to key pair authentication:

\n
    \n
  1. Create a new key pair.
    2. \n
  2. From your Snowflake console, assign the public key to your Snowflake user by running the following, where mparticle_user is replaced with the name of user you created for your configuration and YOUR-PUBLIC-KEY is replaced with your new public key:
    3. \n
\n
// If you are rolling over to a new RSA key, you can use RSA_PUBLIC_KEY_2 instead of RSA_PUBLIC_KEY\nALTER USER mparticle_user SET RSA_PUBLIC_KEY='YOUR-PUBLIC-KEY';
\n\n\n
    \n
  1. In mParticle, navigate to Data Platform > Setup > Inputs and select the Feeds tab.
    2. \n
  2. Under Snowflake, select the configuration you want to migrate, and click Settings.
    3. \n
  3. Remove the password you have entered under Password, and enter the Key ID for the key pair you created in mParticle under Key ID.
    4. \n
  4. Click Save.
    5. \n
\n

\n \n \n \n \n \n \n \n 1.1 Create a new user

\n

First, you must create a user in your Snowflake account that mParticle can use to access your warehouse.

\n
    \n
  1. Log into your Snowflake account at app.snowflake.com
    2. \n
  2. Click + Create in the left hand navigation, and select SQL Worksheet.
    3. \n
  3. Select the database you want to connect to mParticle, and run the following SQL statement from the console:
    4. \n
\n
CREATE OR REPLACE USER {{user_name}} RSA_PUBLIC_KEY={{YOUR_PUBLIC_KEY}};
\n

Where:

\n\n

\n \n \n \n \n \n \n \n 1.2 Create a new role

\n

mParticle recommends creating a dedicated role for your mParticle warehouse sync user created in step 1.1 above.

\n
    \n
  1. \n

    From the Users & Roles page of your Snowflake account’s Admin settings, select the Roles tab.

    \n

    \n
    2. \n
  2. \n

    Click the + Role button.

    \n

    \n
    3. \n
  3. \n

    Enter a name for your new role, and select the set of permissions to assign to the role using the under Grant to role dropdown.

    \n

    \n
    4. \n
  4. Enter an optional description under Comment, and click Create Role.
    5. \n
  5. \n

    Select your new role from the list of roles, and click the Grant to User button.

    \n

    \n
    6. \n
  6. \n

    Under User to receive grant, select the new user you created in step 1.1, and click Grant.

    \n

    \n
    7. \n
\n

Before continuing, make sure to save the following information, as you will need to refer to this when connecting Data Warehouse to your Snowflake account:

\n\n
\n
\n\n

\n \n \n \n \n \n \n \n 1.1 Create a new service account for mParticle

\n
    \n
  1. Go to console.cloud.google.com, log in, and navigate to IAM & Admin > Service Accounts.
    2. \n
  2. Select Create Service Account.
    3. \n
  3. Enter a new identifier for mParticle in Service account ID. In the example below, the email address is the service account ID. Save this value for your Postman setup.
    4. \n
\n

\n
    \n
  1. Under Grant this service account access to project, select BigQuery Job User under the Role dropdown menu, and click DONE.
    2. \n
\n

\n
    \n
  1. Select your new service account and navigate to the Keys tab.
    2. \n
  2. Click ADD KEY and select Create new key. The value for service_account_key will be the contents of the generated JSON file. Save this value for your Postman setup.
    3. \n
\n

\n

\n \n \n \n \n \n \n \n 1.2 Identify your BigQuery warehouse details

\n

Navigate to your BigQuery instance from console.cloud.google.com.

\n

\n\n

\n \n \n \n \n \n \n \n 1.3 Grant access to the dataset in BigQuery

\n
    \n
  1. From your BigQuery instance in console.cloud.google.com, click Sharing and select Permissions.
    2. \n
  2. Click Add Principle.
    3. \n
  3. Assign two Roles, one for BigQuery Data Viewer, and one for BigQuery User.
    4. \n
  4. Click Save.
    5. \n
\n

\n
\n
\n\n
    \n
  1. Navigate to your AWS Console, log in with your administrator account, and navigate to your Redshift cluster details.
    2. \n
  2. Run the following SQL statements to create a new user for mParticle, grant the necessary schema permissions to the new user, and grant the necessary access to your tables/views.
    3. \n
\n
-- Create a unique user for mParticle\nCREATE USER {{user_name}} WITH PASSWORD '{{unique_secure_password}}'\n\n-- Grant schema usage permissions to the new user\nGRANT USAGE ON SCHEMA {{schema_name}} TO {{user_name}}\n\n-- Grant SELECT privilege on any tables/views mP needs to access to the new user\nGRANT SELECT ON TABLE {{schema_name}}.{{table_name}} TO {{user_name}}
\n
    \n
  1. Navigate to the Identity And Access Management (IAM) dashboard, select Roles under the left hand nav bar, and click Create role.
    2. \n
\n

\n
    \n
  1. In Step 1 Select trusted entity, click AWS service under Trusted entity
    2. \n
  2. Select Redshift from the dropdown menu titled “Use cases for other AWS services”, and select Redshift - Customizable. Click Next.
    3. \n
\n

\n
    \n
  1. In Step 2 Add permissions, click Create Policy.
    2. \n
\n

\n
    \n
  1. Click JSON in the policy editor, and enter the following permissions before clicking Next.
    2. \n
\n\n
{\n    \"Statement\": [\n        {\n            \"Action\": \"sts:AssumeRole\",\n            \"Effect\": \"Allow\",\n            \"Resource\": \"arn:aws:iam::{{mp_pod_aws_account_id}}:role/ingest-pipeline-data-external-{{mp_org_id}}-{{mp_acct_id}}\",\n            \"Sid\": \"\"\n        }\n    ],\n    \"Version\": \"2012-10-17\"\n}
\n

\n
    \n
  1. Enter a meaningful name for your new policy, such as mparticle_redshift_assume_role_policy, and click Create policy.
    2. \n
\n

\n
    \n
  1. Return to the Create role tab, click the refresh button, and select your new policy. Click Next.
    2. \n
\n

\n
    \n
  1. Enter a meaningful name for your new role, such as mparticle_redshift_role, and click Create role.
    2. \n
\n

\n

Your configuration will differ between Amazon Redshift and Amazon Redshift Serverless. To complete your configuration, follow the appropriate steps for your use case below.

\n

Make sure to save the value of your new role’s ARN. You will need to use this when setting up Postman in the next section.

\n

\n \n \n \n \n \n \n \n Amazon Redshift (not serverless)

\n
    \n
  1. Navigate to your AWS Console, then navigate to Redshift cluster details. Select the Properties tab.
    2. \n
\n

\n
    \n
  1. Scroll to Associated IAM roles, and select Associate IAM Roles from the Manage IAM roles dropdown menu.
    2. \n
\n

\n
    \n
  1. Select the new role you just created. The name for the role in this example is mparticle_redshift_role.
    2. \n
\n

\n \n \n \n \n \n \n \n Amazon Redshift Serverless

\n
    \n
  1. Navigate to your AWS Console, then navigate to your Redshift namespace configuration. Select the Security & Encryption tab, and click Manage IAM roles.
    2. \n
\n

\n
    \n
  1. Select Associate IAM roles from the Manage IAM roles dropdown menu.
    2. \n
  2. Select the new role you just created. The name for the role in this example is mparticle_redshift_role.
    3. \n
\n

\n
\n
\n

Warehouse Sync uses the Databricks-to-Databricks Delta Sharing protocol to ingest data from Databricks into mParticle.

\n\n

Complete the following steps to prepare your Databricks instance for Warehouse Sync.

\n

\n \n \n \n \n \n \n \n 1. Enable Delta Sharing

\n
    \n
  1. Log into your Databricks account and navigate to the Account Admin Console. You must have Account Admin user privileges.
    2. \n
  2. Click Catalog from the left hand nav bar, and select the metastore you want to ingest data from.
    3. \n
  3. Select the Configuration tab. Under Delta Sharing, check the box labeled “Allow Delta Sharing with parties outside your organization”.
    4. \n
  4. Find and save your Databricks provider name: the value displayed under Organization name. You will use this value for your provider name when creating the connection between mParticle Warehouse Sync and Databricks.
    5. \n
\n

\n

\n \n \n \n \n \n \n \n 2. Configure a Delta Sharing recipient for mParticle

\n
    \n
  1. From the Unity Catalog Explorer in your Databricks account, click the Delta Sharing button, and select Shared by me.
    2. \n
  2. Click New Recipient in the top right corner.
    3. \n
  3. Within the Create a new recipient window, enter mParticle_{YOUR-DATA-POD} under Recipient name where {YOUR-DATA-POD} is either us1, us2, eu1, or au1 depending on the location of the data pod configured for your mParticle account.
    4. \n
  4. \n

    In Sharing identifier, enter one of the following identifiers below, depending on the location of your mParticle account’s data pod:

    \n
      \n
    • US1: aws:us-east-1:e92fd7c1-5d24-4113-b83d-07e0edbb787b
      • \n
    • US2: aws:us-east-1:e92fd7c1-5d24-4113-b83d-07e0edbb787b
      • \n
    • EU1: aws:eu-central-1:2b8d9413-05fe-43ce-a570-3f6bc5fc3acf
      • \n
    • AU1: aws:ap-southeast-2:ac9a9fc4-22a2-40cc-a706-fef8a4cd554e
      • \n
    \n
    5. \n
\n

\n

\n \n \n \n \n \n \n \n 3. Share your Databricks tables and schema with your new Delta Sharing recipient

\n
    \n
  1. From the Unity Catalog Explorer in your Databricks account, click the Delta Sharing button.
    2. \n
  2. Click Share data in the top right.
    3. \n
  3. \n

    Within the Create share window, enter mparticle_{YOUR-MPARTICLE-ORG-ID}_{YOUR-MPARTICLE-ACCOUNT-ID} under Share name where {YOUR-MPARTICLE-ORG-ID} and {YOUR-MPARTICLE-ACCOUNT-ID} are your mParticle Org and Account IDs.

    \n
      \n
    • To find your Org ID, log into the mParticle app. View the page source. For example, in Google Chrome, go to View > Developer > View Page Source. In the resulting source for the page, look for “orgId”:xxx. This number is your Org ID.
      • \n
    • Follow a similar process to find your Account ID (“accountId”:yyy) and Workspace ID (“workspaceId”:zzz).
      • \n
    \n
    4. \n
\n

\n
    \n
  1. Click Save and continue at the bottom right.
    2. \n
  2. In the Add data assets section, select the assets to add to the schemas and tables you want to send to mParticle. Make sure to remember your schema name: you will need this value when configuring your Databricks feed in mParticle.
    3. \n
\n

\n
    \n
  1. Click Save and continue at the bottom right until you reach the Add recipients step. (You can skip the Add notebooks step.)
    2. \n
  2. In the Add recipients step, make sure to add the new mParticle recipient you created in Step 2.
    3. \n
  3. Finally, click the Share data button at the bottom right.
    4. \n
\n

\n

\n \n \n \n \n \n \n \n Unsupported data types between mParticle and Databricks

\n

Databricks Delta Sharing does not currently support the TIMESTAMP_NTZ data type.

\n

Other data types that are not currently supported by the Databricks integration for Warehouse Sync (for both user and events data) include:

\n\n

If you are ingesting events data through Warehouse Sync, the following data types are unsupported:

\n\n

While multi-dimensional, or nested, arrays are unsupported, you can still ingest simple arrays with events data.

\n
\n
\n

\n \n \n \n \n \n \n \n 2 Create a new warehouse input

\n
    \n
  1. Log into your mParticle account.
    2. \n
  2. Navigate to Data Platform > Setup > Inputs in the left-hand nav bar and select the Feeds tab.
    3. \n
  3. Under Add Feed Input, search for and select your data warehouse provider.
    4. \n
\n

\n

You can also create a new warehouse input from the Integrations Directory:

\n
    \n
  1. Log into your mParticle account.
    2. \n
  2. Navigate to Data Platform > Setup > Directory.
    3. \n
  3. Search for either Google BigQuery, Snowflake, Amazon Redshift, or Databricks.
    4. \n
  4. After clicking the integration tile, check Input Feed and click Configure.
    5. \n
\n\n

\n

After selecting your warehouse provider, the Warehouse Sync setup wizard will open where you will:

\n
    \n
  1. Enter your warehouse details.
    2. \n
  2. Create your data model.
    3. \n
  3. Create any necessary mappings between your warehouse data and mParticle fields.
    4. \n
  4. Enter your sync schedule settings.
    5. \n
\n

\n \n \n \n \n \n \n \n 3 Connect warehouse

\n

The setup wizard presents different configuration options depending on the warehouse provider you select. Use the tabs below to view the specific setup instructions for Amazon Redshift, Google BigQuery, and Snowflake.

\n
\n
\n

\n \n \n \n \n \n \n \n Connect to Amazon Redshift

\n

\n

\n \n \n \n \n \n \n \n 3.1 Enter a configuration name

\n

The configuration name is specific to mParticle and will appear in your list of warehouse inputs. You can use any configuration name, but it must be unique since this name is used when configuring the rest of your sync settings.

\n

\n \n \n \n \n \n \n \n 3.2 Enter your Amazon Redshift database name

\n

The database name identifies your database in Amazon Redshift. This must be a valid Amazon Redshift name, and it must exactly match the name for the database you want to connect to.

\n

\n \n \n \n \n \n \n \n 3.3 Enter your Redshift AWS IAM Role ARN

\n

This is the unique string used to identify an IAM (Identity and Access Management) role within your AWS account. AWS IAM Role ARNs follow the format arn:aws:iam::account:role/role-name-with-path where account is replaced with your AWS account number, role is replaced with the role type in AWS, and role-name-with-path is replaced with the name and location for the role in your AWS account.

\n

Learn more about AWS identifiers in IAM identifiers in the AWS documentation.

\n

\n \n \n \n \n \n \n \n 3.4 Enter your warehouse host and port

\n

mParticle uses the host name and port number when connecting to, and ingesting data from, your warehouse.

\n

\n \n \n \n \n \n \n \n 3.5 Enter your Amazon Redshift credentials

\n

Provide the username and password associated with the AWS IAM Role you entered in step 1.4. mParticle will use these credentials when logging into AWS before syncing data.

\n
\n
\n

\n \n \n \n \n \n \n \n Connect to Google BigQuery

\n

\n

\n \n \n \n \n \n \n \n 3.1 Enter a configuration name

\n

The configuration name is specific to mParticle and will appear in your list of warehouse inputs. You can use any configuration name, but it must be unique since this name is used when configuring the rest of your sync settings.

\n

\n \n \n \n \n \n \n \n 3.2 Enter your BigQuery project ID

\n

Enter the ID for the project in BigQuery containing the dataset. You can find your Project ID from the Google BigQuery Console.

\n

\n \n \n \n \n \n \n \n 3.3 Enter your BigQuery Dataset ID

\n

Enter the ID for the dataset you’re connecting to. You can find your Dataset ID from the Google BigQuery Console.

\n

\n \n \n \n \n \n \n \n 3.4 Enter the region

\n

Enter the region where your dataset is localized. For example, aws-us-east-1 or aws-us-west-2. You can find the region for your data set from the Google BigQuery Console.

\n

\n \n \n \n \n \n \n \n 3.5 Add your BigQuery credentials

\n

Finally, enter the service account ID and upload a JSON file containing your source account key associated with the Project ID you entered in step 1.3. mParticle uses this information to log into BigQuery on your behalf when syncing data.

\n

Your service account ID must match the source account key.

\n
\n
\n

\n \n \n \n \n \n \n \n Connect to Snowflake

\n

\n

\n \n \n \n \n \n \n \n 3.1 Enter a configuration name

\n

The configuration name is specific to mParticle and will appear in your list of warehouse inputs. You can use any configuration name, but it must be unique since this name is used when configuring the rest of your sync settings.

\n

\n \n \n \n \n \n \n \n 3.2 Select the environment type where your data originates

\n

Select either Prod or Dev depending on whether your want your warehouse data sent to the mParticle development or production environments. This setting determines how mParticle processes ingested data.

\n

\n \n \n \n \n \n \n \n 3.3 Enter your Snowflake account ID

\n

Your Snowflake account ID uniquely identifies your Snowflake account. You can find your Snowflake account ID by logging into your Snowflake account and finding your account_locator, cloud_region_id, and cloud. More details here and here.

\n

\n \n \n \n \n \n \n \n 3.4 Enter your Snowflake warehouse name

\n

The Snowflake warehouse name is used to find the specific database you are connecting to. Each Snowflake warehouse can contain multiple databases.

\n

\n \n \n \n \n \n \n \n 3.5 Enter your Snowflake database name

\n

The specific database you want to sync data from.

\n

\n \n \n \n \n \n \n \n 3.6 Enter the region

\n

Enter the region identifier corresponding to where your Snowflake data is localized following the format <cloud_region_id> or <cloud_region_id>.<cloud>, where <cloud> is an additional cloud provider ID, depending on the region your data is localized. To determine if you need to include the additional cloud provider ID:

\n
    \n
  1. First, refer to Snowflake’s list of region IDs to find the cloud region ID for your account. At the least, you will enter this value for your region ID.
    2. \n
  2. Next, refer to Snowflake’s documentation on account locator formats by cloud platform and region to see if you need to include an additional cloud identifier by looking for your region in the left-most column and looking to see if a cloud ID is appended in the second column (e.g. aws, gcp, azure, etc.)
    3. \n
\n

For example, if your data is localized in the US West (Oregon) region, you only need to enter us-west-2 as your cloud region ID. However, if your data is localized in US East (Ohio), you must enter us-east-2.aws, making sure to append .aws to your cloud region ID (us-east-2).

\n

Do not include your account identifier (Snowflake uses the example account identifier xy12345 in their documentation linked above) when entering your region ID. You should only enter your cloud region ID and cloud provider ID (if necessary).

\n

\n \n \n \n \n \n \n \n 3.7 Add your Snowflake credentials

\n

Finally, you must provide your Snowflake credentials.

\n

If you are creating a new pipeline, you should have created a new private/public key pair and assigned the public key to your Snowflake user role. Enter the Key ID for the private/public key pair you created and Role you assigned the public key to.

\n

\n \n \n \n \n \n \n \n 3.8 Add your storage access

\n

\n

Follow the instructions on the configuration page in order to setup your storage integration and optional external stage. Refer to Snowflake’s storage integration and stage documentation for more details.

\n
\n
\n

\n \n \n \n \n \n \n \n Connect to Databricks

\n

\n

\n \n \n \n \n \n \n \n 3.1 Enter a configuration name

\n

The configuration name is specific to mParticle and will appear in your list of warehouse inputs. You can use any configuration name, but it must be unique since this name is used when configuring the rest of your sync settings.

\n

\n \n \n \n \n \n \n \n 3.2 Enter your Databricks provider

\n

The value you enter for your Databricks provider must match the value of the Databricks organization that contains the schema you want to ingest data from. This is the same value that you saved when following Step 4 of the Data Warehouse setup, Find and save your Databricks provider name, prior to creating your warehouse connection.

\n

\n \n \n \n \n \n \n \n 3.3 Enter your Databricks schema

\n

Enter the name of the schema in your Databricks account that you want to ingest data from. Databricks uses the terms database and schema interchangeably, so in this situation the schema is the specific collection of tables and views that mParticle will access through this Warehouse Sync connection.

\n
\n
\n

\n \n \n \n \n \n \n \n 4 Create data model

\n

Your data model describes which columns from your warehouse to ingest into mParticle, and which mParticle fields each column should map to. While mParticle data models are written in SQL, all warehouse providers process SQL slightly differently so it is important to use the correct SQL syntax for the warehouse provider you select.

\n

For a detailed reference of all SQL commands Warehouse Sync supports alongside real-world example SQL queries, see Warehouse Sync SQL reference.

\n

\n \n \n \n \n \n \n \n Steps to create a data model

\n
    \n
  1. Write a SQL query following the guidelines outlined below and the Warehouse Sync SQL reference. Make sure to use SQL commands supported by your warehouse provider.
    2. \n
  2. Enter the SQL query in the SQL query text box, and click Run Query.
    3. \n
  3. Click Next.
    4. \n
\n

\n

mParticle submits the SQL query you provide to your warehouse to retrieve specific columns of data. Depending on the SQL operators and functions you use, the columns selected from your database are transformed, or mapped, to user profile attributes in your mParticle account.

\n

If you use an attribute name in your SQL query that doesn’t exist in your mParticle account, mParticle creates an attribute with the same name and maps this data to the new attribute.

\n

mParticle automatically maps matching column names in your warehouse to reserved mParticle user attributes and device ids. For example, if your database contains a column named customer_id, it is automatically mapped to the customer_id user identifier in mParticle. For a complete list of reserved attribute names, see Reserved mParticle user or device identity column names.

\n
\n \n \n \n \n \n \n \n Example database and SQL query
\n

Below is an example of a simple table and SQL query to create a data model:

\n

Table name: mp.demo.userdata

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Column names:first_namelast_nameemail
JohnDoejohn@example.com
\n

Example SQL query:

\n
SELECT\n    first_name,\n    last_name,\n    email\nFROM mp.demo.userdata
\n

This SQL query selects the first_name, last_name, and email columns from the table called mp.demo.userdata. In the next step, we will set up the mappings for this user data.

\n

\n \n \n \n \n \n \n \n 5 Create data mapping

\n

After creating a data model that specifies which columns in your warehouse you want to ingest, you must map each column to its respective field within mParticle with a data mapping.

\n\n

To create a data mapping, first use the dropdown menu titled Type of data to sync to select either User Attributes & Identities Only or Events, depending on whether you want to ingest user data or event data.

\n

To continue with our example user data table and SQL query from above, we’ll select User Attributes & Identities Only:

\n

\n \n \n \n \n \n \n \n User Attributes & Identities Only

\n

When mapping attributes and identities from your warehouse to fields in mParticle, you must always create a user_identities or user_attributes object first. You can then create the individual mappings for your identities and attributes within these entities, as shown in the next two sections.

\n\n
\n \n \n \n \n \n \n \n Map user identities
\n

To map your user identities:

\n
    \n
  1. Click Add Mapping.
    2. \n
  2. Under Mapping Type, select Object.
    3. \n
  3. Under Field in mParticle, select user_identites.
    4. \n
  4. Click Save.
    5. \n
  5. Within your new user_identities object, click the + button.
    6. \n
  6. Under Mapping Type, select Column.
    7. \n
  7. Under Column in warehouse enter the name of the column containing the identities you want to ingest. In this example, we’ll use a column called email.
    8. \n
  8. Under Field in mParticle use the dropdown menu to select the correct user identity in mParticle you want to map your identities to. In this example, we’ll select Email Address.
    9. \n
\n

\n
    \n
  1. Click Save.
    2. \n
\n
\n \n \n \n \n \n \n \n Map user attributes
\n

To map your user attributes:

\n
    \n
  1. Click Add Mapping.
    2. \n
\n

\n
    \n
  1. Under Mapping Type, select Object.
    2. \n
  2. Under Field in mParticle, select user_attributes.
    3. \n
\n

\n
    \n
  1. Click Save.
    2. \n
  2. Within your new user_attributes object, click the + button.
    3. \n
\n

\n
    \n
  1. Under Mapping Type, select Column.
    2. \n
  2. Under Column in warehouse, enter the name of the column you want to map to mParticle. You can enter a specific column name, or $unmapped to select all currently unmapped columns.
    3. \n
\n\n
    \n
  1. Under Field in mParticle enter the name of the field in mParticle that you want to map your column to.
    2. \n
\n\n

\n
    \n
  1. Click Save.
    2. \n
\n\n

\n \n \n \n \n \n \n \n Events

\n

If you are ingesting event data instead of user data (as shown above), select Events under Type of data to sync once you reach the Review Mapping step.

\n
    \n
  1. Click Add Mapping.
    2. \n
\n

\n
    \n
  1. Under Mapping Type, select Object.
    2. \n
  2. Under Field in mParticle, select events.
    3. \n
\n

\n
    \n
  1. Click Save.
    2. \n
  2. Within the new events object, click the + button to add a new mapping.
    3. \n
\n

\n
    \n
  1. \n

    Select the type of mapping you want to use from the Mapping Type dropdown:

    \n\n
    2. \n
\n

The next steps will vary depending on the data you are ingesting and the mapping type you select. Following are several examples of how to use each mapping type.

\n
\n \n \n \n \n \n \n \n Column mapping
\n
    \n
  1. Under Column in warehouse, select the name of the column in your database you are mapping fields from.
    2. \n
  2. Under Field in mParticle, select the field in mParticle you are mapping fields to.
    3. \n
  3. Click Save.
    4. \n
\n

\n
\n \n \n \n \n \n \n \n Static mapping
\n
    \n
  1. Under Input Value select either String, Number, or Boolean for the data type of the static value you are mapping.
    2. \n
  2. In the entry box, enter the static value you are mapping.
    3. \n
  3. Under Field in mParticle, select the field you want to map your static value to.
    4. \n
  4. Click Save.
    5. \n
\n

\n
\n \n \n \n \n \n \n \n Ignore mapping
\n
    \n
  1. Under Column in warehouse, select the name of the column that you want your pipeline to ignore when ingesting data.
    2. \n
  2. Click Save.
    3. \n
\n

\n

To add additional mappings, click Add Mapping. You must create a mapping for every column you selected in your data model.

\n

\n\n

If you are back-loading data that is older than 30 days (for example, a historical replay or migration) add two additional mappings inside your events object:

\n\n

When these two mappings are present, mParticle processes the batch as historical. This means the events in the batch are not forwarded in real time to any connected event, data warehouse, or audience outputs, timestamp validation is relaxed, and the data remains eligible for long look-back audiences.

\n\n

When you have finished creating your mappings, click Next.

\n\n

\n \n \n \n \n \n \n \n 6 Set sync settings

\n

The sync frequency settings determine when the initial sync with your database will occur, and how frequently any subsequent syncs will be executed.

\n

\n \n \n \n \n \n \n \n 6.1 Select environment

\n

Select either Prod or Dev depending on whether you want your warehouse data sent to the mParticle development or production environments. This setting determines how mParticle processes ingested data.

\n

\n \n \n \n \n \n \n \n 6.2 Select input protection level

\n

Input protection levels determine how data ingested from your warehouse can contribute to new or existing user profiles in mParticle:

\n\n

To learn more about these settings and how they can be used in different scenarios, see Input protections.

\n

\n \n \n \n \n \n \n \n 6.3 Select sync mode and schedule

\n

There are two sync modes: incremental and full.

\n\n

Choosing Between Full and Incremental Syncs

\n

There are two common strategies for ingesting historical data efficiently:

\n
    \n
  1. Single incremental pipeline with an early from date – Set the from value to the start of the period you want to ingest, and leave until blank. The initial run will backfill the entire range, after which the pipeline will automatically switch to incremental updates on its schedule.
    2. \n
  2. Separate historical full and incremental pipelines – Create a full On Demand pipeline and use a WHERE clause in your data model to limit the dataset (for example, WHERE event_timestamp BETWEEN '2024-01-01' AND '2024-06-30'). Trigger the pipeline whenever you need to ingest another slice, updating the clause between runs as needed. Once you have finished ingesting the historical data, disable the full pipeline and create a new incremental pipeline to sync ongoing new data on a regular schedule, setting the from value to the last date you ingested with the full pipeline.
    3. \n
\n

Important notes:

\n\n
\n \n \n \n \n \n \n \n Deduplication and replays
\n

Warehouse Sync deduplicates inbound data to prevent identical batches from being processed multiple times (for example, due to retries, replays, or repeatedly reading the same rows in a full sync).

\n

This behavior matters most for full sync pipelines, because they re-read the entire result set each run. If a batch is considered a duplicate, it may be discarded without appearing as a visible error in the mParticle UI.

\n
\n \n \n \n \n \n \n \n How deduplication works
\n\n
\n \n \n \n \n \n \n \n How source_request_id is set
\n\n

The “A → B → A” revert scenario

\n

If a row changes from A → B, and then later changes B → A, the “revert” can be dropped if the batch ends up producing the same source_request_id as a prior run. This is most common with full sync pipelines where the resulting batch payload is identical to something mParticle has already seen in the deduplication retention window.

\n

How to force a re-sync safely

\n\n
\n \n \n \n \n \n \n \n Consent timestamp precedence
\n

When you ingest consent_state through Warehouse Sync, mParticle applies an incoming consent record only if its timestamp_unixtime_ms is newer than the timestamp already stored on the user profile for that regulation and purpose. If the incoming timestamp is older or the same, the existing consent stays. A purpose with no stored consent is always applied.

\n

This is the same rule used for consent on HTTP and SDK event batches.

\n

Example of a consent backfill that would no-op

\n

The profile already has GDPR purpose marketing with consented: false and timestamp_unixtime_ms: 1710000000000. A Warehouse Sync backfill maps consent_state with consented: true and timestamp_unixtime_ms: 1700000000000. The pipeline can complete successfully, but the profile keeps consented: false. To update the profile, the incoming consent timestamp must be greater than 1710000000000.

\n

Consent backfills and source_request_id

\n

A unique source_request_id per batch is required for the backfill to be processed at all. If the batch is treated as a duplicate, mParticle never evaluates the consent timestamps. Map source_request_id so it changes for each backfill run you intend to process, or map timestamp_unixtime_ms so the computed id changes. See How source_request_id is set on this page, Deduplication and replays for historical backfills, Deduplication and source_request_id, and Deduplicated batches.

\n

The remaining setting options change depending on the mode you select from the Sync mode dropdown menu. Navigate between the two configuration options using the tabs below:

\n
\n
\n

Following are the sync settings for incremental syncs:

\n

The main difference between full and incremental sync configurations is the use of an iterator field for incremental syncs. Both full and incremental syncs support all three sync modes (Interval, Once, and On Demand).

\n
\n \n \n \n \n \n \n \n Iterator
\n

Select the column name from your warehouse using the Iterator dropdown menu. The options are generated from the SQL query you ran when creating your data model.

\n

mParticle uses the iterator to determine which data to include and exclude during each incremental sync.

\n

Best Practices for Choosing Iterator Fields:

\n\n
\n \n \n \n \n \n \n \n Iterator data type
\n

Select the iterator data type using the Iterator data type dropdown. This value must match the datatype of the iterator as it exists in your warehouse.

\n
\n \n \n \n \n \n \n \n Sync schedule type
\n

Select one of the following schedule types:

\n\n
\n \n \n \n \n \n \n \n Frequency of sync
\n\n

Sync frequencies can be either Hourly, Daily, Weekly, or Monthly.

\n

The date and time you select for your initial sync is used to calculate the date and time for the next sync. For example, if you select Hourly for your frequency and 11/15/2023 07:00 PM UTC for your initial sync, then the next sync will occur at 11/15/2023 08:00 PM UTC.

\n
\n \n \n \n \n \n \n \n Date & time of sync
\n

Use the date and time picker to select the calendar date and time (in UTC) for your initial sync. Subsequent interval syncs will be scheduled based on this initial date and time.

\n
\n
\n

Following are the sync settings for full syncs:

\n

The main difference between full and incremental sync configurations is the use of an iterator field for incremental syncs. Both full and incremental syncs support all three sync modes (Interval, Once, and On Demand).

\n\n
\n \n \n \n \n \n \n \n Sync schedule type
\n

Select one of the following schedule types:

\n\n
\n \n \n \n \n \n \n \n Frequency of sync
\n

Sync frequencies can be either Hourly, Daily, Weekly, or Monthly.

\n

The date and time you select for your initial sync is used to calculate the date and time for the next sync. For example, if you select Hourly for your frequency and 11/15/2023 07:00 PM UTC for your initial sync, then the next sync will occur at 11/15/2023 08:00 PM UTC.

\n
\n \n \n \n \n \n \n \n Date & time of initial sync
\n

Use the date and time picker to select the calendar date and time (in UTC) for your initial sync. Subsequent interval syncs will be scheduled based on this initial date and time.

\n
\n
\n

\n \n \n \n \n \n \n \n 6.4 Sync historical data

\n

The value you select for Sync Start Date determines how much old, historical data mParticle ingests from your warehouse in your initial sync. When determining how much historical data to ingest, mParticle uses to the column in your database you selected as the Timestamp field in the Create Data Model step.

\n

After your initial sync begins, mParticle begins ingesting any historical data. If mParticle hasn’t finished ingesting historical data before the time a subsequent sync is due to start, the subsequent sync is still executed, and the historical data continues syncing in the background.

\n

Historical data syncing doesn’t contribute to any rate limiting on subsequent syncs.

\n\n

\n

After entering your sync settings, click Next.

\n

\n \n \n \n \n \n \n \n 7 Review

\n

mParticle generates a preview for Data Warehouse syncs that have been configured, but not yet activated. Use this page and the sample enriched user profiles to confirm the following:

\n\n\n

After reviewing your sync configuration, click Activate to activate your sync.

\n

\n \n \n \n \n \n \n \n View and manage existing warehouse syncs

\n
    \n
  1. Log into your mParticle account.
    2. \n
  2. Navigate to Setup > Inputs in the left hand nav bar and select the Feeds tab.
    3. \n
  3. \n

    Any configured warehouse syncs are listed on this page, grouped by warehouse provider. Expand a warehouse provider to view and manage a sync.

    \n\n
    4. \n
\n

\n \n \n \n \n \n \n \n Manage a sync

\n

To view details for an existing sync, select it from the list of syncs. A summary page is displayed, showing the current status (Active or Paused), sync frequency, and a list of recent or in-progress syncs.

\n

To pause a sync, click Pause Sync. Paused syncs will only resume running on their configured schedule after you click Resume.

\n

To run an on-demand sync, click Run Sync under Sync Frequency.

\n

Use the Data Model, Mapping, and Settings tabs to view and edit your sync configuration details. Clicking Edit from any of these tabs opens the respective step of the setup wizard where you can make and save your changes.

","frontmatter":{"title":"Warehouse Sync User Guide","order":1,"seoTitle":null,"seoDescription":null},"parent":{"__typename":"File","modifiedTime":"October 12, 2026"}}},"pageContext":{"slug":"/guides/data-warehouse/warehouse-sync/setup/"}},"staticQueryHashes":["2950460867"]}