Snowflake (meltano variant)

The target-snowflake Meltano loader sends data into Snowflake after it was pulled from a source using an extractor.

• 
• Repository: https://gitlab.com/meltano/target-snowflake
• Maintainer: Meltano Community

Alternative variants #

Multiple variants of target-snowflake are available. This document describes the meltano variant.

Alternative variants are:

• datamill-co
• transferwise (default)

Getting Started #

Prerequisites #

If you haven't already, follow the initial steps of the Getting Started guide:

1. Install Meltano
2. Create your Meltano project
3. Add an extractor to pull data from a source

Installation and configuration #

Using the Command Line Interface #

1. Add the target-snowflake loader to your project using meltano add :

   meltano add loader target-snowflake --variant meltano

2. Configure the settings below using meltano config .

Next steps #

Follow the remaining steps of the Getting Started guide:

1. Run a data integration (EL) pipeline

If you run into any issues, learn how to get help.

Capabilities #

Settings #

target-snowflake requires the configuration of the following settings:

• account
• username
• password
• role
• database
• warehouse
• schema

These and other supported settings are documented below. To quickly find the setting you're looking for, use the Table of Contents at the top of the page.

Account (account) #

• Environment variable: TARGET_SNOWFLAKE_ACCOUNT

Account Name in Snowflake (https://XXXXX.snowflakecomputing.com)

How to use #

Manage this setting using meltano config or an environment variable:

meltano config target-snowflake set account <account>

export TARGET_SNOWFLAKE_ACCOUNT=<account>

Username (username) #

• Environment variable: TARGET_SNOWFLAKE_USERNAME

The username you use for logging in

How to use #

Manage this setting using meltano config or an environment variable:

meltano config target-snowflake set username <username>

export TARGET_SNOWFLAKE_USERNAME=<username>

Password (password) #

• Environment variable: TARGET_SNOWFLAKE_PASSWORD

The password you use for logging in

How to use #

Manage this setting using meltano config or an environment variable:

meltano config target-snowflake set password <password>

export TARGET_SNOWFLAKE_PASSWORD=<password>

Role (role) #

• Environment variable: TARGET_SNOWFLAKE_ROLE

Role to be used for loading the data, e.g. LOADER. Also this role is GRANTed usage to all tables and schemas created

How to use #

Manage this setting using meltano config or an environment variable:

meltano config target-snowflake set role <role>

export TARGET_SNOWFLAKE_ROLE=<role>

Database (database) #

• Environment variable: TARGET_SNOWFLAKE_DATABASE

The name of the Snowflake database you want to use

How to use #

Manage this setting using meltano config or an environment variable:

meltano config target-snowflake set database <database>

export TARGET_SNOWFLAKE_DATABASE=<database>

Warehouse (warehouse) #

• Environment variable: TARGET_SNOWFLAKE_WAREHOUSE

The name of the Snowflake warehouse you want to use

How to use #

Manage this setting using meltano config or an environment variable:

meltano config target-snowflake set warehouse <warehouse>

export TARGET_SNOWFLAKE_WAREHOUSE=<warehouse>

Schema (schema) #

• Environment variable: TARGET_SNOWFLAKE_SCHEMA
• Default: $MELTANO_EXTRACT__LOAD_SCHEMA

Note $MELTANO_EXTRACT__LOAD_SCHEMA will expand to the value of the load_schema extra for the extractor used in the pipeline, which defaults to the extractor’s namespace, e.g. tap_gitlab for tap-gitlab. Values are automatically converted to uppercase before they’re passed on to the plugin, so tap_gitlab becomes TAP_GITLAB.

How to use #

Manage this setting using meltano config or an environment variable:

meltano config target-snowflake set schema <schema>

export TARGET_SNOWFLAKE_SCHEMA=<schema>

Batch Size (batch_size) #

• Environment variable: TARGET_SNOWFLAKE_BATCH_SIZE
• Default: 5000

The number of records sent to Snowflake at a time.

How to use #

Manage this setting using meltano config or an environment variable:

meltano config target-snowflake set batch_size 5000

export TARGET_SNOWFLAKE_BATCH_SIZE=5000

Timestamp Column (timestamp_column) #

• Environment variable: TARGET_SNOWFLAKE_TIMESTAMP_COLUMN
• Default: __loaded_at

Name of the column used for recording the timestamp when Data are uploaded to Snowflake.

How to use #

Manage this setting using meltano config or an environment variable:

meltano config target-snowflake set timestamp_column <timestamp_column>

export TARGET_SNOWFLAKE_TIMESTAMP_COLUMN=<timestamp_column>

Looking for help? #

If you're having trouble getting the target-snowflake loader to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page? #

This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!