The target-kinesis Meltano loader sends data into Kinesis after it was pulled from a source using an extractor.

Alternative variants #

Multiple variants of target-kinesis are available. This document describes the default prontopro variant, which is recommended for new users.

Alternative variants are:

Getting Started #

Prerequisites #

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project
  3. Add an extractor to pull data from a source

Installation and configuration #

Using the Command Line Interface #

  1. Add the target-kinesis loader to your project using meltano add : 

    meltano add loader target-kinesis
  2. Configure the settings below using meltano config .

Next steps #

Follow the remaining steps of the Getting Started guide:

  1. Run a data integration (EL) pipeline

If you run into any issues, learn how to get help.

Capabilities #

Settings #

target-kinesis requires the configuration of the following settings:

These and other supported settings are documented below. To quickly find the setting you're looking for, use the Table of Contents at the top of the page.

Stream Name (stream_name) #

The name of the Kinesis stream to write to.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set stream_name <stream_name>

export TARGET_KINESIS_STREAM_NAME=<stream_name>

AWS S3 Access Key ID (aws_access_key_id) #

S3 Access Key Id. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set aws_access_key_id <aws_access_key_id>

export TARGET_KINESIS_AWS_ACCESS_KEY_ID=<aws_access_key_id>

AWS S3 Secret Access Key (aws_secret_access_key) #

S3 Secret Access Key. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set aws_secret_access_key <aws_secret_access_key>

export TARGET_KINESIS_AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>

AWS Region (region) #

The AWS region to use i.e. us-east-2.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set region <region>

export TARGET_KINESIS_REGION=<region>

Is Firehouse (is_firehose) #

(Default False) Whether to user a Firehouse client or not.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set is_firehose true

export TARGET_KINESIS_IS_FIREHOSE=true

Record Chunks (record_chunks) #

(Default - 10) The max number of records per chunk.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set record_chunks 1234

export TARGET_KINESIS_RECORD_CHUNKS=1234

Data Chunks (data_chunks) #

(Default - 1000) The max size of data per chunk.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set data_chunks 1234

export TARGET_KINESIS_DATA_CHUNKS=1234

Partition Key (partition_key) #

(Default - id) The partition key to use when writing to Kinesis.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set partition_key <partition_key>

export TARGET_KINESIS_PARTITION_KEY=<partition_key>

Looking for help? #

If you're having trouble getting the target-kinesis loader to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page? #

This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!