The tap-rest-api-msdk Meltano extractor pulls data from REST API that can then be sent to a destination using a loader.

Getting Started #

Prerequisites #

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project

Installation and configuration #

Using the Command Line Interface #

  1. Add the tap-rest-api-msdk extractor to your project using meltano add : 

    meltano add extractor tap-rest-api-msdk
  2. Configure the settings below using meltano config .

Next steps #

Follow the remaining steps of the Getting Started guide:

  1. Select entities and attributes to extract
  2. Add a loader to send data to a destination
  3. Run a data integration (EL) pipeline

If you run into any issues, learn how to get help.

Capabilities #

Settings #

tap-rest-api-msdk requires the configuration of the following settings:

These and other supported settings are documented below. To quickly find the setting you're looking for, use the Table of Contents at the top of the page.

Name (name) #

(required) name of the stream.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-rest-api-msdk set name <name>

export TAP_REST_API_MSDK_NAME=<name>

API Base URL (api_url) #

(required) the base url/endpoint for the desired api.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-rest-api-msdk set api_url <api_url>

export TAP_REST_API_MSDK_API_URL=<api_url>

Path (path) #

(optional) the path appeneded to the api_url.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-rest-api-msdk set path <path>

export TAP_REST_API_MSDK_PATH=<path>

Params (params) #

(optional) an object of objects that provide the params in a requests.get method.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-rest-api-msdk set params <params>

export TAP_REST_API_MSDK_PARAMS=<params>

Headers (headers) #

(optional) an object of headers to pass into the api calls.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-rest-api-msdk set headers <headers>

export TAP_REST_API_MSDK_HEADERS=<headers>

Records Path (records_path) #

(optional) a jsonpath string representing the path in the requests response that contains the records to process. Defaults to $[*].

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-rest-api-msdk set records_path <records_path>

export TAP_REST_API_MSDK_RECORDS_PATH=<records_path>

Primary Keys (primary_keys) #

(required) a list of the json keys of the primary key for the stream.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-rest-api-msdk set primary_keys <primary_keys>

export TAP_REST_API_MSDK_PRIMARY_KEYS=<primary_keys>

Replication Key (replication_key) #

(optional) the json key of the replication key. Note that this should be an incrementing integer or datetime object.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-rest-api-msdk set replication_key <replication_key>

export TAP_REST_API_MSDK_REPLICATION_KEY=<replication_key>

Except Keys (except_keys) #

This tap automatically flattens the entire json structure and builds keys based on the corresponding paths. Keys, whether composite or otherwise, listed in this dictionary will not be recursively flattened, but instead their values will be turned into a json string and processed in that format. This is also automatically done for any lists within the records; therefore, records are not duplicated for each item in lists.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-rest-api-msdk set except_keys <except_keys>

export TAP_REST_API_MSDK_EXCEPT_KEYS=<except_keys>

Number Inference Records (num_inference_records) #

(optional) number of records used to infer the stream’s schema. Defaults to 50.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-rest-api-msdk set num_inference_records <num_inference_records>

export TAP_REST_API_MSDK_NUM_INFERENCE_RECORDS=<num_inference_records>

Looking for help? #

If you're having trouble getting the tap-rest-api-msdk extractor to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page? #

This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!