The target-bigquery Singer target sends data into BigQuery after it was pulled from a source using a Singer tap.

Alternative variants #

Multiple variants of target-bigquery are available. This document describes the default adswerve variant, which is recommended for new users.

Alternative variants are:

Standalone usage #

Install the package using pip: 

pip install git+https://github.com/adswerve/target-bigquery.git@0.11.3

For additional instructions, refer to the README in the repository.

Prerequisites #

Then, follow the steps in the “Activate the Google BigQuery API” section of the repository’s README.

Usage with Meltano #

Meltano helps you manage your configuration, incremental replication, and scheduled pipelines.

View the Meltano-specific target-bigquery instructions to learn more.

Capabilities #

Settings #

target-bigquery requires the configuration of the following settings:

These and other supported settings are documented below. To quickly find the setting you're looking for, use the Table of Contents at the top of the page.

Project ID (project_id) #

BigQuery project

Dataset ID (dataset_id) #

  • Default: $MELTANO_EXTRACT__LOAD_SCHEMA

BigQuery dataset.

The default value will expand to the value of the load_schema extra for the extractor used in the pipeline, which defaults to the extractor’s namespace, e.g. tap_gitlab for tap-gitlab.

Location (location) #

  • Default: US

Dataset Location. See https://cloud.google.com/bigquery/docs/locations.

Credentials Path (credentials_path) #

  • Default: $MELTANO_PROJECT_ROOT/client_secrets.json

Fully qualified path to client_secrets.json for your service account.

See the “Activate the Google BigQuery API” section of the repository’s README and https://cloud.google.com/docs/authentication/production.

By default, this file is expected to be at the root of your project directory.

Validate Records (validate_records) #

  • Default: false

Add Metadata Columns (add_metadata_columns) #

  • Default: false

Add _time_extracted and _time_loaded metadata columns

Replication Method (replication_method) #

  • Default: append

Replication method, append or truncate

Table Prefix (table_prefix) #

Add prefix to table name

Table Suffix (table_suffix) #

Add suffix to table name

Max Cache (max_cache) #

  • Default: 50

Maximum cache size in MB

Merge State Messages (merge_state_messages) #

  • Default: false

Whether to merge multiple state messages from the tap into the state file or uses the last state message as the state file. Note that it is not recommended to set this to true when using with Meltano as the merge behavior conflicts with Meltano’s merge process.

Table Config (table_config) #

A path to a file containing the definition of partitioning and clustering.

Looking for help? #

If you're having trouble getting the target-bigquery target to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page? #

This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!