BigQuery

BigQuery (big-query)

Stream data into Google BigQuery.

Data Warehouse json

Minimal example

output:
  big-query:
    connection:
      project: ~

JSON

{
  "output": {
    "big-query": {
      "connection": {
        "project": null
      }
    }
  }
}

Contents

• Minimal example

• Fields

• Reliability

• Processing

• Connection

• Operation

• Streaming

• Load Job

• Schema

• Authentication

• Parameters

• Schema

Fields

Field	Type	Required	Description
connection Connection	Connection	✅	Connection details (project, dataset, table).
retry Reliability	Retry		Retry configuration for transient failures.
batch Processing	Batch		Batching configuration (size/timeout).
mode Operation	Mode		Output mode (streaming or load-job). Allowed values: streaming, load-job
streaming Streaming	Streaming		Configuration for Storage Write API.
load-job Load Job	Load Job		Configuration for Load Jobs (GCS staging).
schema Schema	Schema		Schema handling and coercion settings.
credentials Authentication	Credentials		Authentication credentials.
parameters Parameters	Parameters[]		Parameters for template expansion.

Reliability

Show fields

Field	Type	Required	Description
retry	Retry		Retry configuration for transient failures.

Processing

Show fields

Field	Type	Required	Description
batch	Batch		Batching configuration (size/timeout).

Connection

Show fields

Field	Type	Required	Description
connection	Connection	✅	Connection details (project, dataset, table).

Operation

Show fields

Field	Type	Required	Description
mode	Mode		Output mode (streaming or load-job). Allowed values: streaming, load-job

Streaming

Show fields

Field	Type	Required	Description
streaming	Streaming		Configuration for Storage Write API.

Load Job

Show fields

Field	Type	Required	Description
load-job	Load Job		Configuration for Load Jobs (GCS staging).

Schema

Show fields

Field	Type	Required	Description
schema	Schema		Schema handling and coercion settings.

Authentication

Show fields

Field	Type	Required	Description
credentials	Credentials		Authentication credentials.

Parameters

Show fields

Field	Type	Required	Description
parameters	Parameters[]		Parameters for template expansion.

Schema

• Credentials Options
• Retry Fields
• Batch Fields
• Connection - Impersonation Fields
• Connection Fields
• Streaming Fields
• Load Job Fields
• Schema - Overrides Fields
• Schema Fields
• Credentials - Service Account Fields
• Credentials - Service Account File Fields
• Credentials - Workload Identity Fields
• Parameters Fields
• Batch - Mode Options
• Mode Options
• Streaming - Stream Type Options
• Load Job - Write Disposition Options
• Load Job - Create Disposition Options

Credentials Options

Option	Name	Type	Description
service-account	Service Account	object
service-account-file	Service Account File	object
workload-identity	Workload Identity	object
application-default	Application Default	map

Retry Fields

Field	Type	Required	Description
timeout	time-interval (string)	✅	timeout (e.g. 500ms, 2s etc. - default is 30). Examples: 500ms, 2h
retries	number (integer)		number of retries. Examples: 42, 1.2e-10

Batch Fields

Field	Type	Required	Description
fixed-size	number (integer)		maximum number of events in an output batch. Examples: 42, 1.2e-10
mode	Mode	✅	If ‘document’ send on end of document generated by input. If ‘fixed’, use fixed_size. Allowed values: fixed, document
timeout	time-interval (string)	✅	interval after which the batch is sent, to keep throughput going (default 100ms). Examples: 500ms, 2h
header	multiline-text (string)		put a header line before the batch.
footer	multiline-text (string)		put a header line after the last line of the batch.
use-document-marker	boolean (bool)		Enrich the job metadata with a document marker (for document handling in batch mode).
wrap-as-json	boolean (bool)		Format the output batch as a JSON array.

Connection - Impersonation Fields

Field	Type	Required	Description
service-account	string	✅
delegates	string[]

Connection Fields

Field	Type	Required	Description
project	string	✅
dataset	string
table	string
billing-project	string
location	string
impersonation	Impersonation		Impersonate a service account (requires IAM Credentials API).

Streaming Fields

Field	Type	Required	Description
stream-type	Stream Type		Allowed values: default, committed, pending, buffered
insert-id-field	string		Field to use for deduplication (insertId).

Load Job Fields

Field	Type	Required	Description
staging-uri	string	✅	GCS URI for staging files (gs://bucket/path).
write-disposition	Write Disposition		Allowed values: write-append, write-truncate, write-empty
create-disposition	Create Disposition		Allowed values: create-if-needed, create-never

Schema - Overrides Fields

Field	Type	Required	Description
field-path	string	✅
target-type	string

Schema Fields

Field	Type	Required	Description
auto-fetch	boolean (bool)
overrides	Overrides[]

Credentials - Service Account Fields

Field	Type	Required	Description
secret-id	string	✅

Credentials - Service Account File Fields

Field	Type	Required	Description
path	string	✅

Credentials - Workload Identity Fields

Field	Type	Required	Description
audience	string	✅
service-account-email	string

Parameters Fields

Field	Type	Required	Description
name	string	✅
value	string
r-type	string

Batch - Mode Options

Value	Description
fixed	Fixed
document	Document

Mode Options

Value	Description
streaming	Streaming
load-job	Load Job

Streaming - Stream Type Options

Value	Description
default	Default
committed	Committed
pending	Pending
buffered	Buffered

Load Job - Write Disposition Options

Value	Description
write-append	Write Append
write-truncate	Write Truncate
write-empty	Write Empty

Load Job - Create Disposition Options

Value	Description
create-if-needed	Create If Needed
create-never	Create Never