BigQuery

BigQuery (big-query)

Stream data into Google BigQuery.

Data Warehouse Enterprise edition json

Minimal example

output:
  big-query:
    connection:
      project: ~

JSON

{
  "output": {
    "big-query": {
      "connection": {
        "project": null
      }
    }
  }
}

Contents

• Minimal example

• Authentication

• Connection

• Load Job

• Operation

• Parameters

• Processing

• Reliability

• Schema

• Streaming

• Schema

Authentication

Authentication

Field	Type	Required	Description
credentials	Credentials		Authentication credentials. Allowed values: service-account, service-account-file, workload-identity, application-default

Connection

Connection

Field	Type	Required	Description
connection	Connection	✅	Connection details (project, dataset, table).

Load Job

Load Job

Field	Type	Required	Description
load-job	Load Job		Configuration for Load Jobs (GCS staging).

Operation

Operation

Field	Type	Required	Description
mode	Mode		Output mode (streaming or load-job). Default: streaming Allowed values: streaming, load-job

Parameters

Parameters

Field	Type	Required	Description
parameters	Parameters[]		Parameters for template expansion.

Processing

Processing

Field	Type	Required	Description
batch	Batch		Batching configuration (size/timeout).

Reliability

Reliability

Field	Type	Required	Description
retry	Retry		Retry configuration for transient failures.

Schema

Schema

Field	Type	Required	Description
schema	Schema		Schema handling and coercion settings.

Streaming

Streaming

Field	Type	Required	Description
streaming	Streaming		Configuration for Storage Write API.

Schema

• Credentials Options
• Credentials - Service Account Fields
• Credentials - Service Account File Fields
• Credentials - Workload Identity Fields
• Connection - Impersonation Fields
• Connection Fields
• Load Job Fields
• Parameters Fields
• Batch Fields
• Retry Fields
• Schema - Overrides Fields
• Schema Fields
• Streaming Fields
• Load Job - Write Disposition Options
• Load Job - Create Disposition Options
• Mode Options
• Batch - Mode Options
• Streaming - Stream Type Options

Credentials Options

Option	Name	Type	Description
service-account	Service Account	object
service-account-file	Service Account File	object
workload-identity	Workload Identity	object
application-default	Application Default	map

Credentials - Service Account Fields

Field	Type	Required	Description
secret-id ✓	string	✅

Credentials - Service Account File Fields

Field	Type	Required	Description
path	string	✅

Credentials - Workload Identity Fields

Field	Type	Required	Description
audience ✓	string	✅
service-account-email ✓	string

Connection - Impersonation Fields

Field	Type	Required	Description
service-account ✓	string	✅
delegates ✓	string[]

Connection Fields

Field	Type	Required	Description
project ✓	string	✅
dataset ✓	string
table ✓	string
billing-project ✓	string
location ✓	string
impersonation	Impersonation		Impersonate a service account (requires IAM Credentials API).

Load Job Fields

Field	Type	Required	Description
staging-uri ✓	string	✅	GCS URI for staging files (gs://bucket/path).
write-disposition	Write Disposition		Default: append Allowed values: write-append, write-truncate, write-empty
create-disposition	Create Disposition		Default: create-if-needed Allowed values: create-if-needed, create-never

Parameters Fields

Field	Type	Required	Description
name	string	✅
value ✓	string
r-type	string

Batch Fields

Field	Type	Required	Description
fixed-size ✓	number (integer)		maximum number of events in an output batch. Examples: 42, 1.2e-10
mode	Mode	✅	If ‘document’ send on end of document generated by input. If ‘fixed’, use fixed_size. Allowed values: fixed, document
timeout	time-interval (string)	✅	interval after which the batch is sent, to keep throughput going (default 100ms). Default: 100ms Examples: 500ms, 2h
header	multiline-text (string)		put a header line before the batch.
footer	multiline-text (string)		put a header line after the last line of the batch.
use-document-marker ✓	boolean (bool)		Enrich the job metadata with a document marker (for document handling in batch mode). Default: false
wrap-as-json ✓	boolean (bool)		Format the output batch as a JSON array. Default: false

Retry Fields

Field	Type	Required	Description
timeout	time-interval (string)	✅	timeout (e.g. 500ms, 2s etc. - default is 30). Examples: 500ms, 2h
retries	number (integer)		number of retries. Examples: 42, 1.2e-10

Schema - Overrides Fields

Field	Type	Required	Description
field-path	string	✅
target-type	string

Schema Fields

Field	Type	Required	Description
auto-fetch ✓	boolean (bool)		Default: true
overrides	Overrides[]

Streaming Fields

Field	Type	Required	Description
stream-type	Stream Type		Default: default Allowed values: default, committed, pending, buffered
insert-id-field	string		Field to use for deduplication (insertId).

Load Job - Write Disposition Options

Value	Name	Description
write-append	write-append	Write Append
write-truncate	write-truncate	Write Truncate
write-empty	write-empty	Write Empty

Load Job - Create Disposition Options

Value	Name	Description
create-if-needed	create-if-needed	Create If Needed
create-never	create-never	Create Never

Mode Options

Value	Name	Description
streaming	streaming	Streaming
load-job	load-job	Load Job

Batch - Mode Options

Value	Name	Description
fixed	fixed	Fixed
document	document	Document

Streaming - Stream Type Options

Value	Name	Description
default	default	Default
committed	committed	Committed
pending	pending	Pending
buffered	buffered	Buffered