Azure Blob

Azure Blob (azure-blob)

Send data to a Microsoft Azure Storage blob (Block Storage).

Block Store Cloud binary json raw

Minimal example

output:
  azure-blob:
    blob-destination:
      name: ~
    container-name: ~
    storage-account: ~
    storage-master-key: ~

JSON

{
  "output": {
    "azure-blob": {
      "blob-destination": {
        "name": null
      },
      "container-name": null,
      "storage-account": null,
      "storage-master-key": null
    }
  }
}

Contents

• Minimal example

• Fields

• Location

• Reliability

• Processing

• Behavior

• Object Properties

• Authentication

• Schema

Fields

Field	Type	Required	Description
container-name Location	string	✅	The storage service container for created blobs.
blob-destination Location	Blob Destination	✅	Either a literal blob name or the name of the field to read it from.
storage-account Authentication	string	✅	The Storage Account Name to be used (credential).
storage-master-key Authentication	string	✅	The Storage Master Key to be used (credential).
retry Reliability	Retry		How to retry operation if it fails.
batch Processing	Batch		Batching output events.
input-field Processing	field (string)		Send only the content of the specified field to the blob. Examples: data_field
mode Behavior	Mode		Either writing or deleting a blob. Allowed values: put, delete
disable-blob-name-guid Object Properties	boolean (bool)		Do not append a GUID to the output blob name.
guid-prefix Object Properties	string		GUID Prefix, will be prepended to the GUID, the default value is ”/”.
guid-suffix Object Properties	string		GUID Suffix, will be appended to the GUID if specified.
content-type Object Properties	string		Set the created blob to the following content-type.
preprocessors Processing	Preprocessors[]		Preprocessors (process data before making it available for upload) these processors will be run in the order they are specified. Allowed values: gzip, parquet, base64
track-schema Reliability	boolean (bool)		Check the schema of the written data and update __SCHEMA_NUMBER (written data must be JSON).

Location

Show fields

Field	Type	Required	Description
container-name	string	✅	The storage service container for created blobs.
blob-destination	Blob Destination	✅	Either a literal blob name or the name of the field to read it from.

Reliability

Show fields

Field	Type	Required	Description
retry	Retry		How to retry operation if it fails.
track-schema	boolean (bool)		Check the schema of the written data and update __SCHEMA_NUMBER (written data must be JSON).

Processing

Show fields

Field	Type	Required	Description
batch	Batch		Batching output events.
input-field	field (string)		Send only the content of the specified field to the blob. Examples: data_field
preprocessors	Preprocessors[]		Preprocessors (process data before making it available for upload) these processors will be run in the order they are specified. Allowed values: gzip, parquet, base64

Behavior

Show fields

Field	Type	Required	Description
mode	Mode		Either writing or deleting a blob. Allowed values: put, delete

Object Properties

Show fields

Field	Type	Required	Description
disable-blob-name-guid	boolean (bool)		Do not append a GUID to the output blob name.
guid-prefix	string		GUID Prefix, will be prepended to the GUID, the default value is ”/”.
guid-suffix	string		GUID Suffix, will be appended to the GUID if specified.
content-type	string		Set the created blob to the following content-type.

Authentication

Show fields

Field	Type	Required	Description
storage-account	string	✅	The Storage Account Name to be used (credential).
storage-master-key	string	✅	The Storage Master Key to be used (credential).

Schema

• Blob Destination Options
• Retry Fields
• Batch Fields
• Batch - Mode Options
• Mode Options
• Preprocessors Options

Blob Destination Options

Option	Name	Type	Description
name	Name	string	The name for the output blob, this will be suffixed by a GUID by default.
field	Field	string	A field containing the value for output_blob.

Retry Fields

Field	Type	Required	Description
timeout	time-interval (string)	✅	timeout (e.g. 500ms, 2s etc. - default is 30). Examples: 500ms, 2h
retries	number (integer)		number of retries. Examples: 42, 1.2e-10

Batch Fields

Field	Type	Required	Description
fixed-size	number (integer)		maximum number of events in an output batch. Examples: 42, 1.2e-10
mode	Mode	✅	If ‘document’ send on end of document generated by input. If ‘fixed’, use fixed_size. Allowed values: fixed, document
timeout	time-interval (string)	✅	interval after which the batch is sent, to keep throughput going (default 100ms). Examples: 500ms, 2h
header	multiline-text (string)		put a header line before the batch.
footer	multiline-text (string)		put a header line after the last line of the batch.
use-document-marker	boolean (bool)		Enrich the job metadata with a document marker (for document handling in batch mode).
wrap-as-json	boolean (bool)		Format the output batch as a JSON array.

Batch - Mode Options

Value	Description
fixed	Fixed
document	Document

Mode Options

Value	Description
put	Put Objects
delete	Delete Objects

Preprocessors Options

Value	Description
gzip	Gzip the output data
parquet	Extract the received data as JSON rows from a parquet file
base64	Decode base64 as binary