Log Files

Log Files (files)

Monitor one or more log files for new lines.

File binary json raw

Minimal example

input:
  files:
    path: ""

JSON

{
  "input": {
    "files": {
      "path": ""
    }
  }
}

Contents

• Minimal example

• Fields

• Location

• Behavior

• Filtering

• Performance

• Reliability

• Object Properties

• Schema

Fields

Field	Type	Required	Description
path Location	path (string)	✅	a path with wildcards identifying files to be read. Examples: /path/to/file, c:\users\joe\data\file.txt
encoding Behavior	encoding (string)		the text encoding for the monitored files. Examples: UTF-8
include Location	path[] (string)		any other paths to be included. Examples: /path/to/file, c:\users\joe\data\file.txt
exclude Location	path[] (string)		paths to be excluded. Examples: /path/to/file, c:\users\joe\data\file.txt
ignore-older-than Filtering	number (integer)		ignore files older than this (epoch time in seconds). Examples: 42, 1.2e-10
glob-minimum-cooldown Performance	number (integer)		pause after scanning for new files (in milliseconds). Examples: 42, 1.2e-10
stop-reading-after Behavior	boolean (bool)		do not wait for new files.
ignore-line-breaks Behavior	boolean (bool)		consume each file as one event.
payload-mode Behavior	Payload Mode		Select how file content should be interpreted (auto, json, raw, binary). Allowed values: auto, json, raw, binary
remove-after Behavior	number (integer)		remove files that are done reading after this period (seconds). Examples: 42, 1.2e-10
fingerprinting Reliability	Fingerprinting		File fingerprinting strategy.
oldest-first Behavior	boolean (bool)		default is youngest first.
file-path-field Object Properties	field (string)		fill this field with the file we are currently reading from. Examples: data_field
file-basename Behavior	boolean (bool)		the file-path-field path will be the basename (e.g. ‘/path/frodo.txt’ becomes just ‘frodo’).
start-at-beginning Behavior	boolean (bool)		ignore stored checkpoints and re-read all specified files.
json Behavior	boolean (bool)		assume lines are already JSON.
max-read-size Performance	number (integer)		will not attempt to read files larger than this. Examples: 42, 1.2e-10
max-line-size Performance	number (integer)		will not attempt to read lines longer than this. Examples: 42, 1.2e-10

Location

Show fields

Field	Type	Required	Description
path	path (string)	✅	a path with wildcards identifying files to be read. Examples: /path/to/file, c:\users\joe\data\file.txt
include	path[] (string)		any other paths to be included. Examples: /path/to/file, c:\users\joe\data\file.txt
exclude	path[] (string)		paths to be excluded. Examples: /path/to/file, c:\users\joe\data\file.txt

Behavior

Show fields

Field	Type	Required	Description
encoding	encoding (string)		the text encoding for the monitored files. Examples: UTF-8
stop-reading-after	boolean (bool)		do not wait for new files.
ignore-line-breaks	boolean (bool)		consume each file as one event.
payload-mode	Payload Mode		Select how file content should be interpreted (auto, json, raw, binary). Allowed values: auto, json, raw, binary
remove-after	number (integer)		remove files that are done reading after this period (seconds). Examples: 42, 1.2e-10
oldest-first	boolean (bool)		default is youngest first.
file-basename	boolean (bool)		the file-path-field path will be the basename (e.g. ‘/path/frodo.txt’ becomes just ‘frodo’).
start-at-beginning	boolean (bool)		ignore stored checkpoints and re-read all specified files.
json	boolean (bool)		assume lines are already JSON.

Filtering

Show fields

Field	Type	Required	Description
ignore-older-than	number (integer)		ignore files older than this (epoch time in seconds). Examples: 42, 1.2e-10

Performance

Show fields

Field	Type	Required	Description
glob-minimum-cooldown	number (integer)		pause after scanning for new files (in milliseconds). Examples: 42, 1.2e-10
max-read-size	number (integer)		will not attempt to read files larger than this. Examples: 42, 1.2e-10
max-line-size	number (integer)		will not attempt to read lines longer than this. Examples: 42, 1.2e-10

Reliability

Show fields

Field	Type	Required	Description
fingerprinting	Fingerprinting		File fingerprinting strategy.

Object Properties

Show fields

Field	Type	Required	Description
file-path-field	field (string)		fill this field with the file we are currently reading from. Examples: data_field

Schema

• Fingerprinting Options
• Fingerprinting - Checksum Fields
• Fingerprinting - First Line Checksum Fields
• Payload Mode Options
• Fingerprinting - Dev Inode Options

Fingerprinting Options

Option	Name	Type	Description
checksum	Checksum	object	Fingerprint strategy is set to a checksum of the start of the file determined by the bytes that are skipped (ignored-header-bytes) and then the fingerprint size in bytes.
first-line-checksum	First Line Checksum	object	Fingerprint strategy that uses the first line of the file to detect if a file has changed.
dev-inode	Dev Inode	symbol	Fingerprint strategy is set to the inode of the file causing the file monitor to detect new files when a file is recreated. Allowed values: fingerprinter-dev-inode

Fingerprinting - Checksum Fields

Field	Type	Required	Description
fingerprint-bytes	number (integer)		The number of bytes to use for the fingerprint, starts after ignore-header-bytes if specified. Examples: 42, 1.2e-10
ignored-header-bytes	number (integer)		The number of bytes from the start of the file to ignore before starting the fingerprint. Examples: 42, 1.2e-10

Fingerprinting - First Line Checksum Fields

Field	Type	Required	Description
max-line-length	number (integer)		The maximum number of bytes to read from the first line of the file to calculate the fingerprint. Examples: 42, 1.2e-10

Payload Mode Options

Value	Description
auto	Auto
json	Json
raw	Raw
binary	Binary

Fingerprinting - Dev Inode Options

Value	Description
fingerprinter-dev-inode	Fingerprint strategy is set to the inode of the file causing the file monitor to detect new files when a file is recreated