FileWriter
Read the API documentation »
Output FileWriter
Overview
This is a raw output, for each record it receives a buffer containing the serialized record, plus a list holding a set of fields (output.fields
in TOML).
This output writes the records into compressed files in a directory. Files will be compressed using Gzip or Zstandard based on the filename extension in PathString. The file names can contain placeholders that are populated by the output (see the keys help below). When the special {{.Field0}} placeholder is used, then the user must specify the field name to use for replacement in the fields configuration list. The value of that field, extracted from each record, is used as replacement and, moreover, this also means that each created file will contain only records with that same value for the field. Note that, with this option, the FileWriter creates as many workers as the different values of the field, and each one of these workers concurrently writes to a different file.
Configuration
Keys available in the [output.config]
section:
Name | Type | Default | Required | Description |
---|---|---|---|---|
PathString | string | "" | false | Template to describe location of the output directory: supports .Year, .Month, .Day and .Rotation. Also .Field0 if a field name has been specified in the output’s fields list. |
RotateInterval | duration | 60s | false | Time after which data will be rotated. If -1, it will not rotate until the end. |
ZstdCompressionLevel | int | 3 | false | zstd compression level, ranging from 1 (best speed) to 19 (best compression). |
ZstdWindowLog | int | 0 | false | Enable zstd long distance matching. Increase memory usage for both compressor/decompressor. If more than 27 the decompressor requires special treatment. 0:disabled. |