SQS

Input SQS

Overview

This input listens on multiple SQS queues for paths to S3 files. It can be used with SQS queues subscribed to SNS topics (with raw_message_delivery subscription) and supports arbitrary payload (plain or json). It never exits.

Supported formats (MessageFormat):

  • “plain”: the message payload is a plain S3 path.
  • “sns”: the payload is an SNS notification
  • “s3::ObjectCreated”: the payload is the message generated by an S3::ObjectCreated notification
  • “json”: the message is an arbitrary JSON string, the S3 path is extracted from it via a jmespath expression specified in MessageExpression.

Configuration

Keys available in the [input.config] section:

Name Type Default Required Description
AwsRegion string “us-west-2” false AWS region to connect to
Bucket string "" false S3 Bucket to use if paths do not have one
QueuePrefixes array of strings [] true Prefixes of the names of the SQS queues to monitor
MessageFormat string “sns” false SQS message format. See help string for supported formats
MessageExpression string "" false The expression to extract an S3 path from arbitrary message formats
FilePathFilter string "" false If provided, will only use S3 files with the given path.
Last modified November 17, 2021