SQS

Input SQS

Overview

This input listens on multiple SQS queues for paths to S3 files. It can be used with SQS queues subscribed to SNS topics (with raw_message_delivery subscription) and supports arbitrary payload (plain or json). It never exits.

Supported formats (MessageFormat):

• “plain”: the message payload is a plain S3 path.
• “sns”: the payload is an SNS notification
• “s3::ObjectCreated”: the payload is the message generated by an S3::ObjectCreated notification
• “json”: the message is an arbitrary JSON string, the S3 path is extracted from it via a jmespath expression specified in MessageExpression.

Configuration

Keys available in the [input.config] section:

Name	Type	Default	Required	Description
AwsRegion	string	“us-west-2”	false	AWS region to connect to
Bucket	string	""	false	S3 Bucket to use if paths do not have one
QueuePrefixes	array of strings	[]	false	Prefixes of the names of the SQS queues to monitor
QueueNames	array of strings	[]	false	Names of the SQS queues to monitor
MessageFormat	string	“sns”	false	SQS message format. See help string for supported formats
MessageExpression	string	""	false	The expression to extract an S3 path from arbitrary message formats
FilePathFilter	string	""	false	If provided, will only use S3 files with the given path.