Why are messages waiting so long in Amazon SQS?
An export is overdue even though the API responds. Use message age, processing status and the dead-letter queue to work out where the delay starts.
A user requests an export. The site accepts the request and will show a download link once the file is ready. The file never appears. If exports run through a standard Amazon SQS queue, check how long messages have been in the queue and what the processes that receive them are doing. A backlog of the same size might clear quickly during a burst or sit there for minutes.
Base the waiting limit on when the user needs the file
Start with the deadline for making the export available. If the target is 5 minutes from accepting the request and you reserve 1 minute to generate and deliver the file, that leaves at most 4 minutes of waiting. This is an example budget. Set the alert threshold early enough to allow time to evaluate the metric and deliver the notification.
In CloudWatch, track ApproximateAgeOfOldestMessage. It estimates the age of the oldest unprocessed message in seconds. This tells you when work is already being delayed in the queue. Measure the time from accepting the request to making the file available in the application to see whether you met the target. SQS has no visibility into steps before enqueueing or the result the user receives.
Check whether messages are waiting to be received or completed
Plot ApproximateNumberOfMessagesVisible and ApproximateNumberOfMessagesNotVisible on the same timeline. The first estimates how many messages are available to receive. The second counts messages that workers have received and have yet to delete. If the age and visible count rise while few messages are in flight, check whether the workers are running and can read from the intended queue. To estimate how long a message waited for its first receive, subtract SentTimestamp from ApproximateFirstReceiveTimestamp. SQS can return both as message system attributes.
If more messages are in flight and few exports are finishing, check processing time, errors and storage latency. Once the visibility timeout expires, another worker can receive an undeleted message. Compare repeated receives with the export ID in application logs. SQS receive and delete counters can include repeated operations. Measure success by completed exports.
Track messages that keep coming back
In a standard queue, SQS can move a repeatedly received, undeleted message to the back. The age metric can then omit that message. Track processing errors by job and, if you configured a dead-letter queue (DLQ), its ApproximateNumberOfMessagesVisible. The DLQ holds messages that left the main queue after reaching the configured receive limit.
The age reported for a message moved to the DLQ is measured from the move. To find out how long the user has been waiting, use the request time recorded by the application. Before sending messages back to the main queue, fix the cause of failure and make sure running the same job again cannot generate an extra export or duplicate notification.
Check the oldest unfinished jobs after the fix
After the fix, check that message age falls and exports are completing. Add workers when they can handle more work without overloading storage or the database. If any requests are still outstanding, look them up by ID and check their status in the application and the DLQ.
Run a test export through the usual path. Measure the time from requesting the export to opening the file, and account for requests delayed during the incident. A new export can succeed while an older job still needs attention.
- Set a waiting limit based on when the user needs the result.
- Plot message age, visible messages and in-flight messages together.
- Check repeated receives and the number of messages in the DLQ.
- Confirm that both new and delayed exports are available.
What to take away
Message age tells you when to look at the queue. Message counts and logs help you decide where to intervene. Confirm recovery by checking when users get their finished files.
