AWS Athena Error: Query exhausted resources at this scale factor
Author: Omid Vahdaty 12.2.2019
Athena is a Serverless technology. i.e. It makes use of shared resources available with AWS and hence, when large amount of queries are submitted by users concurrently around the world at the same time, sometimes resource exhaustion take place.
Athena service team has identified this as a known issue.
However, this error is transient in nature, if you can submit the query again, it might be successful.
If you repeatedly get the same error consistently, then you might need to partition your data and optimize the query further as mentioned in Performance Tuning Best Practices for Athena. Another option is to follow this blog- Tips to reduce costs on AWS SQL Athena , which might reduce resource consumption.
AWS support team suggestions:
- Avoid submitting queries at the beginning or end of an hour. If query fails, Back off exponentially by some minutes and try to submit query again. [ Wierd, but thats an official answer…]
- highly recommended to adopt Amazon Athena best practices to optimize your query and your data.
- Use columnar formatted data which can drastically reduce the resource consumption.
——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way.
If you have any comments, thoughts, questions, or you need someone to consult with,
feel free to contact me via LinkedIn: