AWS athena

AWS Athena Error: Query exhausted resources at this scale factor

Athena is a Serverless technology i.e. It make use of shared resources available with AWS and hence, when large amount of queries are submitted by users concurrently around the world at the same time, sometimes resource exhaustion take place.  Athena service team has identified this as a known issue.

However, this error is transient in nature. By that I mean if you can submit the query again, it might be successful. If you repeatedly get the same error consistently, then you might need to partition your data and optimize the query further as mentioned in Performance Tuning Best Practices for Athena [1] and another aritcle in this blog about cost reduction which in turn might reduce resource consumption [2].

You can find suggestions below  from AWS support team:

1) Avoid submitting queries at the beginning or end of an hour. If query fails, Back off exponentially by some minutes and try to submit query again. [ Wierd, but thats an official answer…]
2)  highly recommended to adopt Amazon Athena best practices [1] to optimize your query and your data.
3) Use columnar formatted data which can drastically reduce the resource consumption.

[1] Top 10 Performance Tuning Best Practices for Athena — https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-tips-for-amazon-athena/

[2]  aws atehna Cost reduction (might also reduce resource consumption)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s