I am using percentile in one of my sql's but due to large data skewness, the query is failing with out of memory error. Even after increasing memory, the query is failing. I found this percentile_approx function which works on my dataset but there are differences between percentile and percentile_approx.
Can some kind soul please explain the differences?
My dataset contains dollar amounts up to 2 decimal points. The number of rows will vary from month to month but it's usually over 300 million rows.
Thanks
question from:https://stackoverflow.com/questions/66067352/percentile-vs-percentile-approx-in-sparksql
