oData Pagination: how to get more results at once
There was a recent change in oData in which includes pagination for reports. This caused every request to be limited to 500 rows and you have to use the parameter $skip in the URL to get the following rows recursively until you get them all. The problem is that depending on the report each request is taking an average of 15-20 seconds. This causes a report with 100.000 rows to take more than 1 hour to load. Before the pagination, that could be load in 1-2 minutes. This is a huge issue for performance, especially when using AWS lambda for the ETL, because it is limited to 15 minutes.
My question is if there is any way to bypass that pagination. I tried using the url paramenter $top. It works if I set it to a number less than 500, but unfortunately, that number is also limited to 500.
What can I do to load the reports faster?