Before going any further, it would be good to state some data volumes, such as number of rows you are expecting.
Otherwise OS consultants may waste time proposing solutions that may not suit your specific data volumes.
I don't think there is a way of increasing the memory allocation for TableViews. It is going to be highly dependent on the memory on the client PC/laptop, not just on the server.
One option would be to split up the TableView into chunks, and each sheet in the workbook retrieves a separate chunk of the data table. That would reduce the memory demands compared to a single monster-sized data retrieve.
Another option would be to find out what the end-use of this report would be (such as loading the data into a Data Warehouse?), and treat this as a data integration exercise rather than a reporting exercise, and not pass the data through the client PC.
Apologies if I am making suggestions you have already discarded : pls consider we don't have the background context to this exercise.
The last resort would be using a CSV extract, but then of course you get into all sorts of problems with number & date formats, meanings of commas, and expected delimiter characters.