You will always have memory problems when the Java code hauls/copies the entire dataset of a datastore (e.g. a RDBMS) in Java's memory and then do the sorting and filtering right in Java's memory using Java code. It will get worse when you even store it in the session scope of a webapplication.
The most memory efficient approach is to let the DB do the task where it is invented for. The SQL language offers you under each the ORDER BY clause to do the sorting, the WHERE clause to do the filtering and the (DB vendor specific) LIMIT/OFFSET clauses/subselects/functions to return only a subset of records based on firstrow and rowcount or lastrow. This way you end up with only the dataset in Java's memory which is actually to be displayed.
There is no standard JSF component which does exactly that. It will always require the entire dataset being available in the Java memory, because the filtering and sorting needs to happen with pure Java code or Javascripts. JSF knows nothing about SQL, you'll need to provide a custom implementation of javax.faces.model.DataModel to the datatable and/or control/manage that in the data access layer yourself. You can get a lot of new insights/ideas and find a kickoff example in this article. You can also find examples of the needed SQL queries in this JSP-targeted answer I posted before here.
Good luck.