How can I handle a list with several hundred items in HTML? (Using Javascript)

Question

I am creating a web page which is designed for mobile safari, and I create a list dynamically. The list is created from a text file in which I receive information and I append this information to the list. It works fine when there are about 200 items, but when the file is very large, (I have tried with up to 4000 items) the page becomes very slow, scrolling and selecting this items is very hard. I know I should not create so much HTML elements, but I am looking for a way to create a shorter list and replace the information on the list's elements depending on how much scrolling you have done. Any Ideas?

Answer 1

Instead of appending the items to the list, how about displaying up to 200 at once, then replacing the current items on a scroll forward - scroll back motion? You could use innerHTML for this, or DOM methods that create the elements.

Answer 2

You could create something like "virtual scrolling" - show only a few items (that can be viewed on the screen) and make the scrollbar appear by offsetting them with margins. This way, you'll have the user think he has 4000 items on the screen and have only 20 items in memory.

Answer 3

Use a dataTable. Two I know of are YUI2 dataTable and www.dataTables.net. I use www.dataTables.net because it's really nice and I'm quite used to it so far (not an expert though). You can use server-side processing which you can use to call a php file or method or other javascript, but it will grab x number of records for you and know where to start when you paginate through (pagination is already provided with the dataTable.

You'd instantiate the table like so:

<script type="text/javascript" char="utf-8>
  $(document).ready(function() {
    $('#tableName').dataTable({
      "bProcessing": true,
      "bServerSide": true,
      "sAjaxSource": "php_filename.php",
      "sPaginationType": "full_numbers"
    });
  });
</script>

That's an example image of what it would look like. The only issue with doing server-side processing is you'd have to pass the information in JSON format.

 "aaData": [
  "info1",
  "info2",
  etc...
]

Controls can be added as well into the aaData records. Here is a good example. Hope this helps.

Update 2: You will have to download the jQuery file (1.4.1 is the version I used) and you'll have to reference the jQuery file, the dataTable.js file, and the css. The class name of the table will have to be "display". The css name is demo_table.css.

<script type="text/javascript" language="javascript" src="../Scripts/jquery-1.4.1.js"></script>
<script type="text/javascript" language="javascript" src="../Scripts/jquery.dataTables.js"></script>

<style type="text/css" title="currentStyle">
  @import "../../Content/stylesheets/demo_table.css"
</style>

Answer 4

Even if you want to render the entire big list at one go, use a document fragment

Answer 5

Clarification

To keep things simple I will consume right from the start the possibility of using a specialized JavaScript UI library. Personally I would not use such a solution since getting involved with a library creates additional constrains (and possibly bloat) to your project (unless you are working on a project that relies heavily on client-side UI components, in which case you should choose one UI library and stick with it).

Also, I will not consider "scrolling" as a solution (i.e. creating an endless list of DOM objects and simply scrolling through them). IMHO that is not really a "solution" to the problem, it is simply a method of generating the problem that led you here.

Think flow

You have to keep in mind that on the client side, the JavaScript language is in a sense "attached" to the browser's Document Object model. Your JavaScript code is practically "telling" the browser to modify the DOM hierarchy in a myriad of ways, and then the browser conforms. This is the main reason why pure JavaScript code tends to run fast and then things slow down dramatically when starting to manipulate DOM objects.

The main rule to follow, in my opinion, is to minimize as much as humanly possible the amount of DOM object manipulation by following this flow:

1. If possible, do it server-side: this can prove to be a very consistent and reliable solution; if your project allows for this, you should only send reasonably-sized segments of the data to the browser at any single time (e.g. one page from the entire set), this will allow the client to properly display and manipulate that data in a speedy and responsive way.

2. If you must do it on the client-side, avoid touching the DOM as much as possible: this is essential; if you receive 4000 objects in a JS script, keep them in memory, don't rush to translate all those elements into DOM objects; that's a death sentence.

3. Only manipulate the DOM at the last possible moment and touching the least amount of objects: since modifying the DOM takes a lot of effort (compared with other in-memory operations). Another danger here is memory leaking (browsers aren't perfect), since when creating and discarding many DOM objects, some of those discarded object might still survive in memory, thus producing a memory leak (which could eventually hog the browser or even the system).

Solutions

These are the approaches on the table:

1. Server-side pagination: perhaps this is obvious, but too many times people try to overlook this option, and I think that's wrong. Again, if your projects allows this, it is very difficult for such a solution to fail you: it can be very consistent and reliable (which is why it is a classic).

2. Client-side pagination: Obviously you could send the entire data to the browser, handle it in JS and then display it in reasonably-sized chunks at once, never exceeding a certain amount of generated DOM objects. This might seem attractive and more simple than Solution 1, however, beware that if your data is bidirectional (i.e. implies intervention and feedback from the user that must return to the server) and important (e.g. more important that statistical/reporting data that only needs to be displayed), you should probably avoid this option.

   The reason is that even if JavaScript is fast and efficient enough to juggle this data in memory, it is not as safe; you never know what might happen client-side: the memory could get corrupted, the browser might crash etc.; basically you have no reasonable guarantee for your data. A server is better prepared to handle important data. What's more, it will be very difficult to have history navigation between pages and URL addresses that go to specific pages within the data-set (both are possible but not without a headache anyway).

3. Client-side dynamic scrolling: instead of displaying the objects from JS memory as full pages, you simply show a reasonable sub-set, let's say 100 items and then scroll down by taking one object (or more) from the top and moving it to the bottom of the list, altering it's contents accordingly. Obviously this option presents the same dangers as Solution 2, but it is still a very good solution.

Example

Considering that your projects works for mobile devices I consider the last approach as probably more feasible to you, so here is a (very) simplistic example of how it might be done using MooTools (obviously the principle can be applied using any framework):

<html>

<head>

    <title>Endless scrolling example.</title>

    <!-- Include the MooTools framework. -->
    <script type="text/javascript" src="mootools-1.2.4-core-yc.js"></script>

    <!-- Note: we'll use id's to select objects since it's faster. -->

</head>

<body>

    <!-- Scroll up. -->
    <a href="javascript:void(0);" id="list_up_button">Up!</a>

    <!-- The list (i.e. container). -->
    <ul id="list_container">
    </ul>

    <!-- Scroll down. -->
    <a href="javascript:void(0);" id="list_down_button">Down!</a>

</body>

<!-- Our script. -->
<script type="text/javascript">

    window.addEvent('domready', function() {

        // database
        var list = {};

        // options
        var list_size = 5000;   // total list size
        var list_offset = 0;    // initial list position
        var list_subset = 40;   // the smount of displayed items
        var scroll_step = 10;    // items to scroll in one step
        var time_delay = 50;    // time delay between scroll steps

        // make dummy items
        for (var i = 0; i < list_size; i++) {
            var red = Math.floor(i * 2) % 256;
            var green = Math.floor(i * 3) % 256;
            var blue = Math.floor(i * 4) % 256;

            list[i] = "<span style=\"color:rgb(" + red + ", " + green + ", " + blue + ");\">" +
                      "My name is 'Object no." + (i + 1) + "'." +
                      "</span>";
        }

        // get container
        var list_container = $('list_container')

        // generate DOM objects
        for (var i = 0; i < list_subset; i++) {
            list_container.grab(new Element('li', { html: list[i] }));
        }

        // Up scroller.
        function up() {

            // check end
            if (list_offset <= 0) {
                return false;
            }

            // get element
            var element = list_container.getLast();

            // devance offset
            list_offset--;

            // re-write element
            element.set('html', list[list_offset]);

            // move top element to top
            list_container.grab(element, 'top');

            // success
            return true;
        }

        // Down scroller.
        function down() {

            // check end
            if (list_offset >= list_size - list_subset) {
                return false;
            }

            // get element
            var element = list_container.getFirst();

            // advance offset
            list_offset++;

            // re-write element
            element.set('html', list[list_offset + list_subset - 1]);

            // move top element to bottom
            list_container.grab(element, 'bottom');

            // success
            return true;
        }

        // Repeater function.
        function repeater(direction) {

            for (var i = 0; i < scroll_step; i++) {

                // scroll
                if (direction() == false) {

                    // deactivate repeater
                    $clear(list.scroll_period);
                }
            }

        }

        // configure up scroll
        $('list_up_button').addEvents({

            // activate scroll
            'mouseenter': function() {
                list.scroll_period = repeater.periodical(time_delay, null, up);
            },

            // deactivate scroll
            'mouseleave': function() {
                $clear(list.scroll_period);
            }
        });

        // configure up scroll
        $('list_down_button').addEvents({

            // activate scroll
            'mouseenter': function() {
                list.scroll_period = repeater.periodical(time_delay, null, down);
            },

            // deactivate scroll
            'mouseleave': function() {
                $clear(list.scroll_period);
            }
        });

    });

</script>

Sorry for the rather long code... hope it helps.