A: 

therow = new Array (0,3,2,1,9,5,7,6,8,4,11,10)

That's an array of Number.

therow = form.inputbox.value.split(",");

That's an array of String.

You then attempt to do arithmetic on the strings. '2'+'3'='23' not '5', so you get unexpected results.

for (var i= therow.length; i-->0;)
    therow[i]= +therow[i]; // convert to Number

Also matrix() seems much more complicated that it needs to be. How about:

function matrix(row) {
    var table= document.getElementById('matrixbox');
    if (table)
        table.parentNode.removeChild(table);
    table= document.createElement('table');
    table.id= 'matrixbox';

    for (var i= 0; i<row.length; i++) {
        var tr= table.insertRow(i);
        for (var j= 0; j<row.length; j++) {
            // this is the actual calculation here
            var value= (row[j]-row[i] +row.length)%row.length;
            tr.insertCell(j).appendChild(document.createTextNode(value));
        }
    }

    document.getElementById('matrixarea').appendChild(table);
}

Then, to get rid of that nasty therow global, call using:

<body onload="matrix([0,3,2,1,9,5,7,6,8,4,11,10])">

and:

function execute(form) {
    var row= form.elements.inputbox.value.split(',');
    for (var i= row.length; i-->0;)
        row[i]= +row[i];
    matrix(row);
}

As a bonus, will work for any row length.

(Forget cellpadding/cellspacing; set ‘padding’ and ‘border-spacing’ through CSS instead. And don't use setAttribute() on HTML elements, as there are bugs in it on IE. And it's dead ugly).

bobince
That's it, thank you very much.Btw, I couldn't run your matrix function, there's no table at all, do I miss something?Thank you for other advises too, I really appreciate your help.
buba
Works for me, you just have to make sure to call it in your onload as above.
bobince
http://debuggable.com/posts/7+8===7-in-javascript:4acba016-d204-489b-b5a0-1fd0cbdd56cb
powtac
Luckily the unary `+` operator equates to `parseFloat` rather than `parseInt`; the former does not suffer from the Octal Problem.
bobince
Check it out: http://abbasmacioglu.home.anadolu.edu.tr/m2.html
buba
Btw, there's another weird thing in my script. If the first number isn't zero, the results become wrong again. Fortunately it's not so important.
buba
Whoops! Sorry, I renamed a variable wrongly, sorry — replace the ‘row=’ and subsequently ‘row.insertCell’ with ‘td’ instead (updated answer).
bobince
What are the results supposed to be for non-zero beginnings? Currently they're all relative to each other so naturally all results will be normalised to a zero first entry.
bobince
Your script is incredible, whew!! May I use it instead of mine :)About non-zero beginnings, they supposed to be seen diagonally cross the table like zeros.
buba
A: 
buba