No you cannot set the prototype and have that change effect previously existing instances. You can however, modify the prototype object to have the desired effect. But not the way you have attempted. This is because your instance X is an object, not a class. (This is because you used the "new" operator before the definition of your class.)
Simple test:
var X = new function () { };
alert(X.prototype); // displays "undefined" because X is an
// object not a class
So when you set the "prototype" attribute on X you are not storing the template object in a class, you are merely creating a new attribute on object X with the name "prototype" so, if you did this:
var Y = function () {
this.m = function () {
alert('this is m');
};
};
var X = new function () { };
X.prototype = new Y();
X.prototype.m(); // displays "this is m"
Then you get the output you expected previously.
Even though you created X as an instance of an anonymous class, you can still access that class through the "constructor" property. Once you have that reference, you can then modify the prototype object. Like this:
X.constructor.prototype.m = new Y().m;
X.m(); // displays "this is m"
You should note however, that this will not work:
X.constructor.prototype = new Y();
X.m(); // fails
Because changing the entire prototype object only effects instances created in the future.