Edit: To be more clear, this is your problem:
This:
if (other instanceof Square)
Square realOther = (Square) other;
else
Rectangle realOther = (Rectangle) other;
is the exact same thing as this:
if (other instanceof Square) {
Square realOther = (Square) other;
} else {
Rectangle realOther = (Rectangle) other;
}
Notice that you never do anything with realOther. You define it there, but since it's scoped to that set of braces, you throw it away as soon as hitting the closing brace.
Not including the braces doesn't help, the fact that it's a statement (after an if) makes it an independent code block, as soon as that block exits, "realOther" is discarded.
Java recognizes that this is the case and tells you (To sloppily quote Inigo Montoya) "I don't think that means what you think it means" .
Secondary problem, Java is strongly typed. Even if it might allow syntax like you are saying, right after this if statement, it has to have an absolute final type for realOther. Your if statement would completely break java's brain because it wouldn't know if realOther was supposed to be treated as a rectangle or a square.
Suppose rectangle had a setWidth and setHeight method (and square just had a setHeight). After this IF statement, should Java allow you to call setWidth or not? Java MUST know these things at compile time (if not, we wouldn't need casting).
another edit:
To fix it, you use this:
// class Rectangle extends Square
// Both implement setHeight, Rect also implements setWidth.
public void reSize(Square other, int h, int w) {
other.setHeight(h);
if (realOther instanceof Rectangle) {
((Rectangle)realOther).setWidth(w);
}
In other words, the way you're supposed to work with related objects is to leave them as the more abstract object for as long as possible, only using the specific objects when you need to call a method on that object.
If you code it right, you can almost always use the more abstract and rarely use the specific. With square and rect, this isn't obvious, square/rect is a pretty sloppy problem and is used as an example of some tough decisions you have to make with OO programming, but it really does work out nearly all the time--I very rarely have to cast at all...