First of all, you can't reference to a second row using a rownum = 2 condition. You can either select first two rows by specifying a where rownum < 3 condition, or you may wrap it in another query and reference your rownum as an ordinary column from over there.
Then, you can't reference a column alias in a where clause of a subquery this alias was assigned.
You can either bring it one level up:
SELECT instmax
  FROM (SELECT instmax, rownum r
          FROM (SELECT instmax
                  FROM pswlinstmax
                 ORDER BY instmax DESC NULLS LAST)
         )
         WHERE r = 2;
or just avoid this reference  
-- this will return first two rows
SELECT instmax
  FROM (SELECT instmax, rownum r
          FROM (SELECT instmax
                  FROM pswlinstmax
                 ORDER BY instmax DESC NULLS LAST)
         WHERE rownum < 3
         );