In the absence of a more comprehensive error message from the native wrapper, use truss to identify the low-level system call which fails (most probably with EACCES), while running as the non-root user:
truss -l -d -f -vall -wall -o truss.out java ... TestOrAppClass
What you expect to see in truss.out after the Java application throws the exception and terminates (or is terminated) -- note the interleaved spaces in I O E x c e p t i o n:
...
...
...
...
23515/1: 0.2912 some_system_call(params) Err#13 EACCES [ALL]
...
23515/1: 0.2923 write(2, 0x08044F84, 39) = 39
23515/1: j a v a . i o . I O E x c e p t i o n : P e r m i s s i o n
23515/1: d e n i e d
...
...
...
...
If the application in question is too complex and generates too much output (or is otherwise difficult to invoke from the command line), then write and compile a small test class which reproduces the problem by calling java.nio.channels.Selector.open() directly.
If you do not/cannot write a simple test class then remember that you can always use the extra output logged by the -wall option to correlate the time (and thread/LWP) at which the stack trace printout was produced with the actual system call, which should have failed shortly before.
In the sample truss.out output above, 23515 is the process ID, /1 is the LWP ID (doubling as "thread" identifier if the JVM uses native threads, which it almost certainly does on Solaris), 0.2912 is the timestamp (seconds since java process/trace startup), and some_system_call failing with EACCES is the call of interest.
What version of the JDK are you using?