What are your criteria for measuring the relative superiority of programs for image processing ? For example, if you are a Mathematica expert then you will find it easier to use Mathematica for the task. On the other hand, if you are a penniless student then you will find Java and some of its libraries more to your taste.
EDIT: in answer to OP's comments ...
'ease of image processing' is entirely subjective -- if you don't know Mathematica then it will be difficult to use it for image processing -- so this one is your call.
'processing time' is entirely objective -- but do you have the time to try out all 3 of your suggested options and compare them ? For a BE project you'll be far better using the tool you are most comfortable with and spending as little time as you can wrestling with an unfamiliar tool for the sake of a bit of extra speed.
'cellular automata' for image processing -- don't know how relevant it is, but Mathematica has inbuilt functionality for cellular automata.