I only need to know which angle (x,y,z vector) my cursor is pointing at. So i can shoot any particle in the direction i am pointing at.
I cant use depth testing, so the typical picking methods are not an option.
Is this possible with pure math only? if i know the view port aspect ratio and perspective etc?
Edit: I need to get the coordinate from empty scene. Totally empty, no drawing at all!
Unless your bullet of choice detonates at some range (so distance to the thing is important) then you only need the intersection point of your boresight with a plane to determine your angles.
Constrain your cursor to lie in a plane that's perpendicular to your look direction, and some distance away from you. Then take the displacement from the location of the exit of the gun barrel to the location of the cursor, and calculate your angles from there.
The simple answer is that part of rendering a scene involves converting world co-ordinates to view co-ordinates - where's all that stuff relative to my eyeballs, basically.
The view plane is the screen, which is what the mouse sits on. If your mouse co-ordinates are (xx, yy) then it is probably positioned at either (xx, yy, 1) or (xx, yy, -1) in view co-ordinates. You'll probably need to use a system where (0, 0) is the centre of your viewport on screen - not how mouse co-ordinates normally work, but constant offsets are easy enough.
zz=0 is where the player is, whereas the view plane is a short distance in front, between the player and what he can see. That's why the view plane is taken as either (xx, yy, 1) or (xx, yy, -1) here - if you use +1 (the normal case, I think) objects with view co-ordinates that have positive z are in front of you.
To convert this back to world co-ordinates, apply the same transform you do to transform your world co-ordinates to view co-ordinates, but backwards. You then get your mouse pointer position in world co-ordinates. To get the direction vector, subtract your player position.
The backwards conversion uses the inverse matrix at each step, and applies the steps backwards. In principle, all steps can be combined into a single matrix, and an inverse matrix can be calculated for the whole transform - but that's not the best approach. It's much easier, for instance, to calculate the inverse of a rotation than to calculate a general matrix inverse. If your transform from world co-ordinates to view co-ordinates is composed as needed from components (postion, orientation...) you really only need to calculate inversed components and combine them. Instead of 30-degrees-left (from world to view) you use 30-degrees-right (from view to world) etc.
If you're lucky, you might be able to drop the whole issue of player position completely - calculate the angle for the mouse pointer in view space (Euler angles are enough for this), then apply the orientation part only of the view-to-world transform.
No here's-the-formula, sorry, partly because the view-to-world transform depends on how you do the world-to-view transform and partly because I don't have an easy example to hand.
Though the question is different, you can find some relevant references in the answers here...
If you consider your entire scene to fit within a pyramid, with your eye at the peak (looking toward the base), your screen is a cross-section parallel to the base, on which the entire scene is projected (into 2D space). Presumably, you know the dimensions of the screen, and the location of the cursor in 2D-space.
You can look at that pyramid from the X side, and then again from the Y side, to turn the problem into two right triangles. The base of the triangle is defined by the line from the "eye" to the center of the screen, and the hypotenuse is defined by the line from the "eye" to the cursor.
With the cursor at the extreme corners of the 2D display, you know the angle (given perspective information), so you can determine how far the "eye" is from the 2D projection plane (or the length of the base of your triangles), in pixels. Using that length, and the length of the other side of the triangle (the x- or y-coordinate of the cursor), you can determine the angle. All of this uses basic trig.
If it helps, draw out all the triangles in question:
(0, 0); (base, 0); (base, Y_MAX) where Y_MAX is the maximum value for the cursor (the top edge of the screen). base = Y/tan(max_angle)(0, 0); (base, 0); (base, X_MAX).(0, 0); (base, 0); (base, Y), where Y is the y-position of the cursor. angle = (tan^(-1))(Y/base)(0, 0); (base, 0); (base, X).