You're looking for a non-trivial solution v to A*v=v with v=[x;y;z] and...
A =
0.70710678118655 0 0.70710678118655
-0.50000000000000 0.70710678118655 0.50000000000000
-0.50000000000000 -0.70710678118655 0.50000000000000
You can transform this into (A-I)v=0 where I is the 3x3 identity matrix. What you have to do to find a nontrivial solution is checking the null space of A-I:
>> null(A-eye(3))
ans =
0.67859834454585
-0.67859834454585
0.28108463771482
So, you have a onedimensional nullspace. Otherwise you'd see more than one column. Every linear combination of the columns is a point in this null space that A-I maps to the null vector. So, every multiple of this vector is a solution to your problem.
Actually, your matrix A is a rotation matrix of the first kind because det(A)=1 and A'*A=identity. So it has an eigenvalue of 1 with the rotation axis as corresponding eigenvector. The vector I computed above is the normalized rotation axis.
Note: For this I replaced your 0.7071 with sqrt(0.5). If rounding errors are a concern but you know in advance that there has to be a nontrivial solution the best bet is to do a singular value decomposition of A-I and pick the right most right singular vector:
>> [u,s,v] = svd(A-eye(3));
>> v(:,end)
ans =
0.67859834454585
-0.67859834454585
0.28108463771482
This way you can calculate a vector v that minimizes |A*v-v| under the constraint that |v|=1 where |.| is the Euclidean norm.