Since you are doing this in MATLAB, here is one solution that should work:
crossProds = [V1(1)*V2(2)-V1(2)*V2(1) ...
V1(1)*V3(2)-V1(2)*V3(1) ...
V3(1)*V2(2)-V3(2)*V2(1)];
if (all(crossProds >= 0) || ...
(crossProds(1) < 0) && ~all(crossProds(2:3) < 0)),
disp("V3 is between V1 and V2");
else
disp("out of the interval");
end
EXPLANATION:
The cross product between the 2-D vectors V1 and V2 is stored in the first element of crossProds. This value will be greater than or equal to zero if the counter-clockwise angle between V1 and V2 is between 0 and 180 degrees, inclusive. In this case, when V3 is between V1 and V2 in the counter-clockwise direction then the cross products (V1,V3) and (V3,V2) are also greater than or equal to zero. This explains the first logical check:
all(crossProds >= 0)
If the counter-clockwise angle between V1 and V2 is greater than 180 degrees, then the cross product of these two vectors will be less than zero. In this case, when V3 is between V1 and V2 in the clockwise direction then the cross products (V1,V3) and (V3,V2) are also less than zero. Therefore, if these cross products are not both less than zero then V3 must be between V1 and V2 in the counter-clockwise direction. This explains the next two logical checks:
(crossProds(1) < 0) && ~all(crossProds(2:3) < 0)
The above logical checks should cover all possible situations. The operators || and && are short circuit operators in MATLAB: they will skip the second statements if they are not necessary. For example, if the first statement in an OR is true, there is no reason to check the second statement since only one argument in an OR needs to be true for the result to be true.