Algorithm design and analysis separation

Question

Should the people who analyse the output and results of an algorithm be aware of its design?
By analysis I mean finding cases where the algorithm failed and returned bad results.

Answer 1

There are two types of testing, first is technical testing to make sure the code embodying the algorithm functions, especially within the boundaries specified by the design, and particularly at edge cases.

The second form is functional testing to see whether the outputs created by the algorithm correspond to the design intention. For that you don't necessarily need to know about the implementation and there is a reasonable argument to be made to say that it is done more objectively if you don't.

Of course if the algorithm is trivial and all possiblel outcomes can be known, then testing it is simply a matter of producing test data which exercised the boundary conditions.

However, depending on the algorithm, that may not be possible. In cases where the algorithm is analysing data in a space which is not easily visualised (say it has 27 dimensions and 1M data points) then only trivial cases can be tested for functional accuracy. It gets worse if there is no single "right" answer from the analysis, but an array of plausible outcomes. In those cases an experienced analyst is needed to look at the results to see whether they make sense. Sometimes that is the same person who implements the code, sometimes not. In those cases the implementation will almost certainly be questioned when trying to understand the results, which will be complex themselves.

So, short answer is yes, if possible, but it's not always necessary of the algorithm is completely explicable.

Answer 2

There are two different types of unit testing:

1. Blackbox
2. Glassbox

Both are incredibly important. Blackbox tests should be written based solely on the documentation of the function (its stated inputs, its stated preconditions, its stated outputs, its stated postconditions, and its stated error cases and behavior/results in the event of such errors). Glassbox testing tries to ensure that all branches of the implementation work correctly, and requires looking at the implementation. Such tests generally provide a few example inputs for each branch of execution.

As for analysis... generally when people speak of "algorithm analysis", they are not talking about a particular implementation, but rather a theoretic analysis of the algorithm, usually using its pseudo-code, and usually for the purpose of proving the algorithm's correctness and an upper-bound on its worst-case or expected case running time. Such analyses generally require the pseudo-code (or at least some other formal description of the algorithm's behavior) in order to reason and prove properties about it.