Testing a image/video conversion class

Question

I have a class that converts images and videos to other formats. A number of options are available such as size, scaling mode, background color, etc (think ImageMagick's convert command). Any ideas for how to test it? The code to exercise all of the options isn't that hard to write, but I'm unsure about how to confirm the results are valid, especially for video.

One idea is to run the resulting file through identify or mediainfo and look to see that the results are what I'm expecting. This wouldn't work for an image that's the right size, etc, but looks completely wrong.

Another idea would be to manually confirm the resulting files once, save those files, and then compare them to the newly converted files each time the test is run. The downside of this is the need to have #inputs x #options output files, which could get big, especially with video.

Is this a fool's errand or is there a reasonable way to get this done?

Answer 1

For images, you could compute a few simple metrics (color histogram, edge histogram, etc.) and verify that they're the same (within some tolerance) before and after conversion.

For video, you could do the same thing with frames sampled at some interval. Not a perfect guarantee, of course, but it should work pretty well in practice.

Of course, you'll have to either pick metrics that are invariant to the kinds of changes produced by your conversion process, or account for the expected change in your comparison function.