After watching this presentation from PDC09, I have a much better idea of how the rendering system in Silverlight works. This problem isn't directly addressed by the presentation, but knowing the rendering order of things helps.
The order of the rendering steps relevant to my question is: An object's children (and/or itself) are rendered, that rendering passes through the Effect and then passes through the RenderTransform.
It appears that in any case where a RenderTransform is applied to an object where that object or its children have had an Effect applied (ie: a RenderTransform that comes after an Effect in the rendering tree), that RenderTransform is done in a "low quality" mode that produces this "fuzzyness".
The solution, then, is to move the Effect to after the RenderTransform. In my case this means putting the Image on its own Canvas, applying the RenderTransform to the Image, and the Effect to the Canvas.