The problem is that (according MS) you need to respond ALWAYS in the same way, no matter the specific error you have.
You'd need to redirect the user to the same page on errors 404 and 500. That's why the easiest way would be using the web.config setting.
They say that this would be temporary and you could revert it once they release a patch for this.
This is the Scott answer to a similar question:
I would recommend temporarily updating the module to always redirect to the search page. One of the ways this attack works is that looks for differentiation between 404s and 500 errors. Always returning the same HTTP code and sending them to the same place is one way to help block it.
Note that when the patch comes out to fix this, you won't need to do this (and can revert back to the old behavior). But for right now I'd recommend not differentiating between 404s and 500s to clients.