Scott Hanselman says yes.
Adding System.Web to your non-web project is a good way to get folks to panic. Another is adding a reference to Microsoft.VisualBasic in a C# application. Both are reasonable and darned useful things to do, though.
MSDN says no.
The Cache class is not intended for use outside of ASP.NET applications. It was designed and tested for use in ASP.NET to provide caching for Web applications. In other types of applications, such as console applications or Windows Forms applications, ASP.NET caching might not work correctly.
So what should I think?