I have a solution containing 4 class library projects and one "web site" project. The web site project contains references to the 3 class library projects, whenever I make changes in any of the class library projects, the only option I see is to rebuild the web site which takes a lot of time.
Is there any option that I can only update the dll references and the changes are reflected in the web site project ?
You should rebuild everything, yes. When you change one of the class libraries, that could subtly change how the code is called. There are obvious changes like removing a method that your web site is relying on, but there are sneakier ones like adding an overload - so that the code will still compile, but have a different effect.
I don't know of any way to tell Visual Studio not to do a rebuild when a dependency changes... I would personally look at finding a way to make the overall build faster instead.
If you're doing a lot of work in the class libraries and you don't actually need the web site for the moment, you could always unload that project until you need to run the whole site again.