If you are developing a single Silverlight application that you want to deliver then you need only serve the XAP.
However if you are not the application developer or you want to deliver multiple apps effeciently then your web server needs also to be able to deliver other files that may come along with these apps. For example the libraries may be be delivered as zip files and they may download external images and XML files. Still this is all likely to be simple static content you will not normally need to implement other services.
Note if you are hosting an app to be referenced by a HTML file served by some other server then you need to get your site to respond with appropriate XML when SL requests the clientaccesspolicy.xml file.