Place the xml files in a certain folder, add web.config to this folder containing:
<configuration>
<system.web>
<authorization>
<deny users="?"/>
<allow roles="admin"/>
</authorization>
</system.web>
</configuration>
Change the '?' (which means anonymous users - i.e. not logged in users) to '*' in order to deny all users (the server will have access [e.g. via Server.MapPath etc.]).
Respectively you can play with the roles or remove this line.
Also, consider that in the web.config file you can deny and allow specific extensions as follows:
<system.web>
<httpHandlers>
<remove verb="*" path="*.xml" />
<!--or-->
<add verb="*" path="*.xml" type="System.Web.HttpForbiddenHandler" />
</httpHandlers>
</system.web>
Please don't rely on this last snippet till you make sure what are your needs.
You can find out more on Http Handlers, or take a look at How to: Register HTTP Handlers.
I also noticed someone asked a similar question here, you may find it helpful.
Hope you to find your quickly find your solution, good luck!