Yestarday I had the same need to use the popup extender and here are the js files which you should include first.
<script src="javascripts/MicrosoftAjax.js" type="text/javascript" ></script>
<script src="javascripts/AjaxControlToolkit/AjaxControlToolkit.Compat.Timer.Timer.js" type="text/javascript"></script>
<script src="javascripts/AjaxControlToolkit/AjaxControlToolkit.Common.Common.js" type="text/javascript"></script>
<script src="javascripts/AjaxControlToolkit/AjaxControlToolkit.ExtenderBase.BaseScripts.js" type="text/javascript"></script>
<script src="javascripts/AjaxControlToolkit/AjaxControlToolkit.Animation.Animations.js" type="text/javascript"></script>
<script src="javascripts/AjaxControlToolkit/AjaxControlToolkit.DropShadow.DropShadowBehavior.js" type="text/javascript"></script>
<script src="javascripts/AjaxControlToolkit/AjaxControlToolkit.DynamicPopulate.DynamicPopulateBehavior.js" type="text/javascript"></script>
<script src="javascripts/AjaxControlToolkit/AjaxControlToolkit.PopupExtender.PopupBehavior.js" type="text/javascript"></script>
The next step is to create the PopupBehavior from javascript like that:
$create(AjaxControlToolkit.PopupControlBehavior, {"PopupControlID":"div_to_popup","Position":3}, null, null, $get("textbox_input_id"));
If you need to use another control from the AjaxControlToolkit with scripts only without the burden of asp, here is one easy way to find the necessary include js files.
Go to its demo page here and look at the source of the page. At the bottom you will see how to create the control. To find the include files needed for this control select CombineScriptsHandler.ashx and search for "//START". Every line with "//START" shows a script which is included in the page.