In practice custom controls are something you implement on the code level while you can use XAML for user controls. The custom controls extend one of the WPF control base classes and provide additional functionality through code so all the added logic and representation must be implemented inside the code.
A user control is technically a normal content control which you can extend in some parts in the code but usually it is extended by placing other controls inside it. So as Kent mentioned a UserControl is an aggregation of other controls. This limits what you can do with a user control considerably. It's easier to use but more limited than a full custom control.
These controls have a small difference from a runtime point of view. When building an application and placing an UserControl into it, the control tree will have a concrete UserControl template inside of it. So if we consider a lame example of a specialized button. If you were using a user control you'd add a button inside the <UserControl>
element. When using a custom control you'd derive the control itself from a button most likely. The difference would be visible in the logical tree.
While the custom control would provide a logical tree similar to
The UserControl would give a logical tree of
So in the end the UserControl is just a normal ContentControl which you can extend a bit and for which you can predefine the content. Custom control provides greater flexibility at the price of ease of implementation as you have to do all the logic and interaction in the code instead of having the benefit of XAML.
Though after all this, I don't think there's that much difference in the Visual Studio templates. Most likely the Visual Studio Custom Control just creates a project with an empty custom control while the User Control project is a project with an empty user control. You can later add any kind of items to the project.
Update
And my opinion on when to use custom control and user control is that if you can get something done with a user control and the extra control element in the logical tree doesn't bother you, use a user control as they are so much easier to create and maintain. Use a custom control only if you have a reason not to use a user control.