Delegates are simply .Net's implementation of first class functions and allow the languages using them to provide Higher Order Functions.
The principle benefit of this sort of style is that common aspects can be abstracted out into a function which does just what it needs to do (for example traversing a data structure) and is provided another function (or functions) that it asks to do something as it goes along.
The canonical functional examples are map and fold which can be changed to do all sorts of things by the provision of some other operation.
If you want to sum a list of T's and have some function add which takes two T's and adds them together then (via partial application) fold add 0
becomes sum. fold multiply 1
would become the product, fold max 0
the maximum. In all these examples the programmer need not think about how to iterate over the input data, need not worry about what to do if the input is empty.
These are simple examples (though they can be surprisingly powerful when combined with others) but consider tree traversal (a more complex task) all of that can be abstracted away behind a treefold
function. Writing of the tree fold function can be hard, but once done it can be re-used widely without having to worry about bugs.
This is similar in concept and design to the addition of foreach loop constructs to traditional imperative languages, the idea being that you don't have to write the loop control yourself (since it introduces the chance of off by one errors, increases verbosity that gets in the way of what you are doing to each entry instead showing how you are getting each entry. Higher order functions simply allow you to separate the traversal of a structure from what to do while traversing extensibly within the language itself.
It should be noted that delegates in c# have been largely superseded by lambdas because the compiler can simply treat it as a less verbose delegate if it wants but is also free to pass through the expression the lambda represents to the function it is passed to to allow (often complex) restructuring or re-targeting of the desire into some other domain like database queries via Linq-to-Sql.
A principle benefit of the .net delegate model over c-style function pointers is that they are actually a tuple (two pieces of data) the function to call and the optional object on which the function is to be called. This allows you to pass about functions with state which is even more powerful. Since the compiler can use this to construct classes behind your back(1), instantiate a new instance of this class and place local variables into it thus allowing closures.
(1) it doesn't have to always do this, but for now that is an implementation detail