Most difficult, hmmm
First and foremost the most difficult part of project managment is that you are dealing with people. People who don't know what they want, prima donnas who don't want to do what the client wants, people who have sudden personal problems that affect your schedule such as these that I've personally encountered at work:
the analyst who left her husband the
week before a deadline when she
hadn't finished the requirements yet
The dev who found out a two days
before the deadline that his wife had
lung cancer
The dev whose husband died four
days before a deadline
Next most difficult - pining down the actual requirements. First because people don't know what they want and second because there are many small details that are hard to know about especially if they are in the back end and not the user interface.
Project managers are often in the middle and have to deal with many crises. They are the ones the who get slammed when the the client discovers a bug or when the application doesn't work the way it was expected to work (even though that particular expectation was often not ever expressed). They are the ones who get to tell the client that the project is running behind or over budget or that the requirements need to be scaled back. They are the ones who have to tell the devs that they have to work overtime to meet an arbitrary dealine that "Can't" be moved even though they know and the devs know that when the the client has a last minute change (which they will) , the client will hapily move that can't move deadline. When things go down the tubes in a major way, they have to find out why and they have to explain to senior managment what went wrong and what they are doing about it.
Another difficult issue for the PM is hiring and firing. It's hard to get the best staff. It's even harder when you aren't the hiring official and have to negotiate internally to get good people assigned to your projects. It's even more stressful to be the one who has to say, this person isn't performing and has to go. Especially when you know he's the sole income and his wife is pregant and they need the health insurance. But sometimes you have to be the bad guy.