The hardest language to learn is one that is in a paradigm that you haven't learnt.
Here are four examples of languages each one from a different paradigm:
Prolog (logic programming) - learning to make the computer do things by evaluating logical statements that occur as Horn clauses.
Haskell (Functional Programming) - learning to make the computer do things by evaluating functions (and functions of functions of functions).
Pascal (Imperative, Structured programming) - e.g. learning to make the computer do things by assembling ordered packages of instructions against data structures
Smalltalk (Object-oriented programming) - learning to make the computer do things by sending messages to objects, making new objects and creating and destroying associations between those objects
Every time you learn a new paradigm, it's going to be very difficult.
Learning within the same paradigm - especially when there's syntactic similarities should be pretty easy:
e.g. moving from C to non-OO perl to Pascal (all imperative), or moving from C++ to Java (all OOP)