I'm an Object Oriented programmer looking forward to learning a functional language. My questions are:
You don't necessarily have to choose between the two paradigms. You can write software with an OO architecture using many functional concepts. FP and OOP are orthogonal in nature.
Take for example C#. You could say it's mostly OOP, but there are many FP concepts and constructs. If you consider Linq, the most important constructs that permit Linq to exist are functional in nature: lambda expressions.
Another example, F#. You could say it's mostly FP, but there are many OOP concepts and constructs available. You can define classes, abstract classes, interfaces, deal with inheritance. You can even use mutability when it makes your code clearer or when it dramatically increases performance.
Many modern languages are multi-paradigm.
As I'm in the same boat (OOP background, learning FP), I'd suggest you some readings I've really appreciated:
Functional Programming for Everyday .NET Development, by Jeremy Miller. A great article (although poorly formatted) showing many techniques and practical, real-world examples of FP on C#.
Real-World Functional Programming, by Tomas Petricek. A great book that deals mainly with FP concepts, trying to explain what they are, when they should be used. There are many examples in both F# and C#. Also, Petricek's blog is a great source of information.
If you're in a heavily concurrent environment, then pure functional programming is useful. The lack of mutable state makes concurrency almost trivial. See Erlang.
In a multiparadigm language, you may want to model some things functionally if the existence of mutable state is must an implementation detail, and thus FP is a good model for the problem domain. For example, see list comprehensions in Python or std.range in the D programming language. These are inspired by functional programming.
How you define your object oriented character in development? Is there any case that you are already using FP principles without knowing? Can you tell? If you found something that looks like more fp than oop then that's your answer. Look eval, lambdas, closures, pattern matching and many many more. Do you use any of these concepts? Then, that's where fp code fits in.
When do you choose functional programming over object oriented?
When you anticipate a different kind of software evolution:
Object-oriented languages are good when you have a fixed set of operations on things, and as your code evolves, you primarily add new things. This can be accomplished by adding new classes which implement existing methods, and the existing classes are left alone.
Functional languages are good when you have a fixed set of things, and as your code evolves, you primarily add new operations on existing things. This can be accomplished by adding new functions which compute with existing data types, and the existing functions are left alone.
When evolution goes the wrong way, you have problems:
Adding a new operation to an object-oriented program may require editing many class definitions to add a new method.
Adding a new kind of thing to a functional program may require editing many function definitions to add a new case.
This problem has been well known for many years; in 1998, Phil Wadler dubbed it the "expression problem". Although some researchers think that the expression problem can be addressed with such language features as mixins, a widely accepted solution has yet to hit the mainstream.
What are the typical problem definitions where functional programming is a better choice?
Functional languages excel at manipulating symbolic data in tree form. A favorite example is compilers, where source and intermediate languages change seldom (mostly the same things), but compiler writers are always adding new translations and code improvements or optimizations (new operations on things). Compilation and translation more generally are "killer apps" for functional languages.
Object Oriented Programming offers:
Functional Programming, in Haskell or even in Scala, can allow substitution through more general mechanism of type classes. Mutable internal state is either discouraged or forbidden. Encapsulation of internal representation can also be achieved. See Haskell vs OOP for a good comparison.
Norman's assertion that "Adding a new kind of thing to a functional program may require editing many function definitions to add a new case." depends on how well the functional code has employed type classes. If Pattern Matching on a particular Abstract Data Type is spread throughout a codebase, you will indeed suffer from this problem, but it is perhaps a poor design to start with.
EDITED Removed reference to implicit conversions when discussing type classes. In Scala, type classes are encoded with implicit parameters, not conversions, although implicit conversions are another means to acheiving substitution of compatible types.