At it's heart, programming is the art and science of solving problems within a set of constraints. I believe most people recognize that multiple modes of thinking exists for solving problems, object-orientation being one such paradigm. In practice, a single problem/program may require multiple modes of problems solving, and may benefit from using multiple approaches to modeling different aspects of the problem. In answering this question, I also think it's important to differentiate a style of problem solving, from a particular kind of language. One can certainly program in a object oriented or functional style in a procedural language like C.
The "object-oriented" paradigm is but one approach of decomposing a problem into a set of smaller problems that are easier to reason about and whose correctness we wish to ensure. OO is characterized by the identification of discrete actors (objects) with well-defined responsibilities, which interact by passing messages (themselves often consisting of objects) between one another. OO has proven effective at addressing problems "in the large" - such designing solutions for large scale systems whose responsibilities cover a large domain.
Other modes of thinking about problems certainly exist - and offer different vantage points from which to examine how to solve them. For instance, some problems can be viewed as a transformation of information from one representation to another. Functional programming paradigms can be an effective way of tackling such problems - essentially the problem is decomposed into a sequence of transformations from the source representation to the output representation. Functional programming, by avoiding state an mutable data lends itself better for problems where we can exploit parallelism and sub-computation caching. Not that such programs cannot be created in other styles (they certainly can) - it's just that the constructs and idioms provided by functional languages often are easier to compose with one another for such problems.
We should also be cognizant of the limitations and overhead that are introduced by certain programming styles. The approach of treating a problem as a set of interacting "objects" (whether they are materialized using language constructs or merely conceptualized in the mind of the programmer) requires establishing boundaries and constraints on how a program behaves. Such constraints may actually prove to be a barrier to effective construction of certain solutions. Take device drivers for instance. A common expectation of device drivers are that they need to be efficient in their operation. OO design does not necessarily improve efficiency of operations - in fact, OO tends to introduce additional layers of indirection and abstraction primarily to aid in maintinability and extensibility - neither of which is necessarily as important in the creation of device drivers. Similarly, programs designed in a functional style often require more memory or steps to process information due to the immutable nature of how such programs represent data.
So, to summarize my thoughts above, I would say that:
- Different kind of problems lend themselves to different kinds of problem solving approaches (data transformation vs. request/response vs. process-oriented).
- Some problems composed of multiple subproblems may benefit from multiple modalities of problem solving techniques (OO, functional, procedural, etc).
- The right tools make tackling a problem easier, while the wrong tools can make a problem much, much harder - if not intractable.
- Recognizing which problems lend themselves to which techniques requires experience, open-mindedness, and a willingness to think creatively.
- Different languages lend themselves to certain style of programming, with varying degrees of flexibility and clarity.
Finally, let's address the core of your question: where would an object oriented approach to a problem be cumbersome or otherwise suboptimal?
For specific examples, I would suggest that:
- Extremely small or trivial programs may gain less from an OO approach than the overhead imposed.
- Programing problems that demand native interaction with hardware or extreme performance requirements (eg device drivers) may be suboptimal in OO models.
- Problems which are better modeled as transformations or pipelined operations on a sequence of inputs may be better expressed in functional forms.
- Problems where insufficient time or requirements are present to clearly identify the underlying actor/interaction scheme may actually incurr a penalty from inappropriate OO design.