I know relational databases are based on set-theory, functional programming is based on lambda calculus, logic programming is based on logic (of course :)), and now that I think of it; I'm not sure if imperative and generic programming is based on any particular branch of mathematics either.
OOP does not originate from any strict formalism, but it is a formalism indeed. There were a number of attempts to define that formalism properly. Most notable work is done by Luca Cardelli: http://lucacardelli.name/indexPapers.html (see the whole "Objects" section)
Imperative programming could be based on any Turing-equivalent formalism, including lambda calculus, SK logic, Turing abstract machine, Markov algorithms, or any other similar Term Rewriting System (TRS). Generic programming is not any different, it is a term rewriting system of a sort.
So, for the most common mathematical grounds for literally everything you'd need to dig into term rewriting systems.
OOP has its origins in programming languages like Simula-67 and Smalltalk-80, rather than any mathematical theory or formalism. But I suppose that you could say that OOP's object, class and inheritance concepts are based on naive or common-sense systems of categories and classification; e.g. the taxonomies developed by Linnaeus.