I know of class-based and protype based object oriented programming languages, are there any other alternatives? What are they?
mixins allow you to extend a class with code that is defined elsewhere, such as in a module.
See Ruby Mixin Tutorial for an introduction.
You may want to check Wikipedia's article on programming paradigms. The one I've worked with is aspect-oriented programming, which is where the mixins come in.
Go has a concept that is similar to classes, but without inheritance and with very flexible interfaces. You can read more about it in Effective Go.
These are indeed the two main approaches behind object-oriented languages, and I'm not aware of another completely different underlying principle.
But there exists a lot of variants of both approaches, as well as a lot of other programming language constructs that tackles reuse/extensibility in either class-based or prototype-based language. Examples: traits, mixin, extension methods, partial class, generics, first-class slots, split objects, etc. A lot of such constructions are first proposed in research papers (ECOOP, OOPSLA, POPL conferences), and a few of them become mainstream in popular languages. But I would qualify them as variations and not as groundbreaking new underlying principle.
Note though than you can mimic object-oriented programming in languages which are not object-oriented per-se. For instance, with Clojure multi-method. Object-oriented and functional programming are also slowly merging, for instance in Scala.
EDIT
It's actually hard to make a list of classic/seminal papers, and I don't pretend to have sufficient knowledge to do so. If there is one somewhere, I would be very interested to see it :) Still, here are a few ones that might interest you.
Inheritance, delegation, subtyping:
- Genericity vs inheritance
- Inheritance Is Not Subtyping
- On the notion of inheritance
- Split objects: a disciplined use of delegation within objects
Module, composition, adaptation