Jon is (as usual) spot on. It is not as easy as you think!
The spec is vague and slightly contradictory. In this particular case it is probably best to squint a little bit and get the gist of what the spec means to convey rather than narrowly parsing it for precise definitions.
The simple fact of the matter is that "inheritance" is a very overused term in object-oriented programming. (I seem to recall that C++ has six different kinds of inheritance, though I'd be hard pressed to name them all on short notice.)
If I had my druthers then the C# specification would clearly call out a difference between inheritance, which is a code-sharing technique for class (and delegate) and struct (and enum) types, and interface implementation which is a requirement that an implementing type have a certain set of public members. Those two things seem conceptually very different to me; one is about sharing implementation details and the other is about requiring public members.
However, the spec does not do so; it conflates the two under the rubric of inheritance. Given that these two somewhat different things have the same name in the spec, it is hard to reason clearly and precisely about the differences between them.
I personally prefer to think that object is not the "base type" of any interface, and that the members of object are not inherited by the interface. That you can call them on an instance of an interface is more like a courtesy extended to you by the compiler so that you do not have to insert a cast to object in there.