YUI and prototype are completely separate and have nothing in common except for both being a JS library.
Prototype is what is is, a javascript framework that makes working with the DOM very easy. It depends on scriptaculous for effects. Prototype is the first popular javascript framework since ajax became popular, imho it has been exceeded by jquery, mootools and many more since then.
YUI is a javascript framework too but features a much bigger API and has serveral built in widgets (datepicker, grid, validation,...) which ofcourse makes it a bigger library too.
What i like most about YUI is the fact that the code organised in very clear namespaces.