Binary is really a misnomer, since everything on your computer is "binary" at some level. Binary, when it comes to file or network stream formats, means not-easily-human-understandable. It also tends to be compact.
Examples of textual or "human readable" (human understandable) file and stream formats:
Examples of "binary" file and stream formats:
The thing of most note here is that human understandable formats need a lot less explanation if you simply crack them open and start reading. Binary file formats might need whole books to explain :)
A format isn't necessarily purely "binary" or purely human understandable, though. For example, you could probably understand a series of single digit numbers with no spaces, which represent an array of single digit numbers. You probably couldn't understand a series of 48 numbers (with no spaces), which represent x, y, and z values for 16 3d vertices, even though you can "read" them. Also, there is Skeet's example of encoded "binary" data, especially if it is embedded in a more human understandable format.