This is called data or information hiding.
Basically, you don't want a user (read: other programmer, or yourself) poking in the internals of your class, because that makes it very hard to change things.
On the other hand, a clean separation between interface and implementation (theoretically) makes it easy to change something internally, without affecting any users of your class.
For example, suppose I have a Button control with a public String text field. Now everybody starts using my Button, but I realize that the button should actually be repainted on the screen whenever the text changes. I'm out of luck, because my object can't detect when text is assigned. Had I made it private and provided a setText() instead, I could just have added a repaint call to that setter method.
As another example, suppose I have some class that opens a file in its constructor and assigns it to a public FileStream file. The constructor throws an exception if the file could not be opened. The other methods in the class can therefore assume that this field is always valid. However, if somebody goes poking around in my class and sets file to null, all methods in my class will suddenly start crashing.