As the other posters noted, these objects need to be explicitly freed. This is usually done manually by the use of a try..finally block, as Ray demonstrated. There are exceptions, though, that you ought to be aware of.
Components (TComponent descendants) pass an Owner parameter to their constructor. If the owner is not nil, then the owner component will take ownership of the new component and free it when it gets freed. This is why you don't have to clean up your own forms; they're connected to the Application object, which knows how to free itself when the program's finished. However, if you create a component at runtime, you need to either assign it an owner or pass nil to the constructor and then free it yourself. Do not mix the two by freeing a component with an owner. That can cause a double-free condition under certain circumstances.
Interfaced objects that implement reference counting (TInterfacedObject descendants, mostly) are freed by the reference counting mechanism if you refer to them specifically as an interface (not as an object.) They get freed automatically when the last interface reference to them is removed. Don't free a TInterfacedObject manually if you've already assigned it to an interface reference. This will raise an exception. Also, be aware that not all objects with interfaces implement reference counting. Mostly just the ones that are descended from TInterfacedObject.
It's not always practical to create an object, use a try..finally block, and then free it. Sometimes that just doesn't work for what you're doing, especially if you're assigning the object to some sort of list (and making a whole lot of them.) In that case, it's a good idea to use a TObjectList (or better still, if you have D2009, a TObjectList) with the OwnsObjects property set to true. This causes the list to become the owner of the objects in it and free them when it gets freed, just like components do. Again, don't free an object manually if it's owned by an object list.
Dynamic arrays (including strings) are managed by the compiler with a reference-counting system, and most other types of variables are allocated on the stack. You never have to worry about manually freeing anything other than objects, unless you're playing around with pointers.
This probably sounds complicated, but you'll get used to it soon enough. Just remember that every object is owned by one of three things: another object, the interface reference-counting system, or your code, and the owner should free all of its objects when they're no longer needed. Nothing should try to free something that's owned by something else. (Thou shalt not steal.) Remember these guidelines and you'll end up with good memory management. You can also set "ReportMemoryLeaksOnShutdown := true" in the main routine in your DPR for a bit more help.