Yep, lots of things inside the Windows data structures change when a debugger is attached. It changes how memory is allocated/freed, it adds additional housekeeping code and "markers" on the stack (Ever noticed the F00D values in newly allocated memory) in fact many of the changes are used by anti-debuggers to detect if an application is being debugged.
In interpreted languages (Java, .NET) the runtime will often generate different machine instructions when running under a debugger to help it trap and display exceptions, show the original code, etc. It will usually generate unoptimized code as well when a debugger is attached.
Some of these changes affect the way the software behaves and can result complicate transient bugs that are caused by optimizations or extremely fine timinig dependencies.