As already mentioned by some people: the application pool of the site in IIS will restart (this typically takes a couple of seconds). As a result the next page request(s) will be slower (since nothing will be cached anymore). Also the session state of the users will be lost; BUT in WSS session state is not used by default, in MOSS it is used by InfoPath Form Services. So it could be that you don't have big issues related to losing session state.
On the other side; to overcome those issues: what is typically done is to create a SharePoint Solution (WSP) that deploys and starts a Timer Job to make the changes to the web.config from code (using the SPWebConfigModification class of the Object Model). The nice thing is that you can schedule the execution of the change, so your users won't notice it.