If a bug in the module messes up configuration settings or other data in your database, these will remain even after you uninstalled the module. So yes, it could cause issues for other modules or Drupal core, even after removal.
Note that this could happen with 'stable' modules as well, as the decision on when to declare a module as 'ready for production' is more or less up to the maintainer!
This is one of many reasons to never, ever test new modules (stable or not) directly on a production install. Create a test/staging install first, install the new module(s) and test, test, test.
The same is valid for updates, even for Drupal core updates. Always do a test run on a separate install first (at the very least, make sure you have a working, restorable backup of your database and code that you can switch back to, in case anything goes wrong).