Leaving out your support libraries and build tools makes it a huge pain for:
anybody who just wants to try out the software, perhaps on a platform for which you haven't provided an installer, or from a newer version that you've already released
anybody who wants to contribute or hack on your project.
These are the people that you need to pull out all the stops to cater to in a project that won't have a marketing department pimping it out and won't have a full-time paid team developing new features and fixing bugs (aka, personal/open-source/hobby projects).
Nobody is going to play around with or hack on your project if its a huge pain in the ass to even get it to build, and they'll just move on to the next thing. Somebody who, for whatever reason, has an interest in spoofing your software with malicious intentions, is already going to be putting in much more effort than it would take to hunt down a few other packages to put an installer together, so you're deterring the wrong subset of users for no appreciable gain. (Consider it a form of security through obscurity. Which never works.)
Focus on making your repository accessible to users and other developers.
As an aside, people who are downloading and building software should be in the habit of checking the code anyways, or at the very least deciding whether or not they trust the distributor before installing and running stuff locally.