Well, first of all there's more to life than just RPM and YUM. An SRPM would be (somewhat) useless to Debian, for instance.
As for why you'd use a package repository over building everything yourself, well I don't know about you, but I've much rather just run (I'm using Ubuntu so I have apt-get instead of yum):
# apt-get install firefox
Than trying to figure out all the dependencies, as well as all the dependencies dependencies, make sure I have the correct versions of everything, download/build/install any that I don't have (or are out of date: if updating existing dependencies, make sure the newer versions don't break any existing software that I have and make sure I don't end up with 15 different versions of the same thing), and only after all that then download/configure/build/install firefox.
Then realise I'll also want Open Office or MySQL and start all over again!
That said, there are some packages that I install the latest version of from source. For example, I run my media centre off MythTV and I always like to build the latest version of that from Subversion. But even then, with a package manager, that's as easy as:
# apt-get build-dep mythtv
> cd ~/src/mythtv/
> svn co <svn repo of mythtv>
> configure && (etc)
That is, the package management software already knows all the dependencies for MythTV and it can download and install them automatically. Why spend hours tracking it all down manually?
In the end, it sounds to me like maybe you'd prefer a distro like Gentoo... that's the benefit of Linux, of course. If you don't like how things are run in the Fedora/RedHat distribution, you can just choose a different one.