yes and no. i think manually downloading the bits you need would be more work (though less bandwidth) than just downloading everything.
as for building, you could probably get away with building just skia and pushing the individual .so files. assuming there are no other files it requires.
lithium:~$ ~/Downloads/android-sdk-linux/tools/adb shell ls /system/lib | sort | grep skia
libskiagl.so
libskia.so
lithium:~$
this also assumes nothing's statically linking it (i haven't checked) and that you don't need to change anything in header files that would mean callers need to be rebuilt.
but if you're only changing skia, you'll only have to rebuild the world once anyway. after that, you can just rebuild the bit you've changed (with something like "mmm external/skia").
don't forget to use -j when building to take advantage of all your processors/cores!