There are some small misconceptions here.
I'll go through the problems in order of occurrence, not severity:
1) Off-by-one error in the calculation of spectrogramWindow (image)
The first array entry ought to be the component of 0Hz, the next is N Hz. The final element of the array should be the component of -N Hz. However, you've calculated 0Hz.
I'm not sure the matlab syntax, but if you flip the image as you have, and then strip the top and bottom lines before appending it to the original, you should be set.
Alternatively, you could consider NOT appending the image to itself, and after extracting spectrogramWindow from image, applying some function to make it Hermitian symmetric.
2) Taking the abs of the IFT. No need. Don't do that.
What you get out of the iFFT, if the iFFT gets the right input, is completely real.
You're seeing complex values out because the input isn't ACTUALLY Hermitian symmetric, as described above. Never use Abs(). If you must cheat, extract the Real part, which won't fold in garbage from the imaginary component.
3) You're throwing away the second half of the signal.
Once you get output from the iFFT, that represents the signal you've asked for. Don't think of it in terms of frequencies, it's now an audio time-series. Keep the whole thing.
Here's how I see it going:
spectrogramWindow = image(:, i);
spectrogramWindow = [spectrogramWindow;reverse(spectrogramWindow(skip first and last))]
signalWindow = ifft(spectrogramWindow);
signal = [signal; signalWindow];