I found this silly hack to make it work. Setting the scrollview to not be a tabstop keeps it from eating the key events.. but then I had another textbox on the page that all of a sudden ALWAYS had focus because the scrollview didn't anymore. So I fixed that by letting an invisible textbox get focus.
scrollView.IsTabStop = false;
invisibleTextBox.Foreground = new SolidColorBrush(Colors.Transparent);
invisibleTextBox.Background = new SolidColorBrush(Colors.Transparent);
Canvas.SetZIndex(invisibleTextBox, -1000);
invisibleTextBox.KeyDown += new KeyEventHandler(HandleKeyDown);
invisibleTextBox.KeyUp += new KeyEventHandler(HandleKeyUp);
Edit: I also had to move the text box off the canvas because despite being invisible, its outline still showed up.
2nd Edit: I used a textbox because that was the first thing I found that could capture KeyDown events. However, a UserControl can. So it would probably be better practice to use a UserControl instead of an invisible text box. You can call Focus() on the UserControl if needed.