I'm rendering a WPF grid with multiple elements (buttons, textbox, ...) to a bitmap which is then used as a texture for a 3D surface in a Direct3D scene. For user interaction I create a 3D ray from the 2D mouse cursor position into the 3D scene finding the intersection point with the gui surface. So I know where the user has clicked on the WPF grid, but from there I'm stuck:
How can I simulate mouse events on the WPF elements while they are not actually shown in an open window but rendered off-screen?
Recently, I was looking into UIAutomation and RaiseEvent but these are used to send events to individual elements, not the whole visual tree. Traversing the tree manually and looking for elements at the cursor position would be an option but I haven't found a way to do this accurately. VisualTreeHelper.HitTest is a good start but instead of finding TextBox it finds TextBoxView and instead of ListBox it finds a Border.
EDIT: returning HitTestResultBehavior.Continue in the result callback of HitTest lets me walk through all elements at a given point. I can now send mouse events to all these elements but the values of the MouseEventArgs object are those of the real mouse. So I have to create a custom MouseDevice which apparently is impossible.
PointHitTestParameters p = new PointHitTestParameters(new Point(
((Vector2)hit).X * sourceElement.ActualWidth,
(1 - ((Vector2)hit).Y) * sourceElement.ActualHeight));
VisualTreeHelper.HitTest(sourceElement,
new HitTestFilterCallback(delegate(DependencyObject o)
{
return HitTestFilterBehavior.Continue;
}),
new HitTestResultCallback(delegate(HitTestResult r)
{
UIElement el = r.VisualHit as UIElement;
if (el != null)
{
MouseButtonEventArgs e = new MouseButtonEventArgs(Mouse.PrimaryDevice, 0, MouseButton.Left);
if (leftMouseDown) e.RoutedEvent = Mouse.MouseDownEvent;
else e.RoutedEvent = Mouse.MouseUpEvent;
el.RaiseEvent(e);
}
return HitTestResultBehavior.Continue;
}), p);