Mixed reality, green screen?

I've watched a few videon about mixed reality. The idea is to use virtual reality to display the outside world and everything above the dashboard (wings, struts, propeller etc.) but cut out a "window" in the VR to display an overlay of the real dashboard of the cockpit that is recorded with a camera. I think this is quite clever as it provides really good 3D immersion but still gives haptic feedback from real controls, switches and buttons.
But defining the window cutout is not easy. Often there is a gap between the two worlds.

Why not use a greenscreen effect and do it the opposite way: Build the cockpit in the real world but paint the windows green (or any color that isn't used on the dashboard like pink). The software could then replace the special color with the view of the outside world.

Ok, maybe it would be difficult to make night simulations. If the cockpit is completely dark the special color of the windows would look black and is not recognised.

Here you're can see un perfect passtrough.... !!!


it's crazy !! with just VD !

They had the genius idea of modifying the interior textures and colors of the plane in FS to give the transparent color!!!

So he only puts in pink what he wants to make transparent, quite simply brilliant, and Vd does the rest!

I remain admiring of the ingenuity of the system, however you have to know how to modify the texture packs under FS....

This is a great idea. I researched some things to see if it is feasible. My conclusion is that it is not easy.

If you search for “SteamVR passthrough” it sounds like it has worked in the past but that would apply to the Home Screen. Maybe you need to have a web cam attached to get the option.

In principle you would need to put the pass through in at the rendering level aka below the game display because modifying games isn’t practical. That means that the pass through layer is on top of the game graphics.

It’s possible that the windows mixed reality API had a method but “Windows Mixed Reality Headsets No Longer Function With Latest Windows 11 Update. Microsoft has removed Windows Mixed Reality from Windows 11”

I think that what was done with Motion Compensation is fantastic. It was done by someone who understood enough about how SteamVR works to read a sensor (aka Quest controller) and inject that into the camera transform for the graphics. That’s at a relatively higher level software call aka SteamVR API. We are extremely lucky to have something that works.

So unless there is some rendering guru who does VR sim racing decides to code it, I don’t see much hope.