HFS World First: Full cockpit VR Passthrough in MSFS202/2024
In something never done before (that I’ve seen anyway..), you are now able to view your real-world cockpit IN FULL , read maps, write notes etc. while outside the cockpit you see the virtual world. Not only that, you can (if you have created a safe space to do so!) also step out of the cockpit, and walk around the aircraft all the while being able to view the real cockpit interior. The instrument panel is remains virtual, so still no need to have the added complication of real working instruments.
Allow me to elaborate, but if you are in a rush to see what this looks like, by all means check out our video here and feel free to come back and learn more 🙂
I have been playing with the Meta Quest 3 headset using the Passthrough functionality through Virtual Desktop. If it all sounds a little confusing, I don’t blame you but let me explain:
The Quest 3 VR headset has dual cameras that allow you to see the real world in full 3D high resolution colour while virtual windows, games etc. are overlain over this real world. This is the basis of Mixed Reality and Meta have really pulled it off very well with an exceptional headset at a very reasonable price.
The Quest 3 is standalone, i.e. it is fed information via WIFI and you download programs to it and these can then run independently. To get it to run PC games or simulators such as DCS World, MSFS2020 or XPlane you can either use a cable or better yet (to maintain independence from any wiring which must be the way to go), use a programme called Virtual Desktop. You load the streamer portion of that programme on your PC and the main programme runs on your Quest. It uses WIFI to connect and to get the right bandwidth you need at least a router with 5Gbps capability. Your PC must be connected to the router with an ethernet cable.
Virtual Desktop (VD)(thankfully not a disease!) provides the facility to set up passthrough. You elect an unusual colour which VD then interprets as having to replace it with the real world footage. In my case I elected mono-coloured in RGB 255,0,255 which gives a strong pink colour. VD provides the passthrough but unfortunately and seemingly unavoidably there is still a small outline of the area that remains in pink. Anyway, it’s not too intrusive but perhaps that will also be resolved going into the future.
So using a program such as OpenKneeboard or MR Passthrough, you can have a little window into the real world. I thought however, if one can colour the cockpit interior in pink then you can select what you wish to have a passthrough. This was my own first attempt:
The problem with this was that there are still too many shadows, thus preventing the passthrough to work properly. It did prove the theory though, so I turned to Alex Kassabian at FlyingIron Simulations for his help. He did a colour application in the base files of the model, and the result of that you can see in the video.
I still need to play around with the settings to reduce some of the pink remnants and optimise the experience overall, but Build License holders are free to contact me for the base files with instructions on how to use them should they wish to try it out.