Tag: Breezy

Posted on
by
Having an action webcam strapped with bow ribbons to my XR glasses grinning mad into the smartphone cam. A bunch of wires are also strapped to the glasses.
Video: How to get 6DOF with older 3DOF XR glasses using Breezy and OpenTrack

Breezy can now turn a 3DOF (degree of freedom) device into a 6DOF device by augmenting the missing positional data from a webcam. Spoiler! It is not the cam strapped to my face – this is just for the demo you can watch here, on PeerTube or YouTube.

The cam, that I used for this task, is sitting on my monitor. How this works? Well not with magic! This requires a somewhat decent webcam – really anything from the last decade should suffice – and OpenTrack, of course.

OpenTrack is a head-tracking application with multiple tracker plugins. One of it’s plugins is the Neuralnet Tracker, an AI powered extension that comes with a bunch of different head pose models to choose from. With a webcam connected this can now locally run the detection model with very low latency – so it’s usually blazing fast on most systems!

This alone is already 6DOF and is used a lot for gaming already – so what does Breezy do with this? Simple! It reads the forwarded data via an UDP listener, a very quick way to transmit data on a local network or system [and complements it’s own rotational data with the missing positional data].

With this a Breezy user still gets the rotational data from the XR’s very sensitive IMU, that is short for Inertial Measurement Unit btw, and the not so important positional data sent from OpenTrack.

This works of course only while the webcam can still see the user. So sadly no walking around while using this.

And the best thing? It can also send the data back! This means that the very same combined values can be forwarded – e.g. to a computer game – benefiting from the best available data sources for rotation and position.

That’s not the main use case, of course, and only of importance for some nerds like myself. This is mostly relevant for the productivity features of Breezy, because sometimes a text may be too small to read with the glasses on. We do no longer have to increase the font size – we can now simply lean in! That is a feature that is usually only available with glasses, that come with little cameras of their own, so they can have native 6DOF support. And when I say native I mean that such glasses usually also outsource exactly this calculation to the connected computer. It’s my understanding that this seems to require a lot of computation power, which is something many XR users with the more modern devices complain about.

Well not so much with OpenTrack and the Neuralnet tracker, that utilizes the ONNX runtime under the hood. That’s a high-performance, cross-platform engine to power exactly such models locally. The runtime automatically makes use of the best available hardware acceleration, if there is any.

Overall I’m rather hyped about this feature – especially because I’m using the OpenTrack output option of Breezy for quite some time now, to get a VR like experience with stereoscopic 3D rendering in Side-By-Side mode. I can now keep using my older XR glasses and still enjoy this more modern 6DOF feature. This is rather expensive hardware after all.

And all that on Linux PC!

Breezy xr_driver: https://github.com/wheaney/breezy-desktop by https://www.youtube.com/@WayneHeaney

Official Announcement XR desktop with 6DoF + multiple displays: https://www.youtube.com/watch?v=eFLmjpjF-rA

Music “Life’s Worth Dying For” CC BY-SA 3.0 “LostDrone”. Licensed to the public under https://creativecommons.org/licenses/by-sa/3.0/ Verify at https://soundcloud.com/lostdrone/rock-lostdrone-lifes-worth-dying-for-free-download-and-creative-commons-license

CC

This content is licensed under a Creative Commons Attribution 4.0 International license.

Posted on
by

Remember my proof concept to read IMU data of my glasses to ?

hodasemi wrote a connector based on the idea that works without : https://github.com/hodasemi/xr_to_opentrack_rs – comes with a systemd service file so it can run in the background.

Once installed the only step left to do is fire up OpenTrack 🤘😄🤘

Posted on
by

So I was asked if my head tracking approach of reading the IMU data from my Viture Pro to OpenTrack and SBS (side-by-side) mode with ReShade would also work with StarCitizen.

Guess it does 🤷

Pick your poison to watch the video: https://www.youtube.com/watch?v=rWUC2Y3TRh4 / https://makertube.net/w/8L7gVN8NnLvjhQCPGNmd6W

I start Star Citizen via Lutris (and not with Steam), which requires slightly different settings once ReShade is installed:

Enable Gamescope: ON
Output Resolution: "3840x1080"
Game Resolution: "3840x2160" (set this also ingame!)
Custom Settings: "--scaler stretch"

Can this get you banned? Who knows 🤷 Jury is still out on this. Do I care? Nope. I won’t miss my puny starter pack.

YMMV.

The proof of concept code to read the IMU data can be found at https://github.com/bekopharm/xr_to_opentrack (pending changes).

It works with the Breezy GNOME xr_driver: https://github.com/wheaney/breezy-desktop (but the Vulkan one works probably too but that’s untested). It should also be compatible with other glasses that have IMU for Breezy available.

There is an unlisted SBS version of this video linked in the description. You will need XR glasses that do FULL SBS though to watch it!

Posted on
by

Until now I used OpenTrack with my DIY IR tracker or the Neuralnet tracker. I knew that my XR glasses feature IMU data though and the xr_driver of the Breezy Desktop project allows to access the data via IPC on Linux PC. So I did what Linux user do: I wrote a script to access the IMU data and forwarded it via UDP to OpenTrack:

Pick your poison to watch the video: https://www.youtube.com/watch?v=njuumLUvqrM / https://makertube.net/w/2bNyxJhdyydTeFq17onikv

This reminded me that I also wrote a proof of concept to implement the FaceTrackNoIR (or OpenTrack) protocol into FreeSpace 2 Open on Linux PC ( https://makertube.net/w/7VtfAjW7EiAUS5aiPwG7if ) so I gave it a spin to test the data bridge. That was smooth sailing!

The mod is Diaspora: Shattered Armistice, still awesome today: http://diaspora.hard-light.net/ (Warning: This may fuel a desire to re-watch the BSG series again 😀).

The bridge code can be found at https://github.com/bekopharm/xr_to_opentrack (pending changes).

It works with the Breezy GNOME xr_driver: https://github.com/wheaney/breezy-desktop (but the Vulkan one works probably too but that’s untested). It should also be compatible with other glasses that have IMU for Breezy available.

Update: hodasemi wrote a Rust connector based on the idea that works without Breezy: https://github.com/hodasemi/xr_to_opentrack_rs – comes with a systemd service file so it can run in the background. Once installed the only step left to do is fire up OpenTrack 🤘

Posted on
by

So… this is news to me, because I don’t have a headset, but I can set my Pro glasses into (side by side) mode by pressing the small button longer. Some games, like , can do this as well without fiddling around with Reshade. I didn’t really expect it but it just works. This way I even get 3D on foot, which is not supported for VR in Elite Dangerous Odyssey at all! Side by side Crosseye mode (right eye left, left eye right) though? Add some head tracking to the mix, which is totally possible, and I get a very nice VR-like experience even on foot in Elite Dangerous – and on Linux PC!

This is the SBS version that does REQUIRE VR/XR glasses and mebbe something like xr-video-player: https://www.youtube.com/watch?v=KEtRijojBx8

This is the MONO version that does NOT require VR/XR glasses: https://www.youtube.com/watch?v=bYPTk1vygM4

The FOV is somewhat cramped. No idea if this can be tweaked any further but I’ll fiddle with the settings on my next test. Mebbe this can be tweaked (or I use Desktop to zoom in somewhat).

Update: I got the aspect ratio somewhat under control. It’s not perfect but much better and an odd combination of window mode and resolution and upscaling, that somehow affects the HUD only but make no sense to me at all. At this point I think it’s simply a bug of Elite. It’s like the HUD doesn’t get the memo to scale up after the intro played. I’m also not sure if this is a side effect of gamescope but I can totally live with this result (though it does start to stutter somewhat on foot but recording this at the same time somewhat overwhelms my rig anyway). These are my gamescope settings with Steam:

SDL_VIDEODRIVER=x11 obs-gamecapture gamescope -h 1080 -w 3840 -H 1080 -W 3840 --max-scale 2 -f -e -- %command%

Max-scale is probably not needed but it was started with this so I won’t omit it now. The ingame settings are 1280×960 and windowed with _NO BORDER_. Every other mode broke the aspect ratio even more! That is also true for 1920×1024, which would have made _some_ sense to me at least, but this did also NOT work FOR ME. This results in a pixelated HUD which is worked around with upscaler INTERNAL or AMD CAS cranked to x2 – AMD FSR did NOT WORK. If someone could enlighten me on this: Bring it! YMMV.

Update2: Haha it works! https://www.reddit.com/r/EliteDangerous/comments/2o5j30/using_google_cardboard_or_equivalent_kit_as_a_vr/ had a lead: It suggests to double the vertical resolution to get a proper aspect ratio with SBS and shrink the resulting window again.

That’s easy with gamescope:

> gamescope -h 2160 -w 3840 -H 1080 -W 3840 –scaler stretch

e voila, perfect aspect ratio. Wonder if my GPU manages to keep this up though. May have to throw FSR into the mix.