Tag: opentrack

Posted on
by
Having an action webcam strapped with bow ribbons to my XR glasses grinning mad into the smartphone cam. A bunch of wires are also strapped to the glasses.
Video: How to get 6DOF with older 3DOF XR glasses using Breezy and OpenTrack

Breezy can now turn a 3DOF (degree of freedom) device into a 6DOF device by augmenting the missing positional data from a webcam. Spoiler! It is not the cam strapped to my face – this is just for the demo you can watch here, on PeerTube or YouTube.

The cam, that I used for this task, is sitting on my monitor. How this works? Well not with magic! This requires a somewhat decent webcam – really anything from the last decade should suffice – and OpenTrack, of course.

OpenTrack is a head-tracking application with multiple tracker plugins. One of it’s plugins is the Neuralnet Tracker, an AI powered extension that comes with a bunch of different head pose models to choose from. With a webcam connected this can now locally run the detection model with very low latency – so it’s usually blazing fast on most systems!

This alone is already 6DOF and is used a lot for gaming already – so what does Breezy do with this? Simple! It reads the forwarded data via an UDP listener, a very quick way to transmit data on a local network or system [and complements it’s own rotational data with the missing positional data].

With this a Breezy user still gets the rotational data from the XR’s very sensitive IMU, that is short for Inertial Measurement Unit btw, and the not so important positional data sent from OpenTrack.

This works of course only while the webcam can still see the user. So sadly no walking around while using this.

And the best thing? It can also send the data back! This means that the very same combined values can be forwarded – e.g. to a computer game – benefiting from the best available data sources for rotation and position.

That’s not the main use case, of course, and only of importance for some nerds like myself. This is mostly relevant for the productivity features of Breezy, because sometimes a text may be too small to read with the glasses on. We do no longer have to increase the font size – we can now simply lean in! That is a feature that is usually only available with glasses, that come with little cameras of their own, so they can have native 6DOF support. And when I say native I mean that such glasses usually also outsource exactly this calculation to the connected computer. It’s my understanding that this seems to require a lot of computation power, which is something many XR users with the more modern devices complain about.

Well not so much with OpenTrack and the Neuralnet tracker, that utilizes the ONNX runtime under the hood. That’s a high-performance, cross-platform engine to power exactly such models locally. The runtime automatically makes use of the best available hardware acceleration, if there is any.

Overall I’m rather hyped about this feature – especially because I’m using the OpenTrack output option of Breezy for quite some time now, to get a VR like experience with stereoscopic 3D rendering in Side-By-Side mode. I can now keep using my older XR glasses and still enjoy this more modern 6DOF feature. This is rather expensive hardware after all.

And all that on Linux PC!

Breezy xr_driver: https://github.com/wheaney/breezy-desktop by https://www.youtube.com/@WayneHeaney

Official Announcement XR desktop with 6DoF + multiple displays: https://www.youtube.com/watch?v=eFLmjpjF-rA

Music “Life’s Worth Dying For” CC BY-SA 3.0 “LostDrone”. Licensed to the public under https://creativecommons.org/licenses/by-sa/3.0/ Verify at https://soundcloud.com/lostdrone/rock-lostdrone-lifes-worth-dying-for-free-download-and-creative-commons-license

CC

This content is licensed under a Creative Commons Attribution 4.0 International license.

Posted on
by

This is Project Wingman mission 01 Black Flag played on a Linux PC with Proton Experimental, OpenTrack with the Neuralnet Tracker plugin and my DIY HOTAS / rudder system based on Arduino Pro Micros replacing the original electronics in my Thrustmaster FLCS/Cougar gear:

Pick your poison: https://makertube.net/w/8MyoVSzDfwMuQR6bCqtbie / https://www.youtube.com/watch?v=dq0sihlgW_Y

I got Project Wingman on a sale months ago and I finally gave it a try. As an Ace Combat player I felt right at home. My initial experiment was with the XR glasses and woah that feels good in 3D and all but today I remembered that old Plasma TV in the basement. Got it second hand a year ago for dead cheap. Today I brought it upstairs to try it with the ViperPit and now I’m not sure what’s more awesome.

Well, that is if I feel like burning ~470W on top for that thing but hey this is for very specific gaming sessions only anyway 🤷

Guess I’ll spend more time in the ViperPit again 😀

Posted on
by

Remember my proof concept to read IMU data of my glasses to ?

hodasemi wrote a connector based on the idea that works without : https://github.com/hodasemi/xr_to_opentrack_rs – comes with a systemd service file so it can run in the background.

Once installed the only step left to do is fire up OpenTrack 🤘😄🤘

Posted on
by

So I was asked if my head tracking approach of reading the IMU data from my Viture Pro to OpenTrack and SBS (side-by-side) mode with ReShade would also work with StarCitizen.

Guess it does 🤷

Pick your poison to watch the video: https://www.youtube.com/watch?v=rWUC2Y3TRh4 / https://makertube.net/w/8L7gVN8NnLvjhQCPGNmd6W

I start Star Citizen via Lutris (and not with Steam), which requires slightly different settings once ReShade is installed:

Enable Gamescope: ON
Output Resolution: "3840x1080"
Game Resolution: "3840x2160" (set this also ingame!)
Custom Settings: "--scaler stretch"

Can this get you banned? Who knows 🤷 Jury is still out on this. Do I care? Nope. I won’t miss my puny starter pack.

YMMV.

The proof of concept code to read the IMU data can be found at https://github.com/bekopharm/xr_to_opentrack (pending changes).

It works with the Breezy GNOME xr_driver: https://github.com/wheaney/breezy-desktop (but the Vulkan one works probably too but that’s untested). It should also be compatible with other glasses that have IMU for Breezy available.

There is an unlisted SBS version of this video linked in the description. You will need XR glasses that do FULL SBS though to watch it!

Posted on
by

Until now I used OpenTrack with my DIY IR tracker or the Neuralnet tracker. I knew that my XR glasses feature IMU data though and the xr_driver of the Breezy Desktop project allows to access the data via IPC on Linux PC. So I did what Linux user do: I wrote a script to access the IMU data and forwarded it via UDP to OpenTrack:

Pick your poison to watch the video: https://www.youtube.com/watch?v=njuumLUvqrM / https://makertube.net/w/2bNyxJhdyydTeFq17onikv

This reminded me that I also wrote a proof of concept to implement the FaceTrackNoIR (or OpenTrack) protocol into FreeSpace 2 Open on Linux PC ( https://makertube.net/w/7VtfAjW7EiAUS5aiPwG7if ) so I gave it a spin to test the data bridge. That was smooth sailing!

The mod is Diaspora: Shattered Armistice, still awesome today: http://diaspora.hard-light.net/ (Warning: This may fuel a desire to re-watch the BSG series again 😀).

The bridge code can be found at https://github.com/bekopharm/xr_to_opentrack (pending changes).

It works with the Breezy GNOME xr_driver: https://github.com/wheaney/breezy-desktop (but the Vulkan one works probably too but that’s untested). It should also be compatible with other glasses that have IMU for Breezy available.

Update: hodasemi wrote a Rust connector based on the idea that works without Breezy: https://github.com/hodasemi/xr_to_opentrack_rs – comes with a systemd service file so it can run in the background. Once installed the only step left to do is fire up OpenTrack 🤘

🔥 Hell Yes! added UDP support for native under and I can finally talk about it 🤓 Yes! Yes! 🚀

🎥 I was to happy about it that I even did a recording while I had a bad cold and was hoarse just to show it off: https://makertube.net/w/wo4zAJiTFLeg8t2o93MLpL or https://www.youtube.com/watch?v=rgOkGwMShT0

✍ Took a while but it’s now in the open beta for 7.50 https://steamcommunity.com/games/392160/announcements/detail/4485117301459255318

It’s great to see another company embracing open standards and care for their niche gamers too ❤️

Posted on
by

This one flew under my radar so far (haha, sorry):

Rescue the civilians, race the clock, and raze the enemy in MH-Zombie, the world’s only helicopter arcade simulator! Three flight physics modes, three difficulty modes, and a tutorial mode provide a stepped learning curve and wider accessibility to realistic helicopter flight.

https://store.steampowered.com/app/1429350/MHZombie

The reason this came to my attention is because it’s one of the few games that [just] implemented via UDP e.g. available by (and various others). This is great because it doesn’t force people to jump the hoops of , which is only supported for Windows and officially limited to their proprietary devices. See this in action at https://www.youtube.com/watch?v=jMGFdO7VXiY

Apparently it’s written for mobile games but runs on PC as well – that seems to include Linux PC which even makes this a title! 🤓

I don’t know about you but for 3 bucks I’ll totally get this for the occasional pew pew fun. Game seems to be a labour of love so sharing is highly appreciated.

It has been a while that I tried . With the new Tracker plugin (AI haha) for we get head tracking without annoying IR LEDs or reflecting stripes just by reading the webcam video feed. This is apparently fast enough to try without a dedicated nowadays. And all that on a PC. Took some fiddling but the concept still works. What a time to be alive.

Demo: https://makertube.net/w/groS1wpAhP8XYE75vJwX32

HowTo: https://simpit.dev/systems/opentrack/

Posted on
by

I’m flabbergasted how good this tracker-neuralnet plugin for works. It does the with just a webcam without any clips, reflectors or LED stripes. I kinda expected this to not work really well in a dark room, that I prefer for gaming, but I was wrong. Even with a tiny light in one corner of the room only it kept tracking flawless.

…can even scratch my nose and it keeps tracking.

To get this neuralnet tracker input in the first place I had to download the ONNX runtime package onnxruntime-linux-x64-1.18.1.tgz from https://github.com/microsoft/onnxruntime/releases/tag/v1.18.1 (My Fedora offered 1.15.1 from it’s repo but this was at the time of writing not sufficient and having it installed resulted in failure due to an API mismatch). I didn’t even install it somewhere, just unpacked it in my Downloads folder.

Back in my OpenTracks folder I run cmake the way I’ve done it before several times but this time I added the unpacked onnxruntime folder to the config.

Configure did it’s magic (note how it picked up module tracker-neuralnet once the ONNXRuntime_DIR was set) and here we are one make later. This is rather impressive 🤓

YMMV

CC

This content is licensed under a Creative Commons Attribution 4.0 International license.

TIL (and I know I’m late for the party): protontricks can set the env for Steam in a very comfortable way to run another exe in the same wine prefix/bottle/compatData folder for an already running game. Useful for companion apps of games or e.g. OpenTrack. I used to do this manually with little scripts 🤓

protontricks-launch ~/.steam/steam/steamapps/compatdata/1069190/pfx/drive_c/opentrack/install/opentrack.exe