Ninjam explanation, learning resources, Combining with video

[stooshbatis]

Hi Folks,

Amateur developer and musician here looking to explore possible enhancements to ninjam.

I want to create a live stream that has people playing over ninjam, with live video feeds of each participant synced with the audio, all combined into on live stream feed, like this - https://clips.twitch.tv/AntsyInterestingShrimpPicoMause

The above was done with ninjam via jamtaba, and then seperate rtmp video feeds grouped together with audio into OBS Studio (software to broadcast live streams). The top left cam is not in sync, the bottom right cam is in sync. It is possible to sync the cams this way via delaying certain channels in OBS, but this isn't stable or efficient, e.g. if someone reconnects to the server, the delay changes.

Jamtaba has a video feature, which per the creator, is sending the data through ninjam, and the video is not in sync perfectly and the sync fluctuates. Also video quality is low.

I'm wondering if it's possible to allow ninjam to receive higher quality video feeds and sync them somehow with the audio.

I'm a beginner with c and c++, and am currently tinkering with how to process video and audio as some kind of starting point. I've poked through the ninjam code, but not enough knowledge to get a solid understanding at this point. Would appreciate any learning resource recommendations!

I'm wondering what others thoughts are on this, practical and technical!
Also wondering if anyone could provide an explanation of how the ninjam server processes the audio.