COCKOS
CONFEDERATED FORUMS
Cockos : REAPER : NINJAM : Forums
Forum Home : Register : FAQ : Members List : Search :

Go Back   Cockos Incorporated Forums > NINJAM Discussion > NINJAM Developer Discussion

Reply
 
Thread Tools Display Modes
Old 11-16-2018, 03:17 PM   #1
stooshbatis
Human being with feelings
 
Join Date: Nov 2018
Posts: 2
Default Ninjam explanation, learning resources, Combining with video

Hi Folks,

Amateur developer and musician here looking to explore possible enhancements to ninjam.

I want to create a live stream that has people playing over ninjam, with live video feeds of each participant synced with the audio, all combined into on live stream feed, like this - https://clips.twitch.tv/AntsyInterestingShrimpPicoMause

The above was done with ninjam via jamtaba, and then seperate rtmp video feeds grouped together with audio into OBS Studio (software to broadcast live streams). The top left cam is not in sync, the bottom right cam is in sync. It is possible to sync the cams this way via delaying certain channels in OBS, but this isn't stable or efficient, e.g. if someone reconnects to the server, the delay changes.

Jamtaba has a video feature, which per the creator, is sending the data through ninjam, and the video is not in sync perfectly and the sync fluctuates. Also video quality is low.

I'm wondering if it's possible to allow ninjam to receive higher quality video feeds and sync them somehow with the audio.

I'm a beginner with c and c++, and am currently tinkering with how to process video and audio as some kind of starting point. I've poked through the ninjam code, but not enough knowledge to get a solid understanding at this point. Would appreciate any learning resource recommendations!

I'm wondering what others thoughts are on this, practical and technical!
Also wondering if anyone could provide an explanation of how the ninjam server processes the audio.
stooshbatis is offline   Reply With Quote
Old 11-17-2018, 01:43 AM   #2
pljones
Human being with feelings
 
pljones's Avatar
 
Join Date: Aug 2007
Location: London, UK
Posts: 765
Default

Loss of synchronisation can occur at many points.

For a start, the capture of audio and video will usually be separate devices and there can be different processing delays introduced here.

Then there is the encoding process: video and audio use different encoders, which have different amount of CPU time needed and hence will definitely take (markedly) different amounts of time. If either becomes "too slow", you'll get a definite loss of synchronisation happening unless the processes "know" how to maintain any established synchronisation.

Next, transmission to the server will require an amount of time proportional to the amount of data -- which will be larger for video. Once more, the transmission could separate the streams. However, the NINJAM protocol is designed to prevent this if a supported format is used - it has to do this for audio streams already.

Then the above get handled, in reverse order, by each client: receive, decode, present.

You need a system that will first remove any capture delay as much as possible, then ensures that the data is encoded in a way that NINJAM will transport without loss of synchronisation.

The NINJAM client software, of course, adds further delays to any signal it receives to align all remote clients with its time base. However, of itself, this shouldn't cause separation between two signals from the same client: if it did, that would cause issues for audio as well as mixed audio and video (and I'm not aware of that happening).

Of course, if the video signal is "too slow", then it's going to cause problems - it will cause the "slow" peer to fall out sync with the time base by an interval or more to allow it to catch up (or get kicked, possibly) - but that's not sync between audio and video.
__________________
Quote:
Originally Posted by Tony Williams
...Playing fast around the drums is one thing. But to play with people for others, to listen to, that's something else. That's a whole other world.

Last edited by pljones; 11-17-2018 at 01:55 AM.
pljones is offline   Reply With Quote
Old 11-17-2018, 05:18 PM   #3
stooshbatis
Human being with feelings
 
Join Date: Nov 2018
Posts: 2
Default

Thanks for the reply pljones! Appreciate the info!

A lot to learn, but it sounds like it is a complex technical challenge to attempt, if even possible at all.

Would appreciate feedback from other as well!!
Thanks for your time!
stooshbatis is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 02:21 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.