Old 03-30-2021, 07:42 PM   #1
profquad
Human being with feelings
 
Join Date: Feb 2017
Posts: 185
Default Multiple channel audio to parameter hack

Hey, I recently figured out a way to get different tracks sending audio triggers to different video FX parameters.
It's definitely hacky and may cause problems, but it worked for me! Here's the finished video (youtube effed up the quality but oh well.) https://youtu.be/KmYIsXtoJ20

In this, I have the voice tied to the "rejection" on chromakey and the drum track tied to "sensitivity" on edge-detection.

I did the drums first, so let's start with that.

1.) put video peeker on the track at the point you want audio data from in the FX chain.

2.) add the tasty bits from Decorative Oscilloscope to the video FX you want manipulated i.e.
Code:
//@gmem=jsfx_to_video
right up top, and

Code:
bufplaypos = gmem[0];
bufwritecursor = gmem[1];
bufsrate = gmem[2];
bufstart = gmem[3];
bufend = gmem[4];
nch = gmem[5];
gain = 10^(gain_db*(1/20));

dt=max(bufplaypos - project_time,0);
dt*bufsrate < dotcount ? underrun_cnt+=1;
rdpos = bufwritecursor - ((dt*bufsrate - dotcount)|0)*nch;
rdpos < bufstart ? rdpos += bufend-bufstart;
after the params.

3.) Replace undeclared variables. In this case, I replaced dotcount with 1200, because it's not included in edge detection.

4.) Make a new variable to taste. In this case I used
Code:
gsens=abs(gmem[rdpos])*sens;
so the original parameter "sens" can act as an adjustable baseline value.

5.) Find/replace all instances of the old variable "sens" with the audio-sensitive one "gsens". Do this manually, just in case you replace something sensible.

NB gmem[rdpos] is the value sent by video peeker signifying the amplitude or something like that. rdpos bounces between 0 and 176400 to give us a nice array of gmem[] samples. I think. But this is important for getting more than one video peeker working.
Apparently there's a few million gmem[] spots, I can't remember where I read that or what the number actually is. 3 million? And I honestly don't know what else it's used for. In any case, there is an upper limit to how many times you can do this:

For the vocal effects, I put in another video peeker I creatively saved as "video peeker2"
It looks like this. For those following along at home, notice the difference to the original on your own screen:
Code:
desc:video sample peeker2
// copies samples to shared global memory for use by some video processor presets

slider1:lookahead_seconds=1<0,1.8,0.1>lookahead (seconds, 1.0 is normal)
options:gmem=jsfx_to_video

in_pin:left
in_pin:right
out_pin:none

@init
send_nch=2;
pos=bufstart=16;
bufend=bufstart+((2.0 /*seconds*/ *srate*send_nch)|0);

@slider
pdc_delay=(lookahead_seconds*srate)|0;

@block
gmem[6]=play_position;
gmem[7]=pos+176401;
gmem[8]=srate;
gmem[9]=bufstart;
gmem[10]=bufend;
gmem[11]=send_nch; // stereo
  
@sample
gmem[pos+176401]  =spl0;
gmem[pos+176402]=spl1;
(pos+=send_nch) >= bufend ? pos=bufstart;
ta-da!

Now I do a similar treatment for rejection aka "rej" on RGB Chroma-Key with one important distinction. Can you spot it?
Code:
bufplaypos = gmem[6];
bufwritecursor = gmem[7];
bufsrate = gmem[8];
bufstart = gmem[9]+176401;
bufend = gmem[10]+176401;
nch = gmem[11];
gain = 10^(gain_db*(1/20));

dt=max(bufplaypos - project_time,0);
dt*bufsrate < 120 ? underrun_cnt+=1;
rdpos = bufwritecursor - ((dt*bufsrate - 120)|0)*nch;
rdpos < bufstart ? rdpos += bufend-bufstart;

grej=min(rej,abs(6.6*gmem[rdpos])*rej);
Again, that last bit is just a tweaked variable for my own self.
If memory serves about the gmem potential positions, you can do this 3,000,000/176401=about 17 times before your computer crashes.

Please clean this up if you can! It might use up your RAM, so you'd might have to reboot Reaper if you run this for too long. I'm not a doctor. All I can say is that this works.
profquad is offline   Reply With Quote
Old 03-31-2021, 02:27 AM   #2
jrk
Human being with feelings
 
Join Date: Aug 2015
Posts: 2,969
Default

Or, if you write a little jsfx that automates sliders by audio routed to it (on as many channels as you like) - you can use parameter modulation (from the jsfx slider to the Video Processor param).
__________________
it's meant to sound like that...
jrk is offline   Reply With Quote
Old 03-31-2021, 08:38 AM   #3
profquad
Human being with feelings
 
Join Date: Feb 2017
Posts: 185
Default

How do you do that?
profquad is offline   Reply With Quote
Old 03-31-2021, 09:47 AM   #4
ashcat_lt
Human being with feelings
 
Join Date: Dec 2012
Posts: 7,272
Default

I’ve had real problems with trying to use the PM/Link method. I could sort of get a preview that way, but it flickered and fucked up when I rendered it. I ended up having to use PM to make a slider follow the audio, then write that to an automation item, then move that to the envelope for the video processor parameter that I actually wanted to automate.

Video peeker is kind of already set up to do multichannel, and only needed a small tweak to actually do it. Here’s a thread where I posted a 64 channel version along with a version of the oscilloscope which can access any of them. It’s what prompted Justin to release that bug fix of the spectrum analyzer preset a little while back.

https://forum.cockos.com/showthread.php?t=247691

Note that this, like the original, passes samples in an “interleaved” manner. The basic stereo version is laid out like: L0, R0, L1, R1... and basically there’s a pair of those for every sample in the buffer block. BTW, that is a full block of samples as determined by your audio interface settings. This array will be as long as your latency in samples times the number of channels.

I hope this helps. LMK if you have questions working through the code or need help getting it to work.
ashcat_lt is offline   Reply With Quote
Old 03-31-2021, 12:24 PM   #5
profquad
Human being with feelings
 
Join Date: Feb 2017
Posts: 185
Default

I actually used your code there on my search for solutions, it's very cool, but slightly different from what this does.
If I understand it correctly, that allows for 64 separate ins, but still only has a stereo out?
What my gibberish does is take separate ins from separate tracks, keep them separate, and send them to different parameters.
I was looking for a simple way to send audio data around wherever I want, as alluded to above, but I don't understand how to write/find that script.

EDIT:I probably missed something in the adapted oscilloscope section tho..
profquad is offline   Reply With Quote
Old 03-31-2021, 01:06 PM   #6
ashcat_lt
Human being with feelings
 
Join Date: Dec 2012
Posts: 7,272
Default

Yes my version of the Oscope allows you to pick which of the 64 channels (or 32 pairs) you want to use via one of the parameters.
ashcat_lt is offline   Reply With Quote
Old 03-31-2021, 03:40 PM   #7
jrk
Human being with feelings
 
Join Date: Aug 2015
Posts: 2,969
Default

Quote:
Originally Posted by ashcat_lt View Post
I’ve had real problems with trying to use the PM/Link method. I could sort of get a preview that way, but it flickered and fucked up when I rendered it.
Is this a known bug then?
__________________
it's meant to sound like that...
jrk is offline   Reply With Quote
Old 03-31-2021, 03:56 PM   #8
ashcat_lt
Human being with feelings
 
Join Date: Dec 2012
Posts: 7,272
Default

I guess I’ve never seen anybody else mention it, and I never really reported it. Honestly the thing where it came up for me was pretty abusive (not to mention NSFW) all around, but it really did seem like it was the Parameter Modulation, and specifically that where I was using Link to connect to a slider which itself was PMd to follow audio that was causing the flickering issue. Anyway, I found a way around it and didn’t bother posting a bug report, but I suppose I might should.
ashcat_lt is offline   Reply With Quote
Old 03-31-2021, 04:25 PM   #9
jrk
Human being with feelings
 
Join Date: Aug 2015
Posts: 2,969
Default

I can reproduce the behaviour - a smooth parameter mod / slider link (fine in "preview") ends up flickering when rendered.
__________________
it's meant to sound like that...

Last edited by jrk; 03-31-2021 at 04:31 PM.
jrk is offline   Reply With Quote
Old 03-31-2021, 04:45 PM   #10
ashcat_lt
Human being with feelings
 
Join Date: Dec 2012
Posts: 7,272
Default

That’s actually part of why I ended up looking further into this whole video peeker thing, though it was kind of too late for that specific project.

I have around here somewhere a preset something like the opacity/zoom/pan preset but with audio control. I’m not sure if I ever updated it to be multichannel aware. It is NOT smoothed on the video end, though, so that if you send it regular audio, it just kind of jumps around randomly. It works best if you use some plugin on the audio side which extracts a relatively smooth envelope and puts that out as “control voltage” on audio channels which you then jam into the video peeker.

This will actually be true of most things you try to modulate via audio this way. The Oscope kind of obviously doesn’t want smoothing, and the spectrogram does the whole FFT thing which is smooth by nature, but if you’re trying to do other things, well...audio signals oscillate wildly on a sample-by-sample basis and if you just grab a single sample every 1000 or so, it’s going to look just random. For most things, you need to smooth on one side of the other.
ashcat_lt is offline   Reply With Quote
Old 03-31-2021, 05:58 PM   #11
jrk
Human being with feelings
 
Join Date: Aug 2015
Posts: 2,969
Default

It's not the smoothed / impulsive nature of the control that does it. I can reproduce it with a 1Hz sawtooth. Looks fine until rendered.

__________________
it's meant to sound like that...
jrk is offline   Reply With Quote
Old 03-31-2021, 06:28 PM   #12
ashcat_lt
Human being with feelings
 
Join Date: Dec 2012
Posts: 7,272
Default

Yeah sorry. My most recent post where I was talking about smoothing was kind of a separate thought. Are you making a bug report? I will +1 because I’ve seen it happen.
ashcat_lt is offline   Reply With Quote
Old 03-31-2021, 06:33 PM   #13
jrk
Human being with feelings
 
Join Date: Aug 2015
Posts: 2,969
Default

Someone should, I'll write it up.
__________________
it's meant to sound like that...
jrk is offline   Reply With Quote
Old 03-31-2021, 07:35 PM   #14
ashcat_lt
Human being with feelings
 
Join Date: Dec 2012
Posts: 7,272
Default

I guess I kind of feel like we’re doing something we’re not supposed to and that’s why it doesn’t work and there’s maybe not any good solution for the devs to fix it. Hopefully I’m wrong, and it’s worth a shot, but... :/

PS BTW sort of on a new tangent - if one happened to have JS plugin with a slider that went from 0 to 1, and literally just its own output to that value (spl0 = slider1; ), one could use PM on that, then jam the output through the video peeker, and...there’s still kind of not any presets which would do anything with it except for ones I’ve written.
ashcat_lt is offline   Reply With Quote
Old 04-02-2021, 01:43 AM   #15
jrk
Human being with feelings
 
Join Date: Aug 2015
Posts: 2,969
Default

I've posted a bug report, If you want to +1 it, please do.
https://forum.cockos.com/showthread.php?t=251679
__________________
it's meant to sound like that...
jrk is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 04:18 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.