03-30-2021, 07:42 PM | #1 |
Human being with feelings
Join Date: Feb 2017
Posts: 185
|
Multiple channel audio to parameter hack
Hey, I recently figured out a way to get different tracks sending audio triggers to different video FX parameters.
It's definitely hacky and may cause problems, but it worked for me! Here's the finished video (youtube effed up the quality but oh well.) https://youtu.be/KmYIsXtoJ20 In this, I have the voice tied to the "rejection" on chromakey and the drum track tied to "sensitivity" on edge-detection. I did the drums first, so let's start with that. 1.) put video peeker on the track at the point you want audio data from in the FX chain. 2.) add the tasty bits from Decorative Oscilloscope to the video FX you want manipulated i.e. Code:
//@gmem=jsfx_to_video Code:
bufplaypos = gmem[0]; bufwritecursor = gmem[1]; bufsrate = gmem[2]; bufstart = gmem[3]; bufend = gmem[4]; nch = gmem[5]; gain = 10^(gain_db*(1/20)); dt=max(bufplaypos - project_time,0); dt*bufsrate < dotcount ? underrun_cnt+=1; rdpos = bufwritecursor - ((dt*bufsrate - dotcount)|0)*nch; rdpos < bufstart ? rdpos += bufend-bufstart; 3.) Replace undeclared variables. In this case, I replaced dotcount with 1200, because it's not included in edge detection. 4.) Make a new variable to taste. In this case I used Code:
gsens=abs(gmem[rdpos])*sens; 5.) Find/replace all instances of the old variable "sens" with the audio-sensitive one "gsens". Do this manually, just in case you replace something sensible. NB gmem[rdpos] is the value sent by video peeker signifying the amplitude or something like that. rdpos bounces between 0 and 176400 to give us a nice array of gmem[] samples. I think. But this is important for getting more than one video peeker working. Apparently there's a few million gmem[] spots, I can't remember where I read that or what the number actually is. 3 million? And I honestly don't know what else it's used for. In any case, there is an upper limit to how many times you can do this: For the vocal effects, I put in another video peeker I creatively saved as "video peeker2" It looks like this. For those following along at home, notice the difference to the original on your own screen: Code:
desc:video sample peeker2 // copies samples to shared global memory for use by some video processor presets slider1:lookahead_seconds=1<0,1.8,0.1>lookahead (seconds, 1.0 is normal) options:gmem=jsfx_to_video in_pin:left in_pin:right out_pin:none @init send_nch=2; pos=bufstart=16; bufend=bufstart+((2.0 /*seconds*/ *srate*send_nch)|0); @slider pdc_delay=(lookahead_seconds*srate)|0; @block gmem[6]=play_position; gmem[7]=pos+176401; gmem[8]=srate; gmem[9]=bufstart; gmem[10]=bufend; gmem[11]=send_nch; // stereo @sample gmem[pos+176401] =spl0; gmem[pos+176402]=spl1; (pos+=send_nch) >= bufend ? pos=bufstart; Now I do a similar treatment for rejection aka "rej" on RGB Chroma-Key with one important distinction. Can you spot it? Code:
bufplaypos = gmem[6]; bufwritecursor = gmem[7]; bufsrate = gmem[8]; bufstart = gmem[9]+176401; bufend = gmem[10]+176401; nch = gmem[11]; gain = 10^(gain_db*(1/20)); dt=max(bufplaypos - project_time,0); dt*bufsrate < 120 ? underrun_cnt+=1; rdpos = bufwritecursor - ((dt*bufsrate - 120)|0)*nch; rdpos < bufstart ? rdpos += bufend-bufstart; grej=min(rej,abs(6.6*gmem[rdpos])*rej); If memory serves about the gmem potential positions, you can do this 3,000,000/176401=about 17 times before your computer crashes. Please clean this up if you can! It might use up your RAM, so you'd might have to reboot Reaper if you run this for too long. I'm not a doctor. All I can say is that this works. |
03-31-2021, 02:27 AM | #2 |
Human being with feelings
Join Date: Aug 2015
Posts: 2,969
|
Or, if you write a little jsfx that automates sliders by audio routed to it (on as many channels as you like) - you can use parameter modulation (from the jsfx slider to the Video Processor param).
__________________
it's meant to sound like that... |
03-31-2021, 08:38 AM | #3 |
Human being with feelings
Join Date: Feb 2017
Posts: 185
|
How do you do that?
|
03-31-2021, 09:47 AM | #4 |
Human being with feelings
Join Date: Dec 2012
Posts: 7,272
|
I’ve had real problems with trying to use the PM/Link method. I could sort of get a preview that way, but it flickered and fucked up when I rendered it. I ended up having to use PM to make a slider follow the audio, then write that to an automation item, then move that to the envelope for the video processor parameter that I actually wanted to automate.
Video peeker is kind of already set up to do multichannel, and only needed a small tweak to actually do it. Here’s a thread where I posted a 64 channel version along with a version of the oscilloscope which can access any of them. It’s what prompted Justin to release that bug fix of the spectrum analyzer preset a little while back. https://forum.cockos.com/showthread.php?t=247691 Note that this, like the original, passes samples in an “interleaved” manner. The basic stereo version is laid out like: L0, R0, L1, R1... and basically there’s a pair of those for every sample in the buffer block. BTW, that is a full block of samples as determined by your audio interface settings. This array will be as long as your latency in samples times the number of channels. I hope this helps. LMK if you have questions working through the code or need help getting it to work.
__________________
Lorenzo's Tractor is Everywhere --- Ash's Tube --- Join the Partnership for a Drum Free Amerika |
03-31-2021, 12:24 PM | #5 |
Human being with feelings
Join Date: Feb 2017
Posts: 185
|
I actually used your code there on my search for solutions, it's very cool, but slightly different from what this does.
If I understand it correctly, that allows for 64 separate ins, but still only has a stereo out? What my gibberish does is take separate ins from separate tracks, keep them separate, and send them to different parameters. I was looking for a simple way to send audio data around wherever I want, as alluded to above, but I don't understand how to write/find that script. EDIT:I probably missed something in the adapted oscilloscope section tho.. |
03-31-2021, 01:06 PM | #6 |
Human being with feelings
Join Date: Dec 2012
Posts: 7,272
|
Yes my version of the Oscope allows you to pick which of the 64 channels (or 32 pairs) you want to use via one of the parameters.
__________________
Lorenzo's Tractor is Everywhere --- Ash's Tube --- Join the Partnership for a Drum Free Amerika |
03-31-2021, 03:40 PM | #7 |
Human being with feelings
Join Date: Aug 2015
Posts: 2,969
|
Is this a known bug then?
__________________
it's meant to sound like that... |
03-31-2021, 03:56 PM | #8 |
Human being with feelings
Join Date: Dec 2012
Posts: 7,272
|
I guess I’ve never seen anybody else mention it, and I never really reported it. Honestly the thing where it came up for me was pretty abusive (not to mention NSFW) all around, but it really did seem like it was the Parameter Modulation, and specifically that where I was using Link to connect to a slider which itself was PMd to follow audio that was causing the flickering issue. Anyway, I found a way around it and didn’t bother posting a bug report, but I suppose I might should.
__________________
Lorenzo's Tractor is Everywhere --- Ash's Tube --- Join the Partnership for a Drum Free Amerika |
03-31-2021, 04:25 PM | #9 |
Human being with feelings
Join Date: Aug 2015
Posts: 2,969
|
I can reproduce the behaviour - a smooth parameter mod / slider link (fine in "preview") ends up flickering when rendered.
__________________
it's meant to sound like that... Last edited by jrk; 03-31-2021 at 04:31 PM. |
03-31-2021, 04:45 PM | #10 |
Human being with feelings
Join Date: Dec 2012
Posts: 7,272
|
That’s actually part of why I ended up looking further into this whole video peeker thing, though it was kind of too late for that specific project.
I have around here somewhere a preset something like the opacity/zoom/pan preset but with audio control. I’m not sure if I ever updated it to be multichannel aware. It is NOT smoothed on the video end, though, so that if you send it regular audio, it just kind of jumps around randomly. It works best if you use some plugin on the audio side which extracts a relatively smooth envelope and puts that out as “control voltage” on audio channels which you then jam into the video peeker. This will actually be true of most things you try to modulate via audio this way. The Oscope kind of obviously doesn’t want smoothing, and the spectrogram does the whole FFT thing which is smooth by nature, but if you’re trying to do other things, well...audio signals oscillate wildly on a sample-by-sample basis and if you just grab a single sample every 1000 or so, it’s going to look just random. For most things, you need to smooth on one side of the other.
__________________
Lorenzo's Tractor is Everywhere --- Ash's Tube --- Join the Partnership for a Drum Free Amerika |
03-31-2021, 05:58 PM | #11 |
Human being with feelings
Join Date: Aug 2015
Posts: 2,969
|
It's not the smoothed / impulsive nature of the control that does it. I can reproduce it with a 1Hz sawtooth. Looks fine until rendered.
__________________
it's meant to sound like that... |
03-31-2021, 06:28 PM | #12 |
Human being with feelings
Join Date: Dec 2012
Posts: 7,272
|
Yeah sorry. My most recent post where I was talking about smoothing was kind of a separate thought. Are you making a bug report? I will +1 because I’ve seen it happen.
__________________
Lorenzo's Tractor is Everywhere --- Ash's Tube --- Join the Partnership for a Drum Free Amerika |
03-31-2021, 06:33 PM | #13 |
Human being with feelings
Join Date: Aug 2015
Posts: 2,969
|
Someone should, I'll write it up.
__________________
it's meant to sound like that... |
03-31-2021, 07:35 PM | #14 |
Human being with feelings
Join Date: Dec 2012
Posts: 7,272
|
I guess I kind of feel like we’re doing something we’re not supposed to and that’s why it doesn’t work and there’s maybe not any good solution for the devs to fix it. Hopefully I’m wrong, and it’s worth a shot, but... :/
PS BTW sort of on a new tangent - if one happened to have JS plugin with a slider that went from 0 to 1, and literally just its own output to that value (spl0 = slider1; ), one could use PM on that, then jam the output through the video peeker, and...there’s still kind of not any presets which would do anything with it except for ones I’ve written.
__________________
Lorenzo's Tractor is Everywhere --- Ash's Tube --- Join the Partnership for a Drum Free Amerika |
04-02-2021, 01:43 AM | #15 |
Human being with feelings
Join Date: Aug 2015
Posts: 2,969
|
I've posted a bug report, If you want to +1 it, please do.
https://forum.cockos.com/showthread.php?t=251679
__________________
it's meant to sound like that... |
Thread Tools | |
Display Modes | |
|
|