Audio visualizer for video

[antoo]

Is it possible and how to create waveaudio visualizer like in file attached? Thanks

[ashcat_lt]

There is the Decorative Spectrum Analyzer preset included in the Video Processor. Doesn't look exactly like that, but it certainly could be hacked to do so. It is possible to make more sophisticated visualizers, too. I've messed around with some stuff, but some of things like people used to make for Winamp are kind of above my pay grade.

[jrk]

You'd probably want a combination of a Video Processor preset to draw the "spectrum"
And a jsfx to to handle the audio (and do the analysis heavy lifting), passing an array of "levels" via gmem.

If the OP's still interested I'll get something together.

[ashcat_lt]

The preset that comes with Reaper gets the raw audio data via gmem from the JS video peeker plugin and then does the FFT stuff in the video processor itself. You certainly could do that FFT stuff in a JS plugin and then use gmem to send whatever you want. I'm not sure which way is better as far as CPU and stuff, but Justin wrote the existing code. Honestly, I think you can get pretty close to what the OP actually showed just by choosing proper settings in the existing preset, or at most it would take some pretty minor code tweaks.

[jrk]

Originally Posted by ashcat_lt

The preset that comes with Reaper gets the raw audio data via gmem from the JS video peeker plugin and then does the FFT stuff in the video processor itself. You certainly could do that FFT stuff in a JS plugin and then use gmem to send whatever you want. I'm not sure which way is better as far as CPU and stuff, but Justin wrote the existing code.

Indeed, and All Hail Justin, but this makes the visualisation component quite "heavy".
(I don't even know how to import code into a preset, there must be a way? Is there an "//@import" ?)

Also, developing and debugging Video Processor presets is a right pain, if they're more than trivial.

If we wanted visualisation video plugins - one way of doing it would be to concoct a common jsfx backend, which exports higher level audio features (I guess via gmem), document that, and then anyone can write relatively lightweight visualisers (VP presets) using the features.

e.g. I've got a jsfx which does a 12-band spectrum analysis - I can rejig it to put its measurements into gmem, and then the VP spectrum display preset is fairly simple.

You could write a VP preset, using the same features in a different way, for an entirely different visualisation.

[jrk]

OK, here's the spectrum "backend" jsfx.

It uses cookdsp, which you'd have to download & install yrself.
http://ajaxsoundstudio.com/cookdspdoc/

desc: JRK Spectrum Backend

options:gmem=JRK_spectrum

import cookdsp.jsfx-inc

slider1:0<-20,20,0.1>Adjustment (dB)
slider2:2<1,4,1>Response
slider3:1<1,10,1>Range


@init 
adjustment = 1;
response = 4;
range = 5;

mask = 0;

// bandwidth 1.4 or so
f1.bp(31.25, 1.4);
f2.bp(62.5, 1.4);
f3.bp(125, 1.4);
f4.bp(250, 1.4);
f5.bp(500, 1.4);
f6.bp(1000, 1.4);
f7.bp(2000, 1.4);
f8.bp(4000, 1.4);
f9.bp(8000, 1.4);
f10.bp(12000, 2);
f11.bp(16000, 2);
f12.bp(20000, 2);



level = memalloc(12);

function exp_average(acc*,  sc)
(
	alpha = 1 / (256*response);
	acc = (alpha * sc) + ( (1 - alpha) * acc) ;    
);


function set_op_level(b)
(
  
  z = (log10(level[b])) * -range;
  op_level = floor( z ) ;
  gmem[b] = op_level -1;
  

);

function do_bands(mono)
(
		level[0]  = exp_average(acc1,abs(f1.bp_do(mono)));
		level[1]  = exp_average(acc2,abs(f2.bp_do(mono)));
		level[2]  = exp_average(acc3,abs(f3.bp_do(mono)));
		level[3]  = exp_average(acc4,abs(f4.bp_do(mono)));
		level[4]  = exp_average(acc5,abs(f5.bp_do(mono)));
		level[5]  = exp_average(acc6,abs(f6.bp_do(mono)));
		level[6]  = exp_average(acc7,abs(f7.bp_do(mono)));
		level[7]  = exp_average(acc8,abs(f8.bp_do(mono)));
		level[8]  = exp_average(acc9,abs(f9.bp_do(mono)));
		level[9]  = exp_average(acc10,abs(f10.bp_do(mono)));
		level[10] = exp_average(acc11,abs(f11.bp_do(mono)));
		level[11] = exp_average(acc12,abs(f12.bp_do(mono)));
);


@slider
adjustment = dbtoa(slider1);
response = 2^slider2;
range = slider3+4;


@block

i=0;
while (i <12)
(

  set_op_level(i);
  
  i +=1;
);




@sample

do_bands((( spl0 + spl1 ) /2 ) * adjustment);

[jrk]

... and here's the Video Processor preset

// eye candy spectrum
//@gmem=JRK_spectrum


col_w = 58;
cell_h = 14;
x = 4;
start_y = y = 4;

h = cell_h - y;
w = col_w - (2*x);

project_w = 12*col_w;
project_h = (14*cell_h) + y;


new_img = gfx_img_alloc(project_w,project_h,1);
gfx_dest = new_img;



i = 0;
while (i<12)
(
  gfx_r = 1; gfx_g = 0; gfx_b = 0; gfx_a = 1;
  
  gfx_dest = new_img; 
  
  gfx_fillrect(x, y, w, h);
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_g = 1;
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_r = 0;
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_fillrect(x, y += cell_h , w, h);
  gfx_fillrect(x, y += cell_h , w, h);
  
  y = 0;

  // mask  appropriately
  
  gfx_r = 0;gfx_g = 0;gfx_b = 0;gfx_a = 0.8;
  gfx_fillrect(x, y, w, (cell_h * gmem[i]));
  
  x += col_w;
  y = start_y;
  i+=1;
);
gfx_a = 1;
gfx_dest = -1;
gfx_blit(new_img);
gfx_img_free(new_img);

[jrk]

...and here's a little clip of it in action (overlayed with the VP image overlay preset)

[vitalker]

Originally Posted by jrk

...and here's a little clip of it in action (overlayed with the VP image overlay preset)

This is awesome! I think your preset is worth to be here: https://forum.cockos.com/showthread.php?t=213832
Would you mind to post it there or allow me to do it?

[jrk]

Sure, I'll put a post together. It's really just a proof of concept - the basic idea is from the JS video peeker of course. But the approach does mean you can do the audio wrangling in a jsfx and just pass your important audio features to a (lightweight) VP preset.

[vitalker]

Originally Posted by jrk

But the approach does mean you can do the audio wrangling in a jsfx and just pass your important audio features to a (lightweight) VP preset.

This is the purpose! To make simple audio video. Another cool preset is here:
https://forum.cockos.com/showthread.php?t=237528

[jrk]

Originally Posted by vitalker

This is the purpose! To make simple audio video. Another cool preset is here:
https://forum.cockos.com/showthread.php?t=237528

Yes, that's neat.

[ashcat_lt]

So one thing that the video peeker does is to send a sort of pointer value which I thought was because the video processor and audio engine are not actually synchronous so that it’s not otherwise possible to be sure that the video processor will be looking at the same sample that is actually playing at any given time. But then, it’s also dumping samples to gmem on buffer blocks, so maybe it’s just a way of navigating the audio block/video frame thing. IDK how much it matters in practice...???

I do have a preset out here that does zoom/Pan/opacity modulated by video peeker input. I also have a version of the oscilloscope which puts video from another track in the dots.

[jrk]

I think video peeker also sends play_position, so with some clever business - comparing this to VP "project_time" - I guess it's possible to do something about synchronisation. And there's configurable "lookahead" which can probably help.

I'm not saying that I can do better than video peeker, it does what it does perfectly well. Rather, given the awkwardness of the VP preset "development environment", I'm arguing that it's easier (and more productive) to code the audio "heavy lifting" in jsfx. It also facilitates code reuse in a way that might support a wider "visualiser community".

P.S. Of course, there's no reason why the communication has to be one-way - a VP could tell the jsfx where it's at, and the jsfx could delay or advance the audio features accordingly.

[ashcat_lt]

No, I dig where you're coming from, and I don't mean to imply that you're doing it the wrong way. I frankly appreciate anybody working on developing these kinds of things because it is sorely needed and often requested.

I was kind of thinking, though, the video peeker is multichannel capable, and already built, and frankly you can pass any floating point numbers through it without really having to reinvent that part. Like, your plugin could just as easily spit out its values as "control voltages" on the audio channels, and video peeker will get it to gmem for you to do as you please. Then, too, other JS plugins could have access to that data. Not sure what you might do with it, but people might find uses like Parameter Modulation and stuff. Just a touch more modular and kind of saves some code in the analysis plugin itself.

[jrk]

Originally Posted by ashcat_lt

I was kind of thinking, though, the video peeker is multichannel capable, and already built, and frankly you can pass any floating point numbers through it without really having to reinvent that part. Like, your plugin could just as easily spit out its values as "control voltages" on the audio channels, and video peeker will get it to gmem for you to do as you please.

It's a good point. And probably the right way.

It's an accident that my example does it's levels (a) "upside down" and (b) quantised to 14 levels (it's cos I ripped it off from something else I'm doing).

I'll rejig it so as to work in the way you suggest, as soon as I get time... It's more general.

[ashcat_lt]

Course it might should come with a disclaimer to avoid sending those channels to your speakers, cause it probably won’t turn out very good.

[antoo]

Thanks a lot. Great work. But how to install and run this script?

[vitalker]

Originally Posted by antoo

Thanks a lot. Great work. But how to install and run this script?

It is not script. Reread the thread.

[antoo]

Originally Posted by vitalker

It is not script. Reread the thread.

How to use it? I do not understand many things.

[vitalker]

Originally Posted by antoo

How to use it? I do not understand many things.

https://forum.cockos.com/showthread.php?t=213832

[jrk]

Yep,

1. load in the Video processor preset (follow the link vitalker gave you for instructions).


2.
Also ( on the same track is most logical ) you'll need the jsfx (a JS Plugin)
"After downloading a JS plugin, add the files into the Effects folder of REAPER’s resource folder, then scan for new plugins.
You can quickly find the resource folder from REAPER in the options menu, “Show REAPER’s resource path in explorer/finder…”"
https://reaperblog.net/2015/06/quick...ll-js-plugins/

( you might have to restart Reaper to get it to show up....)

I've attached the file to this message for your convenience.


3. You'll also need the CookDSP JS library

you can get this hereabouts http://ajaxsoundstudio.com/cookdspdoc/

Installation instructions are there also.

4. Route audio to this track (or just drop an audio item on it)

5. View video (of course)

[antoo]

It does not work.
When i insert preset and click Ctrl+S, get error
1: syntax error: 'esc <!> : JRK Spectrum Backen'

[jrk]

The file I attached is a jsfx - NOT a Video Processor preset.

Download the file I attached up there... "SpectrumBackend.txt"

Place it in your Reaper Effects folder.


You ALSO need the video processor preset. That's up there at post 7

[antoo]

Originally Posted by jrk

The file I attached is a jsfx - NOT a Video Processor preset.

Download the file I attached up there... "SpectrumBackend.txt"

Place it in your Reaper Effects folder.

What i have to do with SpectrumBackend.txt in config/Reaper/effects folder? After scan for new plugins nothing happen.
Does not work.
Works only Decorative spectrum

[antoo]

I have renamed SpectrumBackend.txt to SpectrumBackend.js. and it works. Thanks.

[lachinhan]

Dear jrk,

How can we position Spectrum at will?

Thanks

[antoo]

Originally Posted by lachinhan

Dear jrk,

How can we position Spectrum at will?

Thanks

Add preset Video processor -- Image overlay

[jrk]

Originally Posted by antoo

Add preset Video processor -- Image overlay

Yes. That.
You'll often find that you have to "stack" Video Processor instances (with different presets) to get the effect you want.

Of course, most Image Processors do just that - i.e. process an input image (or more than one). This one's a bit different, since it doesn't take any image input.

Incidentally, the constants at the top

col_w = 58;
cell_h = 14;
x = 4;
start_y = y = 4;

can be tweaked, if you wish (I should have made these params).

[jrk]

Originally Posted by jrk

I think video peeker also sends play_position, so with some clever business - comparing this to VP "project_time" - I guess it's possible to do something about synchronisation.

That's what Decorative Oscilloscope does... More code that the VP preset has to do.
The more I look at this, the more I think it's arse up.

[ashcat_lt]

Well with the Oscope, the sync is I think a little more important. Spectrum Analysis is window based anyway, so is never really sample-for-sample accurate. There’s not really any extra code for the preset in Oscope, though. Basically just use the position pointer as an index into the gmem buffer.

[antoo]

Is it possible to show wave, during audio plays. File attached.

[ashcat_lt]

Yes it’s possible. It’s not far off from what the Oscilloscope preset does, from a coding perspective. What I think I’m seeing in your picture, though, is more like if you were recording or maybe doing a real-time render and watching it draw the wave across the screen. It should be a fairly simple hack to what’s there, but I’m not going to get to that any time soon...

[antoo]

Where in settings is possible to change video delay, because Spectr goes for a few seconds with delay?

[jrk]

The easiest way is to put the backend jsfx on a track with a copy of the audio. Then manually drag that audio leftwards (earlier) so that the video lines up with the audio.

If you do this, you wouldn't route the "early" audio to the master.


Alternatively - and again, you'd want the backend on another track, probably routing the audio to it...

Here's a version (of the backend jsfx) that adds lookahead.

desc: JRK Spectrum Backend

options:gmem=JRK_spectrum

import cookdsp.jsfx-inc

slider1:0<-20,20,0.1>Adjustment (dB)
slider2:2<1,4,1>Response
slider3:1<1,10,1>Range
slider4:lookahead_seconds=0.5<0,1.8,0.1>lookahead (seconds)

@init 
adjustment = 1;
response = 4;
range = 5;

mask = 0;

// bandwidth 1.4 or so
f1.bp(31.25, 1.4);
f2.bp(62.5, 1.4);
f3.bp(125, 1.4);
f4.bp(250, 1.4);
f5.bp(500, 1.4);
f6.bp(1000, 1.4);
f7.bp(2000, 1.4);
f8.bp(4000, 1.4);
f9.bp(8000, 1.4);
f10.bp(12000, 2);
f11.bp(16000, 2);
f12.bp(20000, 2);


pdc_bot_ch=0; pdc_top_ch=2; 


level = memalloc(12);

function exp_average(acc*,  sc)
(
	alpha = 1 / (256*response);
	acc = (alpha * sc) + ( (1 - alpha) * acc) ;  
    // ret = acc + ((sc>acc)?(sc/10):0); // biases towards attack...
);


function set_op_level(b)
(
  
  z = (log10(level[b])) * -range;
  op_level = floor( z ) ;
  gmem[b] = op_level -1;
  

);

function do_bands(mono)
(
		level[0]  = exp_average(acc1,abs(f1.bp_do(mono)));
		level[1]  = exp_average(acc2,abs(f2.bp_do(mono)));
		level[2]  = exp_average(acc3,abs(f3.bp_do(mono)));
		level[3]  = exp_average(acc4,abs(f4.bp_do(mono)));
		level[4]  = exp_average(acc5,abs(f5.bp_do(mono)));
		level[5]  = exp_average(acc6,abs(f6.bp_do(mono)));
		level[6]  = exp_average(acc7,abs(f7.bp_do(mono)));
		level[7]  = exp_average(acc8,abs(f8.bp_do(mono)));
		level[8]  = exp_average(acc9,abs(f9.bp_do(mono)));
		level[9]  = exp_average(acc10,abs(f10.bp_do(mono)));
		level[10] = exp_average(acc11,abs(f11.bp_do(mono)));
		level[11] = exp_average(acc12,abs(f12.bp_do(mono)));
);


@slider
adjustment = dbtoa(slider1);
response = 2^slider2;
range = slider3+4;
pdc_delay=(lookahead_seconds*srate)|0;

@block

i=0;
while (i <12)
(

  set_op_level(i);
  
  i +=1;
);




@sample

do_bands((( spl0 + spl1 ) /2 ) * adjustment);