Go Back   Cockos Incorporated Forums > REAPER Forums > REAPER for Video Editing/Mangling

Reply
 
Thread Tools Display Modes
Old 11-06-2018, 10:35 AM   #1
Meo-Ada Mespotine
Human being with feelings
 
Meo-Ada Mespotine's Avatar
 
Join Date: May 2017
Location: Leipzig
Posts: 6,621
Default Getting individual pixels (via gfx_evalrect() -- edited title)

I would like to suggest a function to the video-processor, that allows me to actually get the content of a frame/input.
Sometimes, I would love to compare the actual content of two frames on a pixel-level.
One such example would be tracking objects, analysing the content of a picture, finding out, if a video needs to be stabilized.
So I would love to get a 2-d-array, which holds the actual frame as bitmap with individual pixels to work with, as well as being able to blit this array again.

So for that, I would love to ask for the function:

image_array = gfx_get_image(input) - where image_array is an array of the like:
image_array[x][y][r] - red-color-value
image_array[x][y][g] - green-color-value
image_array[x][y][b] - blue-color-value
image_array[x][y][a] - alpha-color-value


This is based on the question about ability to compare two video-items, from this thread: https://forum.cockos.com/showthread....21#post2054521
__________________
Use you/she/her.Ultraschall-Api Lua Api4Reaper - Donate, if you wish

On vacation for the time being...
Meo-Ada Mespotine is offline   Reply With Quote
Old 11-07-2018, 03:55 PM   #2
wwwmaze
Human being with feelings
 
Join Date: Oct 2009
Posts: 99
Default

Quote:
Originally Posted by mespotine View Post
Sometimes, I would love to compare the actual content of two frames on a pixel-level.
This is already possible using gfx_evalrect():

From the docs:
Quote:
gfx_evalrect(x,y,w,h,code_string[,flags,src2])

processes a rectangle with code_string being executed for every pixel/pixel-group. code should reference, depending on colorspace: RGBA: r/g/b/a, YUY2: y1,y2,u,v, YV12: y1-y4, u, v.
Values are 0-255, and are not clamped.
set flags|=1 in order to prevent multiprocessing (if your routine needs to process pixels in-order;
side-note: variables _1-_99 are thread-local variables which will always be initialized to 0, and _0 will be initialized to the thread index (0 or 1 usually)).
set flags|=2 to ignore output (analysis-only).
if src2 specified, sr/sg/sb/sa, sy1/su/sv etc will be available to read.
In this case only the intersection of valid rectangles between src2 and the destination buffer will be valid.
Example:
Code:
//Pixel blend
//@param 1:byp "bypass?" 0 0 1 0 1
//@param 3:mode "mode (1=add 2=sub)" 1 1 2 1.5 1
//@param 5:plotCode "plot code?" 0 0 1 0 1

curX=0;
function m(msg) local(s,h,id) (
  id = match("%s",msg) ? "%s" : msg|0===msg ? "%d" : "%f";
  sprintf(s=#,id, msg);
  gfx_str_measure(s,0,h);
  gfx_set(1,1,1,1);
  gfx_str_draw(s,0,curX);
  curX+=h;
);


in1=0;
in2=1;

lcolorspace=colorspace;
colorspace='RGBA';
gfx_blit(in1);

op=mode==1 ? "+" : "-";
sprintf(#code,
"
r=min(255,max(0,r%ssr));
g=min(255,max(0,g%ssg));
b=min(255,max(0,b%ssb));
a=1;
",
op,op,op,op);


!byp ? gfx_evalrect(0,0,project_w,project_h,#code,1,in2);

plotCode ? (
  gfx_set(0.15);
  gfx_fillrect(0,0,project_w*0.4,project_h);
  m(#code);
);

colorspace=lcolorspace;
wwwmaze is offline   Reply With Quote
Old 11-09-2018, 05:27 AM   #3
Meo-Ada Mespotine
Human being with feelings
 
Meo-Ada Mespotine's Avatar
 
Join Date: May 2017
Location: Leipzig
Posts: 6,621
Default

Thanks, good to hear that
__________________
Use you/she/her.Ultraschall-Api Lua Api4Reaper - Donate, if you wish

On vacation for the time being...
Meo-Ada Mespotine is offline   Reply With Quote
Old 04-02-2020, 05:35 AM   #4
Robbi
Human being with feelings
 
Join Date: Mar 2012
Location: Italy
Posts: 1
Default adding some more hints...

Hello, I hope nobody gets offended for necroposting this or for stating things that should be obvious to Video Processor programmers.

I was trying to write a simple code in order to overlay some film grain to a video but without "losing clarity" on the processed video.

I found this thread really really useful but I had to spend more than a few hours doing a lot of research and guess work in order to understand this.

The documentation of Video Processor is on one hand very short and concise, which is just perfect for everyday reference and quick start for the advanced user, on the other hand it is almost impossible to find a clear explaination of the knowldgebase needed for non-advanced users that are approaching this system.

Just to make an example of the doubts I had before:
- How is the video signal exactly routed from the clips on the tracks up to the video window output? How do I get the input signal? How do I write an output signal?
- Where does "r%ssr" (and similar) comes from ? What are you doing with that "op,op,op,op" ?? I don't know why but I always get really confused with strings.

In the documentation these answers are a bit in between the lines and I found them not so easy to get.
Also there are a few things mixed together... it gets complicated here!

Don't get me wrong, the example of wwwmaze does an amazing job in showing the possibilities with the tool.
I just would like to repost a much simpler, cropped down and slightly different version of his code, with a lot of comments, in order to help beginners better understand the basics and ultimately what is really going on in his code.

I hope this gives some more hints in order to better understand the documentation.

Code:
//Pixel blend
// the elaboration algorithm is written into a string at lines 17 - 23
// elaboration takes place in gfx_evalrect(...) at line 39

in1 = 0; // index of video signal that comes from the track where this fx is put
in2 = 1; // index of video signal that comes from lower priority tracks

lcolorspace = colorspace; // save input colorspace
colorspace = 'RGBA';      // change color space to RGBA

gfx_blit( in1 ); // put "in1" in the framebuffer (which is the video output of this fx)

// put the pixel elaboration code inside the string #code
// r,g,b,a are video buffer's red, green, blue and alpha channel
// sr,sg,sb are src's red, green, blue channel
// in this example src = in2, see gfx_evalrect(...) at the end
sprintf(#code,
"
r = min(255, max(0, max(r, sr))); // the highest red channel wins
g = min(255, max(0, max(g, sg))); // the highest green channel wins
b = min(255, max(0, max(b, sb))); // the highest blue channel wins
a = 1; // (nothing is transparent)
");

// prepare the bits of "flags" for gfx_evalrect()
FLAGS_PREVENT_MULTIPROCESSING = 1; // process pixels in order
FLAGS_IGNORE_OUTPUT = 0;           // ??? I'm not sure what this does exactly
FLAGS_MULTIPROCESS_DIRECTION = 0;  // 0 = horizontal (1 = vertical)
FLAGS_MULTIPROCESS_SENSE = 0;      // 0 = from the left (or the top) , 1 = opposite sense

// compose the "flags" variable
flags = 0; // init flags value
flags |= FLAGS_PREVENT_MULTIPROCESSING << 0;  // set first bit value
flags |= FLAGS_IGNORE_OUTPUT           << 1;  // set second bit value
flags |= FLAGS_MULTIPROCESS_DIRECTION  << 2;  // set third bit value
flags |= FLAGS_MULTIPROCESS_SENSE      << 3;  // set fourth bit value

// process the framebuffer (which contains "in1") and the "in2" video with the "#code"
gfx_evalrect(0,0,project_w,project_h,#code,flags,in2);

// change back to input colorspace
colorspace = lcolorspace;
Robbi is offline   Reply With Quote
Old 04-02-2020, 02:58 PM   #5
KeithHandy
Human being with feelings
 
Join Date: Mar 2018
Posts: 14
Default

Thanks, Robbi, I am definitely looking for more explicitly-commented code like that to learn from! Reverse-engineering the existing examples has been very time consuming and I've only gotten so far.
KeithHandy is offline   Reply With Quote
Old 04-06-2020, 06:50 AM   #6
Eliseat
Human being with feelings
 
Eliseat's Avatar
 
Join Date: Mar 2018
Location: Cologne
Posts: 1,362
Default

Quote:
Originally Posted by Robbi View Post
Hello, I hope nobody gets offended for necroposting this or for stating things that should be obvious to Video Processor programmers.

I was trying to write a simple code in order to overlay some film grain to a video but without "losing clarity" on the processed video.

I found this thread really really useful but I had to spend more than a few hours doing a lot of research and guess work in order to understand this.

The documentation of Video Processor is on one hand very short and concise, which is just perfect for everyday reference and quick start for the advanced user, on the other hand it is almost impossible to find a clear explaination of the knowldgebase needed for non-advanced users that are approaching this system.

Just to make an example of the doubts I had before:
- How is the video signal exactly routed from the clips on the tracks up to the video window output? How do I get the input signal? How do I write an output signal?
- Where does "r%ssr" (and similar) comes from ? What are you doing with that "op,op,op,op" ?? I don't know why but I always get really confused with strings.

In the documentation these answers are a bit in between the lines and I found them not so easy to get.
Also there are a few things mixed together... it gets complicated here!

Don't get me wrong, the example of wwwmaze does an amazing job in showing the possibilities with the tool.
I just would like to repost a much simpler, cropped down and slightly different version of his code, with a lot of comments, in order to help beginners better understand the basics and ultimately what is really going on in his code.

I hope this gives some more hints in order to better understand the documentation.

Code:
//Pixel blend
// the elaboration algorithm is written into a string at lines 17 - 23
// elaboration takes place in gfx_evalrect(...) at line 39

in1 = 0; // index of video signal that comes from the track where this fx is put
in2 = 1; // index of video signal that comes from lower priority tracks

lcolorspace = colorspace; // save input colorspace
colorspace = 'RGBA';      // change color space to RGBA

gfx_blit( in1 ); // put "in1" in the framebuffer (which is the video output of this fx)

// put the pixel elaboration code inside the string #code
// r,g,b,a are video buffer's red, green, blue and alpha channel
// sr,sg,sb are src's red, green, blue channel
// in this example src = in2, see gfx_evalrect(...) at the end
sprintf(#code,
"
r = min(255, max(0, max(r, sr))); // the highest red channel wins
g = min(255, max(0, max(g, sg))); // the highest green channel wins
b = min(255, max(0, max(b, sb))); // the highest blue channel wins
a = 1; // (nothing is transparent)
");

// prepare the bits of "flags" for gfx_evalrect()
FLAGS_PREVENT_MULTIPROCESSING = 1; // process pixels in order
FLAGS_IGNORE_OUTPUT = 0;           // ??? I'm not sure what this does exactly
FLAGS_MULTIPROCESS_DIRECTION = 0;  // 0 = horizontal (1 = vertical)
FLAGS_MULTIPROCESS_SENSE = 0;      // 0 = from the left (or the top) , 1 = opposite sense

// compose the "flags" variable
flags = 0; // init flags value
flags |= FLAGS_PREVENT_MULTIPROCESSING << 0;  // set first bit value
flags |= FLAGS_IGNORE_OUTPUT           << 1;  // set second bit value
flags |= FLAGS_MULTIPROCESS_DIRECTION  << 2;  // set third bit value
flags |= FLAGS_MULTIPROCESS_SENSE      << 3;  // set fourth bit value

// process the framebuffer (which contains "in1") and the "in2" video with the "#code"
gfx_evalrect(0,0,project_w,project_h,#code,flags,in2);

// change back to input colorspace
colorspace = lcolorspace;
Very cool. Many thanks for participating.
__________________
☆.。.:*・°☆.。.:*・°☆.。.:*・°☆REAPER//✿◔‿◔)°☆.。.:*・°☆.。.:*・°☆
Eliseat is offline   Reply With Quote
Old 04-06-2020, 07:31 AM   #7
Eliseat
Human being with feelings
 
Eliseat's Avatar
 
Join Date: Mar 2018
Location: Cologne
Posts: 1,362
Default

This runs very smooth on my laptop. If you don't mind I put your preset into the video presets thread in the video section. Many thanks.
__________________
☆.。.:*・°☆.。.:*・°☆.。.:*・°☆REAPER//✿◔‿◔)°☆.。.:*・°☆.。.:*・°☆
Eliseat is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 04:12 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.