Old 02-16-2023, 09:13 AM   #1
valankar
Human being with feelings
 
Join Date: Jan 2016
Location: Switzerland
Posts: 130
Default Transparent background and overlays

Is there a standard way to do something like transparent background when overlaying videos? For example, I can record on my webcam with background removal. However when I overlay this webcam video, the background is black.

If I use something like Bandicam that records the webcam overlayed on the video, that black background is transparent. Is there a way to do that when overlaying images in Reaper?

Just to clarify, here is what it looks like using Bandicam to overlay with Nvidia Broadcast doing background removal from the webcam:



And here is when I instead save the Nvidia Broadcast webcam video to a file and try overlaying it in Reaper:



I'm familiar with PNG transparency but have no idea how this is handled in video. Maybe I'm missing something simple. Anyone perhaps know how to do this?
valankar is offline   Reply With Quote
Old 02-16-2023, 10:25 AM   #2
papagirafe
Human being with feelings
 
papagirafe's Avatar
 
Join Date: Aug 2020
Location: Brasil
Posts: 679
Default

Could you go to "item properties -> Properties" of your video and send it. Reaper will only consider transparency on RGBA encoded videos (which are a rarity). I have generated such videos from Reaper but never imported one from external sources. There is a way by converting pure black (RGB=0,0,0) to transparency but the edges of the real image are going to be messy. Just in case: I assume that you are using "overlay image" with transparency enable, right?
papagirafe is offline   Reply With Quote
Old 02-16-2023, 10:45 AM   #3
valankar
Human being with feelings
 
Join Date: Jan 2016
Location: Switzerland
Posts: 130
Default

Quote:
Originally Posted by papagirafe View Post
Could you go to "item properties -> Properties" of your video and send it. Reaper will only consider transparency on RGBA encoded videos (which are a rarity). I have generated such videos from Reaper but never imported one from external sources. There is a way by converting pure black (RGB=0,0,0) to transparency but the edges of the real image are going to be messy. Just in case: I assume that you are using "overlay image" with transparency enable, right?
Properties of the webcam video look like this:

Code:
Length: 0:17.783
Video: 1920x1080@60.03fps, Aspect=1.00, Decoding format=I420/YV12

Using VLC decoder (v3.0.x) for video

Video: h264

Loaded from: C:\Program Files\VideoLAN\VLC
I'm not sure about the 'transparency enable'. I don't see that option. I have on the webcam track the 'Overlay: Image overlay' effect:



I tried filter = 1, alpha channel = 0, but same result. I don't have any effect on the 'Main Video' track.

Thanks!
valankar is offline   Reply With Quote
Old 02-16-2023, 12:36 PM   #4
papagirafe
Human being with feelings
 
papagirafe's Avatar
 
Join Date: Aug 2020
Location: Brasil
Posts: 679
Default

Quote:
Originally Posted by valankar View Post
Properties of the webcam video look like this:

Code:
Length: 0:17.783
Video: 1920x1080@60.03fps, Aspect=1.00, Decoding format=I420/YV12

Using VLC decoder (v3.0.x) for video

Video: h264

Loaded from: C:\Program Files\VideoLAN\VLC
I'm not sure about the 'transparency enable'. I don't see that option. I have on the webcam track the 'Overlay: Image overlay' effect:
<snip>

I tried filter = 1, alpha channel = 0, but same result. I don't have any effect on the 'Main Video' track.

Thanks!
There on the second line, decoding format:I420/YV12. There is so far a limitation in Reaper that prevents it from handling transparency in videos encoded in various YUV pixel formats. It only handles it with RGBA encoding. You need to convert your video to an RGBA encoding. I beleive the command line version of Ffmpeg could do that but the magical formula is beyond my knowledge. There might be other tools around…
By the way, alpha is the pixel plane that tells how transparent is a RGB pixel in the final mix. You need it for transparency.

you might want to give a try to this preset that converts pure black to 100% transparent. To be inserted as first fx on your item (shift e on selected item)
Code:
// black to 100% transparent 
colorspace=‘RGBA’;
input_info(0,sw,sh)?(
  gfx_img_resize(-1,sw,sh);
  gfx_blit(0);
  gfx_evalrect(0,0,sw,sh,”r+g+b==0?a=0”);
);

Last edited by papagirafe; 02-16-2023 at 12:45 PM. Reason: omission in code
papagirafe is offline   Reply With Quote
Old 02-16-2023, 01:00 PM   #5
valankar
Human being with feelings
 
Join Date: Jan 2016
Location: Switzerland
Posts: 130
Default

Awesome, that works great! Good to know about the alpha and RGBA. Thanks!
valankar is offline   Reply With Quote
Old 02-16-2023, 01:49 PM   #6
papagirafe
Human being with feelings
 
papagirafe's Avatar
 
Join Date: Aug 2020
Location: Brasil
Posts: 679
Default

Quote:
Originally Posted by valankar View Post
Awesome, that works great! Good to know about the alpha and RGBA. Thanks!
I am happily surprised the preset worked so well! You might want to add a tolerance treshold (e.g. r,g,b <= treshold) for finer result and making the alpha proportional to the distance from 0.
papagirafe is offline   Reply With Quote
Old 02-17-2023, 02:05 AM   #7
valankar
Human being with feelings
 
Join Date: Jan 2016
Location: Switzerland
Posts: 130
Default

Out of curiosity I searched around for converting with ffmpeg, and perhaps this does it:

Code:
ffmpeg.exe -y -i .\webcam.mp4 -c:v libx264rgb out.mp4
ffmpeg version 5.1.2 Copyright (c) 2000-2022 the FFmpeg developers
  built with clang version 15.0.7
  configuration: --prefix=/d/bld/ffmpeg_1674566436592/_h_env/Library --cc=clang.exe --cxx=clang++.exe --nm=llvm-nm --ar=llvm-ar --disable-doc --disable-openssl --enable-demuxer=dash --enable-hardcoded-tables --enable-libfreetype --enable-libfontconfig --enable-libopenh264 --ld=lld-link --target-os=win64 --enable-cross-compile --toolchain=msvc --host-cc=clang.exe --extra-libs=ucrt.lib --extra-libs=vcruntime.lib --extra-libs=oldnames.lib --strip=llvm-strip --disable-stripping --host-extralibs= --enable-gpl --enable-libx264 --enable-libx265 --enable-libaom --enable-libsvtav1 --enable-libxml2 --enable-pic --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libopus --pkg-config=/d/bld/ffmpeg_1674566436592/_build_env/Library/bin/pkg-config
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\webcam.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: mp41
    creation_time   : 2023-02-16T13:48:21.000000Z
    encoder         : Bandicam 6.0.5.2033 / SHARED
    encoder-eng     : Bandicam 6.0.5.2033 / SHARED
  Duration: 00:00:17.78, start: 0.000000, bitrate: 27251 kb/s
  Stream #0:0[0x1](eng): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080, 27249 kb/s, 60 fps, 60 tbr, 60k tbn (default)
    Metadata:
      creation_time   : 2023-02-16T13:48:21.000000Z
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264rgb))
Press [q] to stop, [?] for help
[libx264rgb @ 0000024927A4B580] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264rgb @ 0000024927A4B580] profile High 4:4:4 Predictive, level 4.2, 4:4:4, 8-bit
[libx264rgb @ 0000024927A4B580] 264 - core 164 r3095 baee400 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=30 lookahead_threads=5 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'out.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: mp41
    encoder         : Lavf59.27.100
  Stream #0:0(eng): Video: h264 (avc1 / 0x31637661), rgb24(pc, gbr/unknown/unknown, progressive), 1920x1080, q=2-31, 60 fps, 15360 tbn (default)
    Metadata:
      creation_time   : 2023-02-16T13:48:21.000000Z
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc59.37.100 libx264rgb
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame= 1067 fps=137 q=-1.0 Lsize=   20485kB time=00:00:17.73 bitrate=9463.0kbits/s speed=2.27x
video:20471kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.066332%
[libx264rgb @ 0000024927A4B580] frame I:5     Avg QP:23.29  size: 87587
[libx264rgb @ 0000024927A4B580] frame P:273   Avg QP:27.24  size: 37795
[libx264rgb @ 0000024927A4B580] frame B:789   Avg QP:29.40  size: 12935
[libx264rgb @ 0000024927A4B580] consecutive B-frames:  0.6%  1.9%  2.0% 95.6%
[libx264rgb @ 0000024927A4B580] mb I  I16..4:  6.5% 84.9%  8.6%
[libx264rgb @ 0000024927A4B580] mb P  I16..4:  4.4%  4.8%  0.3%  P16..4: 14.1%  8.6%  7.6%  0.0%  0.0%    skip:60.2%
[libx264rgb @ 0000024927A4B580] mb B  I16..4:  1.0%  1.0%  0.0%  B16..8: 13.9%  1.0%  0.3%  direct: 4.6%  skip:78.3%  L0:45.8% L1:38.1% BI:16.1%
[libx264rgb @ 0000024927A4B580] 8x8 transform intra:54.2% inter:67.2%
[libx264rgb @ 0000024927A4B580] coded y,u,v intra: 24.7% 35.1% 34.7% inter: 5.7% 12.1% 12.1%
[libx264rgb @ 0000024927A4B580] i16 v,h,dc,p: 23% 26% 23% 29%
[libx264rgb @ 0000024927A4B580] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 16% 47%  2%  3%  3%  3%  2%  3%
[libx264rgb @ 0000024927A4B580] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 26% 15% 32%  5%  6%  5%  5%  4%  3%
[libx264rgb @ 0000024927A4B580] Weighted P-Frames: Y:0.4% UV:0.4%
[libx264rgb @ 0000024927A4B580] ref P L0: 35.6% 11.6% 36.4% 16.2%  0.1%
[libx264rgb @ 0000024927A4B580] ref B L0: 72.5% 20.9%  6.6%
[libx264rgb @ 0000024927A4B580] ref B L1: 91.7%  8.3%
[libx264rgb @ 0000024927A4B580] kb/s:9429.91

ffprobe.exe .\out.mp4
ffprobe version 5.1.2 Copyright (c) 2007-2022 the FFmpeg developers
  built with clang version 15.0.7
  configuration: --prefix=/d/bld/ffmpeg_1674566436592/_h_env/Library --cc=clang.exe --cxx=clang++.exe --nm=llvm-nm --ar=llvm-ar --disable-doc --disable-openssl --enable-demuxer=dash --enable-hardcoded-tables --enable-libfreetype --enable-libfontconfig --enable-libopenh264 --ld=lld-link --target-os=win64 --enable-cross-compile --toolchain=msvc --host-cc=clang.exe --extra-libs=ucrt.lib --extra-libs=vcruntime.lib --extra-libs=oldnames.lib --strip=llvm-strip --disable-stripping --host-extralibs= --enable-gpl --enable-libx264 --enable-libx265 --enable-libaom --enable-libsvtav1 --enable-libxml2 --enable-pic --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libopus --pkg-config=/d/bld/ffmpeg_1674566436592/_build_env/Library/bin/pkg-config
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\out.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf59.27.100
  Duration: 00:00:17.78, start: 0.000000, bitrate: 9436 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), gbrp(pc, gbr/unknown/unknown, progressive), 1920x1080, 9430 kb/s, 60 fps, 60 tbr, 15360 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc59.37.100 libx264rgb
But when I view the properties of this file in Reaper:

Code:
Length: 0:17.784
Video: 1920x1080@60.00fps, Aspect=1.00, Decoding format=I420/YV12

Using VLC decoder (v3.0.x) for video

Video: h264 -- convert from GBR8

Loaded from: C:\Program Files\VideoLAN\VLC
It seems the VLC decoder makes it I420/YV12 again?

I tried setting the video decoder priority back to defaults in settings (being sure to restart Reaper after these changes), which seems to use wmf, but then the video is just blank. Properties show:

Code:
Length: 0:17.784
Video: 1920x1072@60.00fps, Aspect=1.00, Decoding format=I420/YV12

Using Windows Media Foundation v2 decoder for video

Encoded video format: H264

Metadata:
    Data rate: 9316704
    Frame height: 1072
    Frame rate: 60000
    Frame width: 1920
    Length: 177840000
    Protected: 0
    Spherical: 0
    Stereo: 0
    Total bitrate: 9316704
    Video compression: {34363248-0000-0010-8000-00AA00389B71}
    Video orientation: 0
Then I tried ffmpeg as first in priority, and it is similar to vlc:

Code:
Length: 0:17.784
Video: 1920x1080@60.00fps, Aspect=1.00, Decoding format=I420/YV12

Using ffmpeg/libav DLL decoder v58.134.100 for video

Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), gbrp(pc, gbr/unknown/unknown), 1920x1080, 9430 kb/s - converted from pixfmt 73

Loaded from: C:\Users\valan\AppData\Roaming\REAPER\UserPlugins\avcodec-58.dll
Next I noticed Output is set to DirectShow EVR. I tried force RGB but this doesn't seem to change any results here. Anyhow, lots of moving parts make this a bit complicated :/.
valankar is offline   Reply With Quote
Old 02-17-2023, 05:04 AM   #8
papagirafe
Human being with feelings
 
papagirafe's Avatar
 
Join Date: Aug 2020
Location: Brasil
Posts: 679
Default

You really put me on the right track with this info! It seems that the H.264 encoding + mp4 container confuses Reaper into thinking that the pixel encoding is not RGBA. While testing, Reaper crashed quite a few times. So I tried an encoding that I know to be working from a past project: FFV1. Unfortunately the conversion outputs a huge file. With the help of dr. Google I came up with this command line for converting:
Code:
ffmpeg -i "original_video.mp4" -filter:v "format=rgba" -acodec copy -vcodec ffv1 -level 3 -threads  8 -coder 1 -context 1 -g 1 -slices 24 -slicecrc 1 "converted_rgba.mkv"
Hope this works for you as I have no video with transparencey to test it with.
papagirafe is offline   Reply With Quote
Old 02-17-2023, 05:29 AM   #9
valankar
Human being with feelings
 
Join Date: Jan 2016
Location: Switzerland
Posts: 130
Default

Quote:
Originally Posted by papagirafe View Post
Hope this works for you as I have no video with transparencey to test it with.
Thanks, hopefully getting closer. It does show RGB in properties but still with black background:

Code:
Length: 0:17.784
Video: 1920x1080@60.61fps, Aspect=1.00, Decoding format=RGB

Using VLC decoder (v3.0.x) for video

Video: FFV1

Loaded from: C:\Program Files\VideoLAN\VLC
In case it might be useful, I put webcam video with transparent background here. It's about 60 MB:

https://pontus.feralhosting.com/valankar/webcam.mp4
valankar is offline   Reply With Quote
Old 02-19-2023, 02:27 PM   #10
papagirafe
Human being with feelings
 
papagirafe's Avatar
 
Join Date: Aug 2020
Location: Brasil
Posts: 679
Default

Quote:
Originally Posted by valankar View Post
Thanks, hopefully getting closer. It does show RGB in properties but still with black background:

Code:
Length: 0:17.784
Video: 1920x1080@60.61fps, Aspect=1.00, Decoding format=RGB

Using VLC decoder (v3.0.x) for video

Video: FFV1

Loaded from: C:\Program Files\VideoLAN\VLC
In case it might be useful, I put webcam video with transparent background here. It's about 60 MB:

https://pontus.feralhosting.com/valankar/webcam.mp4

It appears your video file has no transparency info. I followed the test with google chrome developer tool and it showed up as non-transparent. https://www.google.com/search?q=test...hrome&ie=UTF-8
papagirafe is offline   Reply With Quote
Old 02-20-2023, 01:58 AM   #11
valankar
Human being with feelings
 
Join Date: Jan 2016
Location: Switzerland
Posts: 130
Default

Quote:
Originally Posted by papagirafe View Post
It appears your video file has no transparency info. I followed the test with google chrome developer tool and it showed up as non-transparent. https://www.google.com/search?q=test...hrome&ie=UTF-8
Argh, I must be recording this wrong. Maybe Bandicam or Nvidia Broadcast is doing something fishy. Will investigate some other recording methods. I'll stick to using your script for now. Thanks for looking into it!
valankar is offline   Reply With Quote
Old 02-20-2023, 02:04 AM   #12
vitalker
Human being with feelings
 
vitalker's Avatar
 
Join Date: Dec 2012
Posts: 13,333
Default

Quote:
Originally Posted by valankar View Post
Argh, I must be recording this wrong. Maybe Bandicam or Nvidia Broadcast is doing something fishy. Will investigate some other recording methods. I'll stick to using your script for now. Thanks for looking into it!
Maybe you should try https://obsproject.com/ instead?
vitalker is offline   Reply With Quote
Old 02-20-2023, 03:54 AM   #13
papagirafe
Human being with feelings
 
papagirafe's Avatar
 
Join Date: Aug 2020
Location: Brasil
Posts: 679
Default

Quote:
Originally Posted by valankar View Post
Argh, I must be recording this wrong. Maybe Bandicam or Nvidia Broadcast is doing something fishy. Will investigate some other recording methods. I'll stick to using your script for now. Thanks for looking into it!
Good luck! In then mean time, here is a version of the preset with a tolerance factor that should improve the visual result by smoothing out the transparency on edges. I would strongly suggest that you trade your chair for one of a different color...black on black doesn't help! ;-)

Code:
//black bg to alpha transparency with tolerence
//by papagirafe
//if you like this preset, please consider a tip to https://paypal.me/papagirafe
//@param tol "tolerance %" 0 0 20 5 0.1
function round(v)((v+0.5)|0);
input_info(0,sw,sh)?(
  colorspace='RGBA';
  gfx_img_resize(-1,sw,sh);
  tol=round(255*(tol/100));
  gfx_blit(0);
  res=gfx_evalrect(0,0,sw,sh,
  "r<=tol&&g<=tol&&b<=tol?a=(255/tol)*((r+g+b)/3)");
);

Last edited by papagirafe; 02-20-2023 at 09:04 AM. Reason: resizing of frame buffer was missing
papagirafe is offline   Reply With Quote
Old 02-20-2023, 04:14 AM   #14
valankar
Human being with feelings
 
Join Date: Jan 2016
Location: Switzerland
Posts: 130
Default

Quote:
Originally Posted by vitalker View Post
Maybe you should try https://obsproject.com/ instead?
Yep I tried with OBS as well. Nvidia Broadcast adds a new camera with its 'background removal' feature. When I record directly from that device everything seems to indicate it is black background and not transparent.

However, when I record the full screen with the webcam overlay, both OBS and Bandicam make that black background transparent. Not sure how they manage to do that. It's only when I try to record the nvidia broadcast camera device directly the transparency seems to not be there.
valankar is offline   Reply With Quote
Old 02-20-2023, 04:19 AM   #15
valankar
Human being with feelings
 
Join Date: Jan 2016
Location: Switzerland
Posts: 130
Default

Quote:
Originally Posted by papagirafe View Post
Good luck! In then mean time, here is a version of the preset with a tolerance factor that should improve the visual result by smoothing out the transparency on edges. I would strongly suggest that you trade your chair for one of a different color...black on black doesn't help! ;-)
Lol yes good point. Awesome, script works great as well.
valankar is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 10:04 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.