Go Back   Cockos Incorporated Forums > REAPER Forums > REAPER for Spatial Audio

Reply
 
Thread Tools Display Modes
Old 03-24-2020, 04:01 PM   #1
Voyage Audio
Human being with feelings
 
Voyage Audio's Avatar
 
Join Date: Feb 2020
Location: San Diego, CA
Posts: 19
Default How to Sync Spatial Audio & 360 Video — Includes Reaper Project Download

We created a tutorial over on the Voyage Audio blog to help people sync Spatial Audio to 360 Video using Facebook 360 Spatial Workstation. There is also a downloadable Reaper Session to help with understanding the workflow. Let me know if we missed something and hope it can help some people out!

https://voyage.audio/how-to-sync-spa...-360-vr-video/
__________________
Download The Free Spatial Mic Reaper Session: https://voyage.audio/listen-to-spatial-mic/
Voyage Audio is offline   Reply With Quote
Old 10-15-2020, 09:12 AM   #2
mrlimbic
Human being with feelings
 
mrlimbic's Avatar
 
Join Date: Nov 2009
Location: UK
Posts: 669
Default

Nice tutorial.

I make an XML to Reaper conversion tool.

It occurred to me that Final Cut Pro X which has native 360 video support also exports pan, tilt, roll metadata for each clip in it's XML exports.

Currently my tool ignores this but it would be possible to convert this metadata into automation in reaper (especially easy if using the ATK as those are JS FX).

Would this be useful?
__________________
Vordio - Post Production Toolkit
http://vordio.net
mrlimbic is offline   Reply With Quote
Old 10-19-2020, 01:40 AM   #3
Boardroom
Human being with feelings
 
Join Date: Oct 2019
Posts: 2
Default

Quote:
Originally Posted by mrlimbic View Post
Nice tutorial.

I make an XML to Reaper conversion tool.

It occurred to me that Final Cut Pro X which has native 360 video support also exports pan, tilt, roll metadata for each clip in it's XML exports.

Currently my tool ignores this but it would be possible to convert this metadata into automation in reaper (especially easy if using the ATK as those are JS FX).

Would this be useful?
This would be a gamechanger for me, and i would purchase vordio just for this!
Currently it is a complete time waster trying to re position spatial audio again, after the video has been rotated..
The perspective is always changed in the video edit, and all my intricate spatial locking goes to waste.
Afterwards i have to automate by eye, ear, and SPL power map.If i am lucky (next to never) I have access to the NLE or the editor tells me what they have done. Then i have idea where to start. Either way it takes ages, and there is no automated process to do this in any DAW/NLE. I searched believe me 🙃

You would get loads of the spatial audio community on board with this function alone!
If this could be done for Premier too, you would get them all! 😉

I use higher order plugins, not the atk, but any automation can be reassigned in reaper anyway. Even easier if its labelled...
This is even useful for vbap/dbap plugs that allow rotation too.. Which i sometimes use for discreet channel based installations.

Quite excited at the prospect!

Cheers
Boardroom is offline   Reply With Quote
Old 10-19-2020, 04:15 AM   #4
mrlimbic
Human being with feelings
 
mrlimbic's Avatar
 
Join Date: Nov 2009
Location: UK
Posts: 669
Default

Quote:
Originally Posted by Boardroom View Post
This would be a gamechanger for me, and i would purchase vordio just for this!
Currently it is a complete time waster trying to re position spatial audio again, after the video has been rotated..
The perspective is always changed in the video edit, and all my intricate spatial locking goes to waste.
Afterwards i have to automate by eye, ear, and SPL power map.If i am lucky (next to never) I have access to the NLE or the editor tells me what they have done. Then i have idea where to start. Either way it takes ages, and there is no automated process to do this in any DAW/NLE. I searched believe me ��

You would get loads of the spatial audio community on board with this function alone!
If this could be done for Premier too, you would get them all! ��

I use higher order plugins, not the atk, but any automation can be reassigned in reaper anyway. Even easier if its labelled...
This is even useful for vbap/dbap plugs that allow rotation too.. Which i sometimes use for discreet channel based installations.

Quite excited at the prospect!

Cheers
I'll check Premiere today to see if that includes 360 automation/static adjustment.

Worth considering that the 360 metadata belongs to the video item, not tracks or audio items. So how best to transfer it?

Vordio already allows you to include video clips as empty items (it can be useful to see where the picture cuts are for sound design). So I could put a JSFX on the empty video item to hold the automation. How easy is it to move item FX automation to another item or track? Or else could dump the metadata into item notes and have a script transfer it. Vordio already dumps useful metadata such as scene/take into item notes.

Another possibility would be to dump ADM axml (audio definition model) metadata. There is now a free reaper extension for importing ADM for spatial audio.

https://www.bbc.co.uk/rd/blog/2020-1...software-tools

https://www.youtube.com/watch?v=u7P5...ture=emb_title

What will be the best workflow?
__________________
Vordio - Post Production Toolkit
http://vordio.net

Last edited by mrlimbic; 10-19-2020 at 04:23 AM.
mrlimbic is offline   Reply With Quote
Old 10-19-2020, 05:30 AM   #5
mrlimbic
Human being with feelings
 
mrlimbic's Avatar
 
Join Date: Nov 2009
Location: UK
Posts: 669
Default

Just checked Premiere XML. It doesn't include the VR rotate filter metadata in the XML.

"Translation issue:
Sequence <wells ambi weds> at , video track 1: Effect <VR Rotate Sphere> on Clip <wells ambi weds.mov> not translated."

However it may be possible to script Premiere to export axml for VR clips which could then be loaded in reaper using the EAR Production Suite.

https://premiere-scripting-guide.rea...nentparam.html
__________________
Vordio - Post Production Toolkit
http://vordio.net
mrlimbic is offline   Reply With Quote
Old 10-19-2020, 10:09 AM   #6
mrlimbic
Human being with feelings
 
mrlimbic's Avatar
 
Join Date: Nov 2009
Location: UK
Posts: 669
Default

OK. I have got something basic working with FCPX.

These time=angle pairs get added to item notes for empty video items. The negative time values are not a bug. My test had out of range keyframes on clips in FCPX. I could filter them out but probably not necessary, especially if they are to be converted to track automation.

A script could read these and apply them to plugin automation parameters. Add the the item position to parameter time and you'd have track automation point time.

Static rotate (clip with no keyframes)

Code:
TILT: 72.2522
PAN: -98.1707
ROLL: -267.598
Automated rotate (clip with keyframes)

Code:
TILT: -62.920=14.192, -50.760=14.192, 12.840=-254.793, 29.160=88.8078, 92.360=14.192
PAN: -62.920=-198.393, -50.760=-184.62, 12.840=-182.885, 29.160=-104.839, 92.360=-10.1287
ROLL: -62.920=0.0, -50.760=0.0, 12.840=177.096, 29.160=438.698, 92.360=0.0
I've asked an adobe developer if a script can access the VR rotate plugin parameters in Premiere. Hopefully the answer is yes.
__________________
Vordio - Post Production Toolkit
http://vordio.net

Last edited by mrlimbic; 10-19-2020 at 10:15 AM.
mrlimbic is offline   Reply With Quote
Old 10-20-2020, 07:44 AM   #7
mrlimbic
Human being with feelings
 
mrlimbic's Avatar
 
Join Date: Nov 2009
Location: UK
Posts: 669
Default

Quote:
Originally Posted by mrlimbic View Post
I've asked an adobe developer if a script can access the VR rotate plugin parameters in Premiere. Hopefully the answer is yes.
Unfortunately the answer is no to being able to script VR parameters in Premiere.

However, that info is in the .pproj file. I could start supporting .pproj format in future but it is a lot of work. It's complex and undocumented so might take a long time to debug. So won't happen soon.

However for FCPX I've done the first part already. Now I just need to write a reaper script that creates automation from the item notes. So that will be possible very soon.
__________________
Vordio - Post Production Toolkit
http://vordio.net
mrlimbic is offline   Reply With Quote
Old 10-26-2020, 05:33 AM   #8
Boardroom
Human being with feelings
 
Join Date: Oct 2019
Posts: 2
Default

Quote:
Originally Posted by mrlimbic View Post
Unfortunately the answer is no to being able to script VR parameters in Premiere.

However, that info is in the .pproj file. I could start supporting .pproj format in future but it is a lot of work. It's complex and undocumented so might take a long time to debug. So won't happen soon.

However for FCPX I've done the first part already. Now I just need to write a reaper script that creates automation from the item notes. So that will be possible very soon.
Awesome! Let me know when you want it beta tested, it will be my pleasure!
If all good I'll spray some promo over the spatial audio forums. There will be a lot if happy people out there 😉
Boardroom is offline   Reply With Quote
Old 10-27-2020, 01:08 PM   #9
mrlimbic
Human being with feelings
 
mrlimbic's Avatar
 
Join Date: Nov 2009
Location: UK
Posts: 669
Default

Quote:
Originally Posted by Boardroom View Post
Awesome! Let me know when you want it beta tested, it will be my pleasure!
If all good I'll spray some promo over the spatial audio forums. There will be a lot if happy people out there 😉
I've got a rough script working fine with ATK.

It searches for parameter names, so in theory it should work with any plugins with roll, tilt, pan, yaw, pitch etc.

However when I tried it with FB360 the VST3 will only accept hard min/max values, nothing in between. So 180 or -180 degrees but not 0.

I don't know if this is a REAPER issue or a FB workstation VST issue.

I'm very confused. Does the FB360 VST allow automation injection? Have you tried using other scripts that generate automation with it?
__________________
Vordio - Post Production Toolkit
http://vordio.net
mrlimbic is offline   Reply With Quote
Old 10-28-2020, 06:59 AM   #10
mrlimbic
Human being with feelings
 
mrlimbic's Avatar
 
Join Date: Nov 2009
Location: UK
Posts: 669
Default

Quote:
Originally Posted by mrlimbic View Post
I've got a rough script working fine with ATK.

It searches for parameter names, so in theory it should work with any plugins with roll, tilt, pan, yaw, pitch etc.

However when I tried it with FB360 the VST3 will only accept hard min/max values, nothing in between. So 180 or -180 degrees but not 0.

I don't know if this is a REAPER issue or a FB workstation VST issue.

I'm very confused. Does the FB360 VST allow automation injection? Have you tried using other scripts that generate automation with it?
OK. I've worked out automation problem.

One limitation though is that FCPX allows infinite rotation (outside -180 to +180 range) but FB360 VST3 doesn't. So it won't work with large adjustments. The video editor will have to be aware of this & stick to -180 to +180.
__________________
Vordio - Post Production Toolkit
http://vordio.net
mrlimbic is offline   Reply With Quote
Old 10-28-2020, 07:51 AM   #11
mrlimbic
Human being with feelings
 
mrlimbic's Avatar
 
Join Date: Nov 2009
Location: UK
Posts: 669
Default

OK. I now have a working demo.

__________________
Vordio - Post Production Toolkit
http://vordio.net
mrlimbic is offline   Reply With Quote
Old 10-29-2020, 10:48 AM   #12
mrlimbic
Human being with feelings
 
mrlimbic's Avatar
 
Join Date: Nov 2009
Location: UK
Posts: 669
Default

I now have an official release of the FCPX/FB360 VR rotate helper script in my mrlimbic reapack repository called "mrlimbic_apply vr rotate from item notes.lua"

https://github.com/mrlimbic/reascrip...ster/index.xml

To test it you will need build Vordio 6.0.1747 or greater. The unlicensed version allows experimenting with short timelines (up to 3 mins).

Downloads are here currently (this location is temporary and will change when new website for Vordio 6.0 is ready)..

https://drive.google.com/drive/folde...93?usp=sharing

You need to turn on the option called "include video as empty items." This feature will create empty items for video clips that contain metadata that the script uses.

After converting the .fcpxml you can then open the RPP and use the script this way:

1. Select the empty item with the VR metadata in item notes
2. Select the FB360 track FX that should have the metadata applied to it's tilt, roll, pan parameters.
3. Run script

This is first version and only works with track FX currently (item FX can come later). I've only tested it with the FB360 plugin.

In theory it should work with any plugin that names parameters as tilt/pitch, roll, pan/yaw because it searches for words in the parameter names.

__________________
Vordio - Post Production Toolkit
http://vordio.net
mrlimbic is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 10:47 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.