Go Back   Cockos Incorporated Forums > REAPER Forums > REAPER for Spatial Audio

Reply
 
Thread Tools Display Modes
Old 08-01-2022, 06:23 PM   #1
Jazzooka
Human being with feelings
 
Jazzooka's Avatar
 
Join Date: May 2013
Posts: 221
Default 360 photography with ambisonic audio : import/view a 360 video of a pic in Reaper?

Hi there Reaperees

Am pretty new to spatial audio. A colleague photographer had this idea of proposing virtual tours of his work with added audio/music that I would provide and mix.

He did a few 360 tours with VR-tours software some time ago where you can add either ambisonic files or mono/stereo objects to place where you want and export the tour.

But he would rather leave the audio to me, to place and mix it how I see fit.

As DAWs are timeline-based, importing just a 360 picture wouldn’t work.

So I thought of creating a 360 video from the picture, given a precise duration that the photographer would like and make/adapt the audio and music based on that duration.

Video questions : any idea on how to transform a 360 picture to a video and most importantly : what format, if any, could be imported by Reaper, while being able to see and audition the result prior to bouncing?

For instance, simplified : if a picture contains a river to the left side surround, with audio/music made for that side and the right side surround has a forest with its own audio/music : can you follow that around in Reaper from the video, either by moving the mouse on the video or do you need a HRTF tool or other?

Thanks for any info

Best

J
__________________
"Who seeks truth will find no comfort, who seeks comfort will find no truth" https://trosta.bandcamp.com
Jazzooka is offline   Reply With Quote
Old 08-02-2022, 02:19 PM   #2
plush2
Human being with feelings
 
Join Date: May 2006
Location: Saskatoon, Canada
Posts: 2,110
Default

That sounds like a very fun project.

You can import an equirectangular projection image (jpeg, png etc.) into a Reaper track by dragging and dropping.

Once it is on the track you can extend it to the time length over which you want it to play.

Once in Reaper you need to put a video processor on the image track and select the "Equirectangular/spherical 360 panner" preset. This will allow you to view the image in a VR projection.
plush2 is offline   Reply With Quote
Old 08-04-2022, 09:29 AM   #3
Jazzooka
Human being with feelings
 
Jazzooka's Avatar
 
Join Date: May 2013
Posts: 221
Default

Thank you very much Plush2 🙏

It is a very fun project, the photographer is a good friend, and he suggested not do "lifelike" inserts of sound, but go to various abstract or unexpected places soundwise, which leaves margin for fun

I will probably need to re-read a couple of times what you wrote as I am a newbie when it comes to spatial audio & video/photo.

Cheers!

J

Quote:
Originally Posted by plush2 View Post
That sounds like a very fun project.

You can import an equirectangular projection image (jpeg, png etc.) into a Reaper track by dragging and dropping.

Once it is on the track you can extend it to the time length over which you want it to play.

Once in Reaper you need to put a video processor on the image track and select the "Equirectangular/spherical 360 panner" preset. This will allow you to view the image in a VR projection.
__________________
"Who seeks truth will find no comfort, who seeks comfort will find no truth" https://trosta.bandcamp.com
Jazzooka is offline   Reply With Quote
Old 01-30-2023, 07:02 AM   #4
fularr
Human being with feelings
 
Join Date: Jan 2023
Posts: 4
Default

Quote:
Originally Posted by Jazzooka View Post
Hi there Reaperees

Am pretty new to spatial audio. A colleague photographer had this idea of proposing virtual tours of his work with added audio/music that I would provide and mix.

He did a few 360 tours with VR-tours software some time ago where you can add either ambisonic files or mono/stereo objects to place where you want and export the tour.

But he would rather leave the audio to me, to place and mix it how I see fit.

As DAWs are timeline-based, importing just a 360 picture wouldn’t work.

So I thought of creating a 360 video from the picture, given a precise duration that the photographer would like and make/adapt the audio and music based on that duration.

Video questions : any idea on how to transform a 360 picture to a video and most importantly : what format, if any, could be imported by Reaper, while being able to see and audition the result prior to bouncing?

For instance, simplified : if a picture contains a river to the left side surround, with audio/music made for that side and the right side surround has a forest with its own audio/music : can you follow that around in Reaper from the video, either by moving the mouse on the video or do you need a HRTF tool or other?

Thanks for any info

Best

J
Hello everyone, I have the same job to do.

I have a 360° photo available and I would like to add a positioned audio file to it, for example on its right.

In your opinion, with REAPER is it possible to have a spatial sound so that when you see the photo (bubble) and you move inside it (turn your head with VR Headset), depending on where you move (left to right or vice versa), do you hear the sound coming from the right direction?

Thanks in advance!
fularr is offline   Reply With Quote
Old 01-30-2023, 10:14 AM   #5
plush2
Human being with feelings
 
Join Date: May 2006
Location: Saskatoon, Canada
Posts: 2,110
Default

Quote:
Originally Posted by fularr View Post
Hello everyone, I have the same job to do.

I have a 360° photo available and I would like to add a positioned audio file to it, for example on its right.

In your opinion, with REAPER is it possible to have a spatial sound so that when you see the photo (bubble) and you move inside it (turn your head with VR Headset), depending on where you move (left to right or vice versa), do you hear the sound coming from the right direction?

Thanks in advance!
It is certainly possible. I guess the first question is, where are you planning to publish these projects? That will determine what sort of audio you need to create. I've been making a spatial audio podcast on YouTube (link in my signature) that uses ambisonics as the method for creating the soundfield.
plush2 is offline   Reply With Quote
Old 01-31-2023, 01:04 AM   #6
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 9,875
Default

There is a video processor for 360°, I dont remember if is builtin to reaper but here is a demo:



https://forum.cockos.com/showpost.ph...1&postcount=22


Maybe there are more avdanced tools as VST or external players. Not my field ^^
X-Raym is offline   Reply With Quote
Old 01-31-2023, 08:45 AM   #7
fularr
Human being with feelings
 
Join Date: Jan 2023
Posts: 4
Default

Quote:
Originally Posted by plush2 View Post
It is certainly possible. I guess the first question is, where are you planning to publish these projects? That will determine what sort of audio you need to create. I've been making a spatial audio podcast on YouTube (link in my signature) that uses ambisonics as the method for creating the soundfield.
It is currently a research project. The final product will be navigable within a VR Headset system. You have to imagine photographic bubbles that can be navigated with a VR headset. Each bubble will be enriched with sounds or narrative voices. Through https://threejs.org/ the sw developers will manage the enriched audio.
fularr is offline   Reply With Quote
Old 01-31-2023, 09:50 PM   #8
plush2
Human being with feelings
 
Join Date: May 2006
Location: Saskatoon, Canada
Posts: 2,110
Default

Quote:
Originally Posted by fularr View Post
It is currently a research project. The final product will be navigable within a VR Headset system. You have to imagine photographic bubbles that can be navigated with a VR headset. Each bubble will be enriched with sounds or narrative voices. Through https://threejs.org/ the sw developers will manage the enriched audio.
So this leads to more questions. When you say bubbles that can be navigated, do you mean in the sense that you can walk around these bubbles with 6 degrees of freedom or do you enter into the bubbles and navigate them through rotation (yaw, pitch, roll) only?

It seems that threejs does do sound but spatially it just offers up what the web audio api natively supplies. If you are assigning a mono (point source) or stereo sound to these bubbles, assuming the first scenario and want the listener to be able to localize them then it should be fully capable of delivering that. On the other hand, if you are wanting to supply a more immersive soundfield with sonic ambience to fill in all the gaps as it where then you might want to do the design using ambisonics. You could get your programmers to add the capabilities of the JSAmbisonics library in conjunction with threejs.

Either way Reaper can definitely deliver your audio assets.
plush2 is offline   Reply With Quote
Old 02-03-2023, 03:25 AM   #9
fularr
Human being with feelings
 
Join Date: Jan 2023
Posts: 4
Default

Quote:
Originally Posted by plush2 View Post
So this leads to more questions. When you say bubbles that can be navigated, do you mean in the sense that you can walk around these bubbles with 6 degrees of freedom or do you enter into the bubbles and navigate them through rotation (yaw, pitch, roll) only?

It seems that threejs does do sound but spatially it just offers up what the web audio api natively supplies. If you are assigning a mono (point source) or stereo sound to these bubbles, assuming the first scenario and want the listener to be able to localize them then it should be fully capable of delivering that. On the other hand, if you are wanting to supply a more immersive soundfield with sonic ambience to fill in all the gaps as it where then you might want to do the design using ambisonics. You could get your programmers to add the capabilities of the JSAmbisonics library in conjunction with threejs.

Either way Reaper can definitely deliver your audio assets.
Like in google street view, you have a defined number of 360° photographs (bubbles) of an archaeological itinerary. For each photograph (bubble) you can move inside it, rotating, looking down, up and zooming. The final result is the possibility of entering "inside" the history of an archaeological site where you, as well as navigating as I have described you, can also listen to a narrator who explains small fragments of history or the sounds and effects of a cart with horses, or the sound of the swords of time warriors clashing, etc.
fularr is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 04:33 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.