Old 09-15-2021, 10:08 AM   #1
mistercw
Human being with feelings
 
Join Date: Nov 2019
Location: UK
Posts: 2
Default JSFX - compensate for latency in MIDI-driven graphics

Hello,

I have written a plugin to animate still images controlled by MIDI - the purpose is to preview MIDI markup used to control a simple lip-sync system in a game. There are a number of "visemes" (mouth shapes) and MIDI is used to switch between them.

The plugin is simple and works well. I monitor incoming MIDI events (midirecv) against the markup specification and blit the visemes. But there's one problem - the MIDI events are not affected by any audio latency, so my animation is out of sync (it's earlier than the audio I'm syncing it to).

I've added controls to apply a sync offset, which works, but it's an imperfect solution because the amount of latency is variable depending on drivers etc on different users' machines. It's also different depending on whether a MIDI editor is open on the track with the plugin. For example, on one PC, I've measured the latency (how far ahead of the synchronised audio the visemes change) at 70ms with the MIDI editor open and 438ms with it closed.

I'm making my measurements as follows:

- On track A I have a series of audio ticks
- On track B I have a MIDI item containing MIDI notes at the same time stamps as the ticks
- On track B I have an instance of my plugin with no sync offset applied - MIDI events are handled immediately when received (although I am accounting for the sample block offset within the MIDI events)
- I screen-record playback then measure the time between a viseme being displayed and the corresponding audio tick

I thought the missing piece might be PDC and I've tried setting pdc_midi to 1 but that seems to have no effect in this setup.

Is there a way to detect the offset between a MIDI event being received and when audio at the same timestamp will be rendered, taking into account the various delays / latencies involved?

Thanks,
Ciaran

PS: I'm aware that there is tech available for generating viseme data from audio sources, but there's a good reason we're not using it in this case!
mistercw is offline   Reply With Quote
Old 09-15-2021, 11:01 AM   #2
Fabian
Human being with feelings
 
Fabian's Avatar
 
Join Date: Sep 2008
Location: Sweden
Posts: 7,417
Default

Quote:
Originally Posted by mistercw View Post
- On track A I have a series of audio ticks
- On track B I have a MIDI item containing MIDI notes at the same time stamps as the ticks
- On track B I have an instance of my plugin with no sync offset applied - MIDI events are handled immediately when received (although I am accounting for the sample block offset within the MIDI events)
- I screen-record playback then measure the time between a viseme being displayed and the corresponding audio tick
You get a MIDI coming in, you set a flag for which viseme to display, and the next time that the gfx thread executes, the blit is made. Is that roughly how it works? MIDI and audio should be in sync. So I'm thinking that the problem is that the blit is done in the gfx thread, which executes roughly 30 times per second.

If you set your plugin to output a sample of some value, 0.5 for example, when a MIDI event arrives, and output zero otherwise, then you can record the output of your plugin and you can check if the delay is there also in that case. It shouldn't be, and would hint at the problem really being that the gfx thread is run too seldom for your approach to work. If you have a delay also between MIDI in and the plugins output sample, then something is wrong.
__________________
// MVHMF
I never always did the right thing, but all I did wasn't wrong...
Fabian is offline   Reply With Quote
Old 09-15-2021, 12:45 PM   #3
gxray
Human being with feelings
 
Join Date: Dec 2020
Location: Miami, FL USA
Posts: 396
Default

I have nothing of value to add, but the ingenuity here is brilliant.
__________________
Seasoned codemonkey
Dunno a thing about making music (here to learn!)
gxray is offline   Reply With Quote
Old 09-15-2021, 01:33 PM   #4
mistercw
Human being with feelings
 
Join Date: Nov 2019
Location: UK
Posts: 2
Default

Quote:
Originally Posted by Fabian View Post
I'm thinking that the problem is that the blit is done in the gfx thread, which executes roughly 30 times per second.
Your summary of the behaviour is correct, but the gfx update rate isn’t the issue. The gfx are displayed significantly -before- the audio it should be in sync with (in the test setup, 70ms with the MIDI item open, 438ms when it’s closed). If the issue was the gfx update (with audio / midi events at the same time stamp rendered / processed simultaneously) you’d expect it to be late, not early, by between 0 and 33ms.

I’m guessing that the midi is being received by the plugin at the same time the audio at the same time stamp enters the audio pipeline. The plugin immediately (well, on next gfx update) draws the gfx, but the audio is delayed by whatever latency / PDC etc exists in the pipeline.

If the same midi was being used to drive an instrument the issue wouldn’t arise because it’s audio would be passing through the same pipeline, adjusted by PDC etc.

Not sure why the offset is smaller when the midi editor is open…
mistercw is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 11:20 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.