Extension API: Link LICE bitmap to REAPER window, for seamless integration into UI

[juliansader]

Cockos has an extensive arsenal of graphics-related code, ranging from LICE functions to WDL/swell's GDI emulation and REAPER's own gfx functions for script GUIs and video editing.

These functions are just *begging* to be used by C++ extensions and ReaScripts to draw inside REAPER's own windows. This will allow seamless integration of extensions and scripts into REAPER's UI.

Here are a few examples of what can be achieved:
(The problems that these examples encounter are described below.)

Scripts such the MIDI Inspector can display information inside the MIDI editor:



CC editing scripts can display their curves:



Extra buttons and other elements can added in open space for easy access:



All kinds of other fancy graphics that would make REAPER even more attractive and interesting to work with:



The js_ReaScriptAPI extension has already exposed most of the LICE and GDI functions to ReaScripts. Using GDI, extensions and scripts can draw in any of REAPER's windows -- but, this only works properly in static windows.

In windows that are continuously being refreshed (such as the MIDI editor when CCs are being drawn, or the arrange view when the project is playing and scrolling), the script-drawn GDI graphics will either flicker or disappear altogether. (In askjf.com, Justin confirmed that it would be tricky to avoid flickering with the existing GDI functions.)

I assume this problem is due to REAPER and the script drawing alternately and independently, so whichever is drawn last in each cycle, overwrites the other and is displayed on top. In Windows OS, it seems as if the script graphics are usually drawn last and is therefore visible (albeit with flickering), but in macOS and Linux, REAPER's own graphics are usually drawn last and the script graphics are therefore not visible at all.


Suggestion:

I suggest that this problem can be solved by adding a single new LICE function that "links" a LICE bitmap with a REAPER window:

* As source bitmap, the function takes a LICE bitmap as argument, and as destination, it takes one of REAPER's windows (or the HDC of the window, or a LICE sysbitmap that is somehow linked to the window). Similar to Alphablend and WDL/swell's StretchBlt, the function can specify source and destination rectangles within the bitmap and window. EDIT: If necessary to make the function easier to code, the function may even be limited to a handful of commonly used windows such as the arrange view and the piano roll.

* Similar to other LICE blitting functions, it should understand per-pixel alpha channel (RGBA color), so that the extension-drawn LICE bitmaps can be overlaid on top of the other window contents.

* Whenever a REAPER window is refreshed, REAPER should draw any bitmaps that were linked by an extension, on top.

* An extension can animate its graphics by simply redrawing the LICE bitmap at any time: The new bitmap will then be blitted to the window in the next refresh cycle. In order to force animation over a static window that isn't continuously being redrawn (such as most of REAPER's windows, excluding trackview and midiview), the extension can send an "InvalidateRect" to the window, to induce a refresh.

* And, of course, a function to UNlink a window and bitmap.

[nappies]

Already started to do a few projects using your JS API. And yes, this flicker is annoying! Great suggestion! Full support for Julian!

PS: To users who do not write scripts!!! Please support this too! Because JS API with LICE/CDI open just crazy opportunities with UI!

[ReaMike]

+1

It will be very interesting to see what the scripting gurus can do with this.

[Vagelis]

Sounds nice +1

[Tod]

+1 from me.

[EvilDragon]

Aye!

Devs please rid us of that annoying flicker and allow scripters to do more awesome things in scripts!!

[X-Raym]

Interesting !



Are these functions accessible to C++ extension already ?

[Andernik]

+1 for me.

[Archie]

from me +1.

[juliansader]

Originally Posted by X-Raym

Are these functions accessible to C++ extension already ?

Which functions are you referring to?

The GDI and LICE functions are accessible to C++ functions, and also to scripts via the ReaScriptAPI extension. These work well for drawing in REAPER windows that are static (for example the rotating dice example), but not for drawing in animated windows.

The requested function will be new, and is not yet available for C++ extensions.

[BAYANBAYAN]

+1 good idea

[vitalker]

I hope this API will allow using dropdown menus for buttons. +1 of course!

[juliansader]

Originally Posted by vitalker

I hope this API will allow using dropdown menus for buttons. +1 of course!

The way I do dropdown menus for buttons, is to open a tiny GUI window next to the button or mouse, and then draw the menu (which is larger than the GUI window) over the GUI. If necessary, you can even make the GUI invisible using JS_Window_SetOpacity. As an example of such a custom dropdown menu, see the MIDI Inspector example above.

[Yanick]

+1, i agree

[YuriOl]

Nice!+1 👍

[vitalker]

Originally Posted by juliansader

As an example of such a custom dropdown menu, see the MIDI Inspector example above.

So this is already possible? That's why I wrote this. Thanks!

[_Stevie_]

+1 from me, too!

[SNJUK2]

+++++

[cfillion]

An extension can draw after REAPER (except in some cases where it appears to be repainting directly, eg. the playback cursor) by intercepting the WM_PAINT window message.


(Red=WM_PAINT, Green=js_ReaScript API redrawing at reaper.defer's 30Hz)

https://gist.github.com/cfillion/0d3...688175ec0026f1

[Aleksandr_Oleynik]

+1 from me

[juliansader]

Originally Posted by cfillion

An extension can draw right after REAPER (except in some cases where it appears to be repainting directly, eg. the playback cursor) by intercepting the WM_PAINT window message.

All hail our hero cfillion!

Using your approach, I've been able to get this to work beautifully! The user creates a LICE bitmap with transparent background, and then simply calls a function to "link" the bitmap to any window. Whenever the window is re-painted (or whenever the user calls InvalidateRect), the bitmap is blitted over the window. EDIT: Alas, as discussed below, after further experimentation I found that blitting LICE bitmaps doesn't work as well as drawing simple graphics such as filled rectangles.

Blitting is a very fast function, and updating the entire screen takes less than 1/10000 of a second, so I don't think it will affect performance.

It works in WindowsOS and macOS, but unfortunately blitting with transparency seems not to have been enabled in WDL/swell for Linux yet. I should check if I can replace the Linux function with my own.

I will upload some GIFs soon.

[deeb]

good luck on this! : )

[Broojacker]

+1 very useful

[EvilDragon]

Originally Posted by juliansader

All hail our hero cfillion!

Using your approach, I've been able to get this to work beautifully!

Meaning no more flicker in your Inspector? *_*

[Dragonetti]

+1 from me.

[cfillion]

Originally Posted by doppelganger

didn't work for me until i made typecasting reinterpret_cast<WNDPROC>(&paintProc) in unique ptr wrapping.

I'm on Win7 32bit (VS2013) if it matters.

Interesting, why you guys use reinterpret_cast, i made send controls window not long time ago and used c-style and it seems working ok,
but i'm noob in all this stuff), as i understand, there may be some consequences, if c-style conversion used on c++ pointer types, right?

Ah, thanks. A WNDPROC is supposed to return a long, not an int. I've fixed the return type of paintProc to be correct.

Old C-style casts do whatever they can to do the conversion. C++ casts are more specialized, making the original intent clearer and helps catch mistakes.
http://www.stroustrup.com/bs_faq2.html#static-cast

[juliansader]

Sadly, I must report that, after some further experimentation with cfillion's approach, there is still flickering when blitting images over an animated window.

I mentioned above that blitting is fast and shouldn't affect REAPER's performance, but apparently it's not fast enough to avoid glitching in the display. When drawing simple graphics such as a filled rectangle directly into the window with GDI, I see almost no flickering. But when blitting more complex LICE images such as the transparent rotating dice example, onto an animated window, flickering becomes very noticeable.

[nappies]

juliansader, So only the Reaper's devs can help with this?Or they can`t too?

[vdubreeze]

Originally Posted by nappies

PS: To users who do not write scripts!!! Please support this too! Because JS API with LICE/CDI open just crazy opportunities with UI!

OK, that's me!

+1 and however else I can support this thing that I have zero understanding of but am down for crazy opportunities with UI

[J Riley Hill]

this seems like a good thing to add. +1

[juliansader]

Originally Posted by nappies

juliansader, So only the Reaper's devs can help with this?

I dunno, perhaps cfillion and the other C++ gurus can find a perfect solution, so we don't have to wait for the devs?

Originally Posted by nappies

Or they can`t too?

I actually think it would be pretty straightforward for the devs to implement. REAPER probably uses some kind of multiple buffering for animated windows, so after updating an off-screen buffer, REAPER should simply blit whatever LICE bitmaps are linked to the window onto the buffer.

In my own experiments, this is a mere handful lines of code:
* A std::map of each "linked" window to its linked bitmaps and the sizes of the source and destination rectangles.
* Whenever WM_PAINT is received, iterate through the linked bitmaps and Alphablend (WindowsOS) or StretchBlt (macOS) each onto the window.

[nappies]

Thank! Then we need to get people's and devs attention!

[Ozman]

+1 from me as well.

More capabilities for scripters, more features can be realized.

[airon]

+1
Maybe the devs can explain some of the inner workings to us to keep things at peak efficiency.

[HDWind]

A Good Thing +1

[AndrewFalcon]

+1 Hey Guys ! Can anybody write area selection like script based on this ui Api?

[_Stevie_]

IIRC Sexan was already doing some first tests.

[Arthur McArthur]

Throwing an enthusiastic +1 on the pile

[Embass]

just workaround.. transparent gfx window example (windows os only).
https://stash.reaper.fm/manage_file/...%20example.lua

[juliansader]

I have added new "reaper.JS_Composite" functions to version 0.980 of the js_ReaScriptAPI extension, to implement cfillion's suggestion. These functions blend LICE bitmaps into REAPER windows.

Results may be affected by OS version (and perhaps even screen refresh rate?), but in my own tests:
* Compositing works perfectly on macOS, without any flickering;
* On WindowsOS, there is still a slight but noticeable glitching when the window is scrolling;
* Linux doesn't flicker, but per-pixel alpha blending isn't working.

An unconventional application of these functions would be for theming UI elements: