Go Back   Cockos Incorporated Forums > REAPER Forums > MIDI Hardware, Control Surfaces, and OSC

Reply
 
Thread Tools Display Modes
Old 04-26-2015, 12:04 AM   #1
fundorin
Banned
 
Join Date: Feb 2014
Location: Moscow, Russia
Posts: 554
Default MIDI Feedback from Reaper to Novation SL MK II

I've been using Novation SL Remote 25 for four years and never been able to achieve a fully customizable experience out of it.
Recently I've bought myself Novation 61 SL MK II and now I'm trying to beat this two controllers, so they would do the things that I want.
Automap's HUI support for Reaper is awful. You can only get some predefined transports controls (no FFwd or Rwd) and a several pages of mixer control with faders (vol), encoders (pan) and buttons for arm, solo and mute. All other controls are unavaliable to map via Automap interface.
I'm aware about Padre's CSurf plugin existence and he did a great work, but still, I didn't like any of the templates that come with it. The options to customize them are also very low. So, I want my own templates and I want to get rid of the Automap.
Of cause, you can fully customize your SL controller in so called "Advanced mode", but you won't get LEDs and LCD feedback from DAW by using any advanced mode factory templates or templates that are made by user with MK II Template Editor.
There's one exception from this rule. You can control LEDs by MIDI while SL is in Ableton template. You'll get "Ableton is offline" message on LCD, but if you'll send CC40 message with 1 or 0, you can light the LED of the particular button, which is sitting under the first fader, for example.
I've been able to light up any LED on my MK II, even LED rings around the encoders. There are two ways how those rings will react:
1. CC message value from 0 to 10 lights the ring from left to right
2. CC message value 5 to 1 lights ring to the left from the center and 5 to 9 lights led to the right from the center.
It's also possible to light row select buttons (located to the left) via separate messages or by sending CC96 with values from 1 to 31.

For now, this is where I stuck and need your help.

First of all, I want to describe the idea:
1. SL MK II sends MIDI data to OSCII-Bot.
2. OSCII-Bot converts MIDI data to OSC commands and sends them to Reaper's Control Surface plugin (Open Sound Control).
3. Reaper reacts and sends OSC feedback to OSCII-Bot via CSurf plugin.
4. OSCII-Bot converts OSC feedback from Reaper to MIDI CC commands that would light up the corresponding LEDs on MK II.

So user can only write a script for OSCII-Bot, in which he'll describe, how should Reaper react on MIDI data from MK II.
That's the idea for now. If it'll work, then the next thing'll be adding LCD feedback support. This method can also be extended to other Novation's controllers, such as Impulse or Nocturn.

Now, I have some issues.

First is that I need to modify a custom .syx template, made with Template Editor, using some data header from Ableton's .syx template, so that the custom template's LEDs could be controlled by host, like in Ableton template and the LCD won't show error message.
I have no idea how to do this. I'm not a programmer myself, so the only thing I've noticed is the the header from Ableton's .syx template is different from every other .syx templates (FL Studio, Access Virus and so on).
I think that someone could write and alternative editor for .syx templates of just find, which SYSEX messages are used to enable LED feedback to the controller. Then we can use it to make custom templates for our controllers.

Since .syx template is a plain SYSEX commands, this task shouldn't be hard for an experienced programmer, which I'm not.

After that, user can upload modified .syx template to the controller via MIDI cable or via USB (using Template Editor) and be able to make bi-directional Controller<MIDI>OSCII-Bot<OSC>Reaper chain.

With a proper OSCII-Bot script, user can make himself a multi-paged layouts to control Reaper's transport, mixer, VST's and any other actions avaliable in Reaper. The script can also have a SHIFT buttons to expand avaliable controls and have many other features.

So, here's the idea. What do you think about it? Is it possible to achieve what I want, using SL MK II, Reaper and OSCII-Bot? Any comments will be appreciated.

P.S. English isn't my native language, so I apologize for any stylistic or spelling mistakes.

UPD. Here we have an initial version of Moss's Transformator. A powerful tool for all Novation Remote controllers to use them as a control surface for Reaper. http://mossgrabers.de/Software/Reape...atorRemote.zip

Last edited by fundorin; 05-09-2015 at 08:44 AM. Reason: added link to the first version of Moss's Transformator
fundorin is offline   Reply With Quote
Old 04-26-2015, 07:45 AM   #2
Banned
Human being with feelings
 
Banned's Avatar
 
Join Date: Mar 2008
Location: Unwired (probably in the proximity of Amsterdam)
Posts: 4,868
Default

Quote:
Originally Posted by fundorin View Post
So, here's the idea. What do you think about it? Is it possible to achieve what I want, using SL MK II, Reaper and OSCII-Bot?
Yes, your basic idea is completely valid - you can create a pretty powerful and heavily customized configuration using REAPER's OSC Control Surface feature coupled with an OSCII-bot script. As you wrote, the (only) technical hurdle seems to be configuring the SL MK II to display feedback on the LEDs and LCD. The rest is just trial, error, and coffee.

Btw, thumbs up for your ambition. I might add that even if you somehow would not get the feedback to the LEDs and LCD working, that would of course be a bummer, but it may still be worth the effort to create a custom setup even without feedback.
Quote:
Originally Posted by fundorin View Post
I think that someone could write and alternative editor for .syx templates of just find, which SYSEX messages are used to enable LED feedback to the controller. Then we can use it to make custom templates for our controllers.

Since .syx template is a plain SYSEX commands, this task shouldn't be hard for an experienced programmer, which I'm not.
Well, here's the thing: SysEx messages may be readable as raw data, but what "plain commands" they represent is entirely up to the manufacturer, and not necessarily publicly documented. If they're not publicly documented, we have to either ask the manufacturer for more specifics (you could certainly try asking Novation for this), or try to reverse engineer them ourselves.

I would suggest to load some templates (including that Ableton template) into the template editor application, and start changing things one step at a time (e.g. change one controller from type 'Disabled' to type 'CC'; change its CC number; change its MIDI channel; change its default value; change its minimum value, etc.), incrementally saving the .sysex file at each step, and carefully analyzing which byte(s) change as a result of each edit. That should probably give you some clues about the structure of the .sysex file. Next, when you're somewhat confident that you've identified what parts of the SysEx message represent the configuration of (a range of) individual controllers, you can try copy/pasting entire sections into that Ableton template, to adjust the configuration to your preferences, while (hopefully) leaving the part intact that (according to your earlier analysis) seems to be responsible for the LED (and LCD) feedback functionality.

I've had a quick look at a few .sysex files created in the template editor app, and this approach seems to be feasible - much of the data is complete gibberish to me, but the configuration of individual control elements is easy enough to identify. Btw, is the .sysex file for that Ableton template (and perhaps some others, such as the Access Virus one, for comparison) available somewhere, so I can also take a look?

(I think you already know this, since you mention noticing a difference in the file headers, but perhaps it's useful for others who read this: )
There are various applications that can show you the individual bytes of SysEx files; on OS X I like to use SysEx Librarian (a simple but handy tool focused on SysEx messages), and 0xED (a more generic binary file viewer / editor) for more advanced tasks like search/replace in .sysex files.

Hope this helps!
__________________
˙lɐd 'ʎɐʍ ƃuoɹʍ ǝɥʇ ǝɔıʌǝp ʇɐɥʇ ƃuıploɥ ǝɹ,noʎ
Banned is offline   Reply With Quote
Old 04-26-2015, 09:32 AM   #3
fundorin
Banned
 
Join Date: Feb 2014
Location: Moscow, Russia
Posts: 554
Default

Quote:
Originally Posted by Banned View Post
Yes, your basic idea is completely valid - you can create a pretty powerful and heavily customized configuration using REAPER's OSC Control Surface feature coupled with an OSCII-bot script. As you wrote, the (only) technical hurdle seems to be configuring the SL MK II to display feedback on the LEDs and LCD. The rest is just trial, error, and coffee.
Gladly, I've found a full programming reference for both feedback and CC layout for SLMkII, a couple hours ago. Here's the link to the PDF file. I've also asked Novation support for three more documents, which are mentioned in this PDF. Here's the link to my public Dropbox. For those could be reading this thread years later, the name of the file is "SLMKII MIDI Programmers Reference.pdf"
https://dl.dropboxusercontent.com/u/...0Reference.pdf

Quote:
Originally Posted by Banned View Post
Btw, is the .sysex file for that Ableton template (and perhaps some others, such as the Access Virus one, for comparison) available somewhere, so I can also take a look?
I've dumped all templates from my MkII and packed them into a single archive. There's also a template 33 - Automap Universal, which is used for automap things and can't be dumped separately, but it dumped when I've tried to dump all templates at once. So, there are 33 templates in this archive, in total.
I can do the same with my Remote SL 25, if needed.
https://dl.dropboxusercontent.com/u/..._Templates.zip

Quote:
Originally Posted by Banned View Post
(I think you already know this, since you mention noticing a difference in the file headers, but perhaps it's useful for others who read this: )
There are various applications that can show you the individual bytes of SysEx files; on OS X I like to use SysEx Librarian (a simple but handy tool focused on SysEx messages), and 0xED (a more generic binary file viewer / editor) for more advanced tasks like search/replace in .sysex files.
I've used a simple text viewer (win 8.1 here) to look at the .syx, but the way it's showing me the content isn't very readable.

In fact, I'm really glad that you said, that this concept is doable. Might take a long time, but I'm willing to complete this task, so the others could also use the setup for their own SL's.
fundorin is offline   Reply With Quote
Old 04-26-2015, 09:59 AM   #4
fundorin
Banned
 
Join Date: Feb 2014
Location: Moscow, Russia
Posts: 554
Default

I've also found a control surface daemon for SL controllers, which is, basically, a self-written Automap substitute for Linux.
It was madу by a guy, named David Greaves. Here's an article about his work - http://technotes-djg.blogspot.co.uk/...ce-daemon.html
Project page at BitBucket - https://bitbucket.org/mixerton/contr...aemon/overview

And here's the archive with source code and documentation of his project -
https://dl.dropboxusercontent.com/u/...ace-daemon.zip
fundorin is offline   Reply With Quote
Old 04-26-2015, 11:15 AM   #5
Banned
Human being with feelings
 
Banned's Avatar
 
Join Date: Mar 2008
Location: Unwired (probably in the proximity of Amsterdam)
Posts: 4,868
Default

Nice, all those documents seem very useful!

Reading p. 3-4 of the SLMKII MIDI Programmer’s Reference seems to confirm what you found earlier:
Quote:
Originally Posted by SLMKII MIDI Programmer’s Reference
IMPORTANT: In Automap mode, most of the SLMKII control information is normally handled by a ‘hidden’ USB port – any control information can, however, usually be sent to one of the other normal ports – and still have the same effect. Control Information emanating from the unit in Automap mode, cannot normally be ‘seen’ by host programs; as this is normally transmitted on the hidden port – special steps have to be taken by any application program in order to ‘see’ control changes etc. The Ableton Template is an exception to this, in that the hidden-Port is not used, and control is normally done on Port2.
So, concluding, you want to have the device in 'Automap mode', but communicating (bi-directionally) over a regular USB MIDI port, not a 'hidden' one, much like the Ableton template is doing.

After quickly browsing through that document, it still isn't clear to me exactly what part of the Ableton template SysEx message is responsible for that, though.

I'd suggest to make a custom template .sysex file (with CC numbers, ranges, names etc. - a rough mock-up should do for now), find a somewhat decent SysEx / binary file editor that works on your platform, and start copy/pasting excerpts (e.g. the first 256 bytes) from the Ableton template into it, and try if it starts working (more or less) as intended.
__________________
˙lɐd 'ʎɐʍ ƃuoɹʍ ǝɥʇ ǝɔıʌǝp ʇɐɥʇ ƃuıploɥ ǝɹ,noʎ
Banned is offline   Reply With Quote
Old 04-26-2015, 11:56 AM   #6
fundorin
Banned
 
Join Date: Feb 2014
Location: Moscow, Russia
Posts: 554
Default

Quote:
Originally Posted by Banned View Post
I'd suggest to make a custom template .sysex file (with CC numbers, ranges, names etc. - a rough mock-up should do for now), find a somewhat decent SysEx / binary file editor that works on your platform, and start copy/pasting excerpts (e.g. the first 256 bytes) from the Ableton template into it, and try if it starts working (more or less) as intended.
This is exactly what I'm going to do.

Quote:
Originally Posted by Banned View Post
So, concluding, you want to have the device in 'Automap mode', but communicating (bi-directionally) over a regular USB MIDI port, not a 'hidden' one, much like the Ableton template is doing.
In fact, I think that it might be possible to make use of the Automap mode/template itself without Automap Server running and achieve the same functionality, using those "user", "inst", "fx", "mixer", "transport", "learn" and "view" buttons as a different layout toggles.
Just need some testing without Automap Server enabled.
It might be even possible to get FX's parameters of the selected track straight to the different pages of the "fx" group. However, I still need to find out how to implement "shift" functions to OSCII-Bot scripts.

As far as I've understood, the only difference between Ableton and Automap templates is that the first one sending data to midi channel 1, while Automap sends it's "hidden" data to channel 16. But, I might be wrong.

Last edited by fundorin; 04-26-2015 at 12:05 PM.
fundorin is offline   Reply With Quote
Old 04-26-2015, 11:29 PM   #7
fundorin
Banned
 
Join Date: Feb 2014
Location: Moscow, Russia
Posts: 554
Default

Quote:
Originally Posted by Banned View Post
I'd suggest to make a custom template .sysex file (with CC numbers, ranges, names etc. - a rough mock-up should do for now), find a somewhat decent SysEx / binary file editor that works on your platform, and start copy/pasting excerpts (e.g. the first 256 bytes) from the Ableton template into it, and try if it starts working (more or less) as intended.
I've been able to send and receive SYSEX data to Automap template, using MIDI-OX, without any changes to the template itself.

Here's the example:


I've received the following, while experimenting:

Ableton template - reacts to CC messages with LED. Can send and receive commands at channel 1. Doesn't accept SYSEX LCD Messages.

Automap template - Receives SYSEX LCD messages and responds to SYSEX requests at channel 16. Doesn't react to LED change commands. Doesn't send midi data. When user presses any control, "Automap is offline" message appears on LCD.

I assume that the host should send an initial sysex greeting to the controller, so it will think that Automap Server is online and then it will communicate with the host.

I've sent the "online" message and "automap is offline" dissapeared, showing me the text strings that I've sent to the LCD earlier, but when I've pressed any control, the "automap if offline" appeared again.

This is the message. VV bb set to 12 00, nn set to 02 (Automap template):


When this message is sent, I can control LED's from the host with CC messages. This works only until I press any control on the SL. Then it shows "Automap is offline", forcing me to send "initial" command once again.


For now, I think that it's time to move straight to the OSCII-bot and start configuring the initial setup, using Automap template, since it's already able to communicate. If everything will go well with Automap template, then we can switch to Ableton template, if user wants to keep Automap functionality and use midi2osc with some of the advanced templates.

Some LED test. It's really interesting to play with those things.
https://www.youtube.com/watch?v=al9Zlq1ve10

Last edited by fundorin; 04-27-2015 at 02:46 AM.
fundorin is offline   Reply With Quote
Old 08-06-2016, 04:30 PM   #8
SEA
Human being with feelings
 
SEA's Avatar
 
Join Date: Jun 2007
Location: By The Sea
Posts: 2,238
Default

Quote:
Originally Posted by fundorin View Post
I've also found a control surface daemon for SL controllers, which is, basically, a self-written Automap substitute for Linux.
It was madу by a guy, named David Greaves. Here's an article about his work - http://technotes-djg.blogspot.co.uk/...ce-daemon.html
Project page at BitBucket - https://bitbucket.org/mixerton/contr...aemon/overview

And here's the archive with source code and documentation of his project -
https://dl.dropboxusercontent.com/u/...ace-daemon.zip
These links do not work.
__________________
JamieSEA

http://www.facebook.com/jamieseamusic
SEA is offline   Reply With Quote
Old 08-06-2016, 02:45 PM   #9
SEA
Human being with feelings
 
SEA's Avatar
 
Join Date: Jun 2007
Location: By The Sea
Posts: 2,238
Default

Quote:
Originally Posted by fundorin View Post
I've been using Novation SL Remote 25 for four years and never been able to achieve a fully customizable experience out of it.
Recently I've bought myself Novation 61 SL MK II and now I'm trying to beat this two controllers, so they would do the things that I want.
Automap's HUI support for Reaper is awful. You can only get some predefined transports controls (no FFwd or Rwd) and a several pages of mixer control with faders (vol), encoders (pan) and buttons for arm, solo and mute. All other controls are unavaliable to map via Automap interface.
I'm aware about Padre's CSurf plugin existence and he did a great work, but still, I didn't like any of the templates that come with it. The options to customize them are also very low. So, I want my own templates and I want to get rid of the Automap.
Of cause, you can fully customize your SL controller in so called "Advanced mode", but you won't get LEDs and LCD feedback from DAW by using any advanced mode factory templates or templates that are made by user with MK II Template Editor.
There's one exception from this rule. You can control LEDs by MIDI while SL is in Ableton template. You'll get "Ableton is offline" message on LCD, but if you'll send CC40 message with 1 or 0, you can light the LED of the particular button, which is sitting under the first fader, for example.
I've been able to light up any LED on my MK II, even LED rings around the encoders. There are two ways how those rings will react:
1. CC message value from 0 to 10 lights the ring from left to right
2. CC message value 5 to 1 lights ring to the left from the center and 5 to 9 lights led to the right from the center.
It's also possible to light row select buttons (located to the left) via separate messages or by sending CC96 with values from 1 to 31.

For now, this is where I stuck and need your help.

First of all, I want to describe the idea:
1. SL MK II sends MIDI data to OSCII-Bot.
2. OSCII-Bot converts MIDI data to OSC commands and sends them to Reaper's Control Surface plugin (Open Sound Control).
3. Reaper reacts and sends OSC feedback to OSCII-Bot via CSurf plugin.
4. OSCII-Bot converts OSC feedback from Reaper to MIDI CC commands that would light up the corresponding LEDs on MK II.

So user can only write a script for OSCII-Bot, in which he'll describe, how should Reaper react on MIDI data from MK II.
That's the idea for now. If it'll work, then the next thing'll be adding LCD feedback support. This method can also be extended to other Novation's controllers, such as Impulse or Nocturn.

Now, I have some issues.

First is that I need to modify a custom .syx template, made with Template Editor, using some data header from Ableton's .syx template, so that the custom template's LEDs could be controlled by host, like in Ableton template and the LCD won't show error message.
I have no idea how to do this. I'm not a programmer myself, so the only thing I've noticed is the the header from Ableton's .syx template is different from every other .syx templates (FL Studio, Access Virus and so on).
I think that someone could write and alternative editor for .syx templates of just find, which SYSEX messages are used to enable LED feedback to the controller. Then we can use it to make custom templates for our controllers.

Since .syx template is a plain SYSEX commands, this task shouldn't be hard for an experienced programmer, which I'm not.

After that, user can upload modified .syx template to the controller via MIDI cable or via USB (using Template Editor) and be able to make bi-directional Controller<MIDI>OSCII-Bot<OSC>Reaper chain.

With a proper OSCII-Bot script, user can make himself a multi-paged layouts to control Reaper's transport, mixer, VST's and any other actions avaliable in Reaper. The script can also have a SHIFT buttons to expand avaliable controls and have many other features.

So, here's the idea. What do you think about it? Is it possible to achieve what I want, using SL MK II, Reaper and OSCII-Bot? Any comments will be appreciated.

P.S. English isn't my native language, so I apologize for any stylistic or spelling mistakes.

UPD. Here we have an initial version of Moss's Transformator. A powerful tool for all Novation Remote controllers to use them as a control surface for Reaper. http://mossgrabers.de/Software/Reape...atorRemote.zip
Hey fundorin! I just picked up the 25 SL MK II and using it on top of my desk with my Yamaha P95 88 keyboard (since it doesn't have any controls, just keys).

I also have the Behringer X TOUCH coming next week.

My main use for the 25 SL MK II is for pitch and mode wheel, stop start record, and being able to (and a push of a button if possible) launch my synths, plugs and use the knobs for controlling those VST's and VSTi's. It would also be cool to use the faders for adjusting track volumes etc.

With your configurations is this pretty doable or am I just chasing rabbits here!

Also, what about the Nektar Impact 25 vs the Novation MK II? Are they any better or basically all these controllers have the same work-a-rounds to get them working right?

Thanks for your help!
__________________
JamieSEA

http://www.facebook.com/jamieseamusic
SEA is offline   Reply With Quote
Old 10-19-2016, 03:42 PM   #10
fundorin
Banned
 
Join Date: Feb 2014
Location: Moscow, Russia
Posts: 554
Default

Quote:
Originally Posted by SEA View Post
I also have the Behringer X TOUCH coming next week.
I couldn't write an osciibot script for sl, so there's no progress for me in this area. Transformator (link in the first post) works fine for those who are interested.

I suggest you to use X touch as a control surface for Reaper.
My only progress was to be able to reveal the third midi port of the controller, which is used for Automap communication. You can have bidirectional access to (and from) every control (knobs, buttons, encoders, faders, leds) oа the SL.
The only thing needed is to make a script for osciibot to utilize those controls and make OSC control surface from SL or Reaper.
fundorin is offline   Reply With Quote
Old 10-20-2016, 01:25 AM   #11
MikeD
Human being with feelings
 
Join Date: Jun 2016
Posts: 7
Default

Quote:
Originally Posted by fundorin View Post
Transformator (link in the first post) works fine for those who are interested.
Unfortunately the link in the first post appears not to work: http://mossgrabers.de/Software/Reape...atorRemote.zip

Do you know if the zip is available somewhere else to download?
MikeD is offline   Reply With Quote
Old 10-20-2016, 08:06 AM   #12
airon
Human being with feelings
 
airon's Avatar
 
Join Date: Aug 2006
Location: Berlin
Posts: 11,814
Default

Try here:
http://mossgrabers.de/Software/Push/Push.html

I have an old SL61 MK1. I'm fairly happy with it, using chroma knobs for easy targeting, but I do have two DJ Techtools Midifighter Twisters.

Those nice LED rings are going completely unused. I shall take a look at Transformator. Maybe it's a way to get general midi feedback out there.

This is way too much fucking work for a hobbyist dweeb developer like me. I mix for a living .
__________________
Using Latch Preview (Video) - Faderport 16 setup for CSI 1.1 , CSI 3.10
Website
"My ego comes pre-shrunk" - Randy Thom
airon is offline   Reply With Quote
Old 10-20-2016, 02:55 PM   #13
MikeD
Human being with feelings
 
Join Date: Jun 2016
Posts: 7
Default

Quote:
Originally Posted by airon View Post
If I've followed this correctly there should be another Transformator for Novation, but maybe I'm wrong.
MikeD is offline   Reply With Quote
Old 08-19-2019, 06:58 AM   #14
CEART
Human being with feelings
 
CEART's Avatar
 
Join Date: Jul 2018
Location: LAITW
Posts: 176
Default

Quote:
Originally Posted by fundorin View Post
There's one exception from this rule. You can control LEDs by MIDI while SL is in Ableton template. You'll get "Ableton is offline" message on LCD, but if you'll send CC40 message with 1 or 0, you can light the LED of the particular button, which is sitting under the first fader, for example.
I've been able to light up any LED on my MK II, even LED rings around the encoders. There are two ways how those rings will react:
1. CC message value from 0 to 10 lights the ring from left to right
2. CC message value 5 to 1 lights ring to the left from the center and 5 to 9 lights led to the right from the center.
It's also possible to light row select buttons (located to the left) via separate messages or by sending CC96 with values from 1 to 31.
Hi,

I know the description say Ableton, but I wanted to try in Reaper as a simple test to get light up the LED's on my 61SL MkII, I recorded in Reaper a MIDI track with CC 40 and step values from 1 to 10. To send back it to my 61SL MkII I set the track 'MIDi Hardware Output' in 'SL MKII: Port 1' and play, but no LED lights up at all.

Probably I am doing things wrong, but since this stuff is still advanced for me, I'm experimenting while reading info about that.

What I want can be achieved without automap just with advanced mode in the 61SL MkII?

What port configuration should be set in 61SL MkII to receive MIDI CC from Reaper?

Thanks on any help.

Last edited by CEART; 08-19-2019 at 10:39 PM.
CEART is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 09:16 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.