Old 01-26-2020, 12:16 PM   #1
dimentorium
Human being with feelings
 
Join Date: Jan 2020
Posts: 12
Default Prototype for Preset Management

Hi all,
its my first post for Reaper, just started a few month ago to work with it.

Wanted to ask for feedback, if my idea is worth continuing.

I build a rough prototype for browsing Halion presets that shows all preset on a 2D graph. The presets are ordered by similarity via a machine learning algorithm (tsne) and I am working to improve this.
You can prelisten every sound (each dot is a preset) and directly load it into reaper. The video is linked via dropbox( hope tats OK to everyone).
Idea behind it is, that I am trying to build a more musically oriented browsing and to find out how good I can map the sounds by similarity with an algorithm. So if you find a sound (or know it) you can search its surrounding for similar sounds.

The idea is not new, currently XLNaudio does this for drum sounds with XO and this seems to be based on a google paper https://experiments.withgoogle.com/drum-machine

But until now, no one has done this for synths and there is no product out, so I thought to give it a try.

Current status:
- LUA Script to render a folder of presets into a folder of wav files
- Python scripts to extract audio features from wavs(very basic)
- Pyton script that draws the plot, plays the sounds and loads presets into Reaper

Here is the video, hope it works
https://www.dropbox.com/s/mw6emdexn6..._Demo.mp4?dl=0

Would be very interested if you find this useful.

Thanks a lot!
Attached Images
File Type: jpg Reaper_tsne.jpg (45.5 KB, 188 views)
dimentorium is offline   Reply With Quote
Old 01-26-2020, 12:22 PM   #2
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 10,533
Default

Nice work, it must have been fun to work on!
schwa is offline   Reply With Quote
Old 01-26-2020, 01:25 PM   #3
Jason Brian Merrill
Human being with feelings
 
Jason Brian Merrill's Avatar
 
Join Date: Jun 2006
Location: Northeastern PA, USA
Posts: 20,967
Default

this looks really neat!!!
__________________
Beliefs do not require respect. People do.
Jason Brian Merrill is online now   Reply With Quote
Old 01-26-2020, 01:26 PM   #4
Jason Brian Merrill
Human being with feelings
 
Jason Brian Merrill's Avatar
 
Join Date: Jun 2006
Location: Northeastern PA, USA
Posts: 20,967
Default

so how would one populate all of this? There's a script that runs through presets? Obviously it would need to have one type of scrollable preset system right?
__________________
Beliefs do not require respect. People do.
Jason Brian Merrill is online now   Reply With Quote
Old 01-26-2020, 01:42 PM   #5
Xenakios
Human being with feelings
 
Xenakios's Avatar
 
Join Date: Feb 2007
Location: Oulu, Finland
Posts: 8,048
Default

Quote:
Originally Posted by Jason Brian Merrill View Post
so how would one populate all of this? There's a script that runs through presets? Obviously it would need to have one type of scrollable preset system right?
Yeah, the problem I see with this is that there's no universal way in plugins to programmatically change presets. (And there's no way to be sure when the plugin has actually fully changed the preset so that it is ready to offline render audio...) Still, a very cool thing.
__________________
For info on SWS Reaper extension plugin (including Xenakios' previous extension/actions) :
http://www.sws-extension.org/
https://github.com/Jeff0S/sws
--
Xenakios blog (about HourGlass, Paul(X)Stretch and λ) :
http://xenakios.wordpress.com/
Xenakios is offline   Reply With Quote
Old 01-27-2020, 01:28 AM   #6
brummbear
Human being with feelings
 
brummbear's Avatar
 
Join Date: May 2016
Location: out west
Posts: 117
Default

Looks quite useful to me. Besides XLN Audio Arturia also uses machine learning to find "similar sounds" in Analog Lab4.

While there is indeed no universal way of scanning through all presets across plugins I think your browser would work really well over NKS previews! These typically contain thousands (if not 10's of thousands) preset sounds only for Native Instruments Komplete Kontrol series. There are quite a few other plugins available by now from other vendors that also come with NKS support. And then there are also dozens of plugins which have received NKS support by users in the meantime. Finally, there is an open source tool "PresetMagician" (https://presetmagician.com/) which automatically extracts presets incl audio previews from many paid and free plugins. I have used it to generate more than ten thousand audio previews of non NI plugins. Your browser would be a nice way of sifting through these vast sound libraries...
brummbear is offline   Reply With Quote
Old 01-27-2020, 07:20 AM   #7
dimentorium
Human being with feelings
 
Join Date: Jan 2020
Posts: 12
Default

Hi guys,
thanks for the feedback to answer the questions:

@Jason: for my test I created VST presets for Halion, so I simply go through the folder of presets and generate WAVS with the same name in a different folder. So if you have a bunch of preset files that reaper can load into the VST, this will work

@Xenakios: You are right! After trying some intelligent ways to get to presets with Halion (Halion only loads Multi Presets, which are not available), I ended up with a Python Mouseautomation script, that loads all programs and clicks the appropriate buttons. Not intelligent, but works for the test and was the fastest solution.
Next issue in Reaper with the loading, I simply waited some seconds. So rendering 5000 patches took around 20k seconds. But I did not need to be present

@brummbear: if the NKS files format is oepn readable, then this is definately possible.

So will continue to work on it and keep you posted ;-)
dimentorium is offline   Reply With Quote
Old 01-27-2020, 11:04 AM   #8
brummbear
Human being with feelings
 
brummbear's Avatar
 
Join Date: May 2016
Location: out west
Posts: 117
Default

Yes, just like wav all NKS audio previews (= rendered snippets of the patches) are simple readable ogg vorbis encoded files.

For automated preset/patch rendering I recommend you take a look at PresetMagician (linked above). At least for the already supported plugins (and there are quite a few!) you get all the audio rendering freely delivered. You can ignore all the NKS stuff - just grab the extracted audio files.

For any other plugin I believe there is no other way but via mouse/keyboard automation as you already did. In fact this is how some folks actually generate the sound previews for NKS libraries that are sold on the web (e.g. here: https://freelancesoundlabs.com/index...trol-vst-packs)
brummbear is offline   Reply With Quote
Old 01-27-2020, 01:02 PM   #9
dimentorium
Human being with feelings
 
Join Date: Jan 2020
Posts: 12
Default

Ok had a look into the demo from freelancelabs and as ogg files are supported by the python library I use, that would work as well.

Also did some rework on the classification (a good friend who is working in machine learning gave me some hints). The new version seems to cluster the sounds way better than before.

Hope it works again with the video
https://www.dropbox.com/s/xxk80mnn2x...NE_V2.mp4?dl=0

In theory this classification would also work on presets for effects if you render the sounds with the same input material (for example guitars or so).

Currently thinking what would be the best way to build a real prototype for distribution.
dimentorium is offline   Reply With Quote
Old 01-27-2020, 03:34 PM   #10
brummbear
Human being with feelings
 
brummbear's Avatar
 
Join Date: May 2016
Location: out west
Posts: 117
Default

Cool. The examples in your latest video seem to work really well.

Generally speaking, I believe that this type of clustering has a great future. Reason is simple: People are sitting on libraries of 10s of thousands of patches/presets spread across many different software instruments. Manufacturers have introduced more sophisticated ways of browsing over time, with NI NKS probably being the most advanced at this point especially when combined with their plugin control via Komplete Kontrol which also covers the parameter mapping alongside the browsing experience.

But all preset browsers irrespective of the vendor suffer from the effort required to tag the presets with actual language: Every preset has to be tagged manually by a human by all sorts of characteristics (say "trumpet, "airy", "polyphonic", "sampled", .... you name it). Without proper tagging presets cannot be found easily - even with prehear functions as no one will listen to thousands of sounds before making a decision.

Clustering along sonic similarity done by machines is an efficient substitute for the tedious manual tagging process employed nowadays.


For a simple prototype I would start with a stand alone application that only does the browsing:

* allow to add folder paths (or "directories") for scanning of rendered audio files. User is responsible for having meaningful file names of rendered audio files allowing to trace back to preset of an individual plugin
* UI: allow to select which folders will be considered in the (2D) plot
* UI: Play the sound and show the path/filename when hovering/clicking with mouse over point
(* UI: maybe allow some algo tweaking to modify clustering criteria? Could also mean we would need more than just 2 dimensional plots... e.g. also move forward backward in a 3D representation.)


In later stages for integration in existing browsing hosts you would return the path/filename to a host application which would use it to actually select a preset.
brummbear is offline   Reply With Quote
Old 01-29-2020, 01:50 PM   #11
dimentorium
Human being with feelings
 
Join Date: Jan 2020
Posts: 12
Default

Hi brummbear,
thanks for the feedback, had also a talk with a neighbor who is working as a composer. He told me kinda the same, that it takes him a lot of time to tag the synthesizer sounds and especially with large libs (Omnisphere) he does not really listen to a large bunch.

So I am currently testing with nksf and making it a bit nicer. So hopefully by end of the week I should have a script that can handle Halion and nksf in parallel ;-)

Will keep you posted.
dimentorium is offline   Reply With Quote
Old 02-09-2020, 05:17 AM   #12
todd_r
Human being with feelings
 
todd_r's Avatar
 
Join Date: Nov 2006
Posts: 628
Default

This looks amazing, would be fantastic if you could figure out how to make it universal
todd_r is online now   Reply With Quote
Old 02-09-2020, 01:55 PM   #13
dimentorium
Human being with feelings
 
Join Date: Jan 2020
Posts: 12
Default

So I am working to get the Python Scripts packed as an EXE to make it bit easier. Current status is.
- The script searches a folder for VSTPRESET, NKSF, WAV and OGG files
- If the names match, a preview file is mapped to the preset file
- The features from the audio files are extracted an store to harddrive
- The clustering algorithm is performed on the data

On the graph you see the presets and can listen to the previes. If the scripts finds a database in the folder, it will load the data and not scan.
Preparing the database takes around 2-3 seconds per preset.
So still a lot to go, but when I have the exe running you can test it if you want.
Next steps are then to add the tagging from the preset files and apply these to not tagged files
dimentorium is offline   Reply With Quote
Old 02-10-2020, 08:46 AM   #14
Ozman
Human being with feelings
 
Join Date: Feb 2015
Posts: 630
Default

I'd like to see this work with any chosen VST as well as one with sample folders (like the original script).
Reminds me somewhat of XLN - XO.
Ozman is offline   Reply With Quote
Old 02-10-2020, 03:02 PM   #15
dimentorium
Human being with feelings
 
Join Date: Jan 2020
Posts: 12
Default

After trying a lot, I can at least build the exe and run it on my system. Maybe someone else can try it (130MB).
https://www.dropbox.com/s/qloev3qszg1o38z/gui.exe?dl=0

Usage is simple. Just start when the GUI has loaded select a folder containing presets (vstpreset and nkfs) and audio (wav and ogg), then wait for quite a long time (~3 seconds per audio) and some more for the calculation. Then you get the 2d graph and by clicking on a dot the audio is played back. Would be really interesting if it works on other machines. My recommendation would be not too many files (it works with my 5000 Halion presets, but is slow).

If you want to render some presets here is ma reaper script
https://www.dropbox.com/s/bva4ep5257...Multi.zip?dl=0
Needs some folder change and a project with one track containing the vst and one midi note.

And before you ask why I use recursion... the wait function does not seem to be working inside a for loop, but with recursion I can stop reaper before rendering the project.

Will continue to work on this and see where its going.
dimentorium is offline   Reply With Quote
Old 02-13-2020, 01:39 AM   #16
brummbear
Human being with feelings
 
brummbear's Avatar
 
Join Date: May 2016
Location: out west
Posts: 117
Default

Quote:
Originally Posted by dimentorium View Post
Would be really interesting if it works on other machines...
Will continue to work on this and see where its going.
Thanks for sharing! Looks good (I tested in a Windows 10 sandbox, worked)! Really promising this :-)

The clustering was quite good for the sounds I tested with (1100 nksf/ogg presets for the Helm synth). It would be great if you made different classification criteria/parameters available to choose from e.g. spectral contrast - I assume you are using the librosa library? And maybe adding a third dimension for that matter like the Google folks did by adding color to the 2D plot...

By making different classification criteria available the community might be able to identify particularly useful combinations for clustering certain sound material.

I would also prefer to run it directly as a Python script rather than the big exe file...
brummbear is offline   Reply With Quote
Old 02-13-2020, 02:57 AM   #17
dimentorium
Human being with feelings
 
Join Date: Jan 2020
Posts: 12
Default

Cool, glad it works. Thoughs th EXE would be easier for most users not familiar with Python.

I am working on cleaning up the code, then we can have a look at the scripts.

Yes, I am using librosa for the feature extraction. Next step I want is to integrate the tag information from the preset files and use that information for coloring and also applying existing tags to not tagged sounds (but that implies the use of a neural network).

But I think this is a really interesting approach for using machine learning in music, so that the user can focus on making music and not searching thousands of presets.
dimentorium is offline   Reply With Quote
Old 02-16-2020, 01:35 PM   #18
dimentorium
Human being with feelings
 
Join Date: Jan 2020
Posts: 12
Default

So heres the link to my current repo.
https://dev.azure.com/ainysynth/_git/ainysynth

It should be accesible. Still a lot of refactoring and cleaning up/documentation before going to some new work here.
There is a requirements file with most of the dependencies. I am currently using the Python 3.7.6 for programming.

Would be good to know, if its accesible.

Last edited by dimentorium; 02-16-2020 at 01:46 PM.
dimentorium is offline   Reply With Quote
Old 02-16-2020, 02:41 PM   #19
nofish
Human being with feelings
 
nofish's Avatar
 
Join Date: Oct 2007
Location: home is where the heart is
Posts: 9,388
Default

Quote:
Originally Posted by dimentorium View Post
Would be good to know, if its accesible.
Yep, it's browsable online and cloneable (I just tried).
nofish is offline   Reply With Quote
Old 02-17-2020, 01:27 AM   #20
dimentorium
Human being with feelings
 
Join Date: Jan 2020
Posts: 12
Default

Cool! Thanks for testing.

I am open for any questions and feedback. Still a lot on my list I would like to cleanup to have a solid base for future ideas ;-)
dimentorium is offline   Reply With Quote
Old 02-18-2020, 01:25 PM   #21
brummbear
Human being with feelings
 
brummbear's Avatar
 
Join Date: May 2016
Location: out west
Posts: 117
Default

Hey Philipp,
Firstly let me make clear that I am not an expert on this topic... maybe my thoughts are not overly helpful.

Quote:
Next step I want is to integrate the tag information from the preset files and use that information for coloring and also applying existing tags to not tagged sounds (but that implies the use of a neural network).
I think this is a very interesting (and ambitious) approach. However, I would first try to hone the feature extraction & dimensional reduction, followed by optional clustering (with or without machine learning). One problem I anticipate is the quantity of the training data even with large tagged preset libraries. This may be compensated by high quality of the preceding feature extraction and (optional) clustering. If you decide to add clustering on top of dimensional reduction one could also have a classical algorithm to extract "dominant" tags for that cluster from pre-tagged libraries.

Either way, in my opinion the most interesting part for practical purposes would be browsing by sonic similarity (= the "humble" approach). Having neural network* created tags on top of that would be "icing on the cake". In order to get sonic similarity right there should be flexibility in the current project phase to play with the features that are extracted and used for dimensional reduction and clustering. As speculated earlier it may depend on the sound material what works best, e.g. percussive sounds would potentially benefit if you included spectral flatness to the extracted features.

As you mentioned in the beginning XO seems to make use of TSNE for its percussive sounds and you wanted to expand the concept also for synths. For a universal approach, maybe it would be useful to include a few more features from librosa in the initial extraction, then exclude non helpful features (which will depend on the sound material) before starting TSNE. Finally, an optional clustering process.

Manuel Sanchez uses GMM and SVM for clustering features and matching to tags. He uses it on biological data sets but this could work really well with audio too. Check this out: https://github.com/mscastillo/Classification

* a GMM or a SVM might be more suitable than a neural network for this task. In particular, a GMM could associate multiple characteristics, e.g. a sound can carry multiple tags like "trumpet", "airy", "pad".

Last edited by brummbear; 02-20-2020 at 05:56 PM.
brummbear is offline   Reply With Quote
Old 02-26-2020, 01:08 AM   #22
dimentorium
Human being with feelings
 
Join Date: Jan 2020
Posts: 12
Default

Hi Brummbear,
still on refatoring and will work on making the analysis part a bit simpler to extend with custom code. Did you manage to run the scripts on your computer?

I read through several papers and nearly all use the standard features (chroma, mfcc, ...) for identifiying sound thus I did not put too much brain into this part. Would also beliave that having the ADSR curve should be interesting to work on sounds.

And thanks for the link!

So hope I can continue by end of the week, this week was a bit busy.

By the way it would be interesting if users would send in their presets, to build up a large scale open database with synthesizer sounds for creating models and having them available to everyone.

Last edited by dimentorium; 02-26-2020 at 03:43 AM.
dimentorium is offline   Reply With Quote
Old 02-26-2020, 04:08 AM   #23
TonE
Human being with feelings
 
Join Date: Feb 2009
Location: Reaper HAS send control via midi !!!
Posts: 2,060
Default

Thanks for sharing.
Quote:
Originally Posted by dimentorium View Post
But until now, no one has done this for synths ...
Wrong , I did long time ago, it worked, however it is not everyone's style of use case, it requires combination of multiple external tools, I took Hierarchical Clustering Explorer*, then converting fxb preset banks to text form, with another tool, then making actions inside HCE triggering out the correct program change events, to reaper, so any HCE selection resulted in immediate correct sound selection in Reaper or elseDAW, you could play around, was good. As a learning tool it was interesting, if you want to compare multiple presets for their similarity or differences in their parameter combinations. Nowadays, taking even the audio output as image, something like spectral peaks views as exported .jpg/.png, if we would have it, you could even compare those.

* http://www.cs.umd.edu/hcil/hce/
TonE is offline   Reply With Quote
Old 02-26-2020, 04:18 AM   #24
TonE
Human being with feelings
 
Join Date: Feb 2009
Location: Reaper HAS send control via midi !!!
Posts: 2,060
Default

Quote:
Originally Posted by Xenakios View Post
Yeah, the problem I see with this is that there's no universal way in plugins to programmatically change presets. (And there's no way to be sure when the plugin has actually fully changed the preset so that it is ready to offline render audio...) Still, a very cool thing.
Bank select, program change? Not universal enough?

Command line tools allow anything, you could allow 1 second wait then after each preset selection, for example. Probably you are talking of huuuge sample libraries? For normal vsti synths it should work immediately without any waiting.
TonE is offline   Reply With Quote
Old 02-26-2020, 04:59 AM   #25
TonE
Human being with feelings
 
Join Date: Feb 2009
Location: Reaper HAS send control via midi !!!
Posts: 2,060
Default

Quote:
Originally Posted by brummbear View Post
Yes, just like wav all NKS audio previews (= rendered snippets of the patches) are simple readable ogg vorbis encoded files.

For automated preset/patch rendering I recommend you take a look at PresetMagician (linked above). At least for the already supported plugins (and there are quite a few!) you get all the audio rendering freely delivered. You can ignore all the NKS stuff - just grab the extracted audio files.

For any other plugin I believe there is no other way but via mouse/keyboard automation as you already did. In fact this is how some folks actually generate the sound previews for NKS libraries that are sold on the web (e.g. here: https://freelancesoundlabs.com/index...trol-vst-packs)
Why preset magician does not work with any vsti?
TonE is offline   Reply With Quote
Old 02-26-2020, 05:18 AM   #26
TonE
Human being with feelings
 
Join Date: Feb 2009
Location: Reaper HAS send control via midi !!!
Posts: 2,060
Default

Quote:
Originally Posted by dimentorium View Post
I read through several papers and nearly all use the standard features (chroma, mfcc, ...) for identifiying sound thus I did not put too much brain into this part. Would also beliave that having the ADSR curve should be interesting to work on sounds.

By the way it would be interesting if users would send in their presets, to build up a large scale open database with synthesizer sounds for creating models and having them available to everyone.
There is also sonic-annotator, command line brother of sonic visualiser.

Which is best tool for generating images from .wav/.ogg files? Which contain as much of their relevant features in that image, visually, meaning similar images (colors and shapes) should sound similarly. Spectral peaks view could be good candidate, but we can not generate .jpg/.png from Reaper.

The trick is finding best command line tools first, if already available somehwere. Then adding the missing rest.

For the audio to image transformation I could imagine a circular approach with a fixed length per circle, maybe 15 seconds? Or shorter? I guess 15s could work nicely, you could compare even 1 minute audio files, which would then contain 4 rounds in the circle, starting from the center going outwards the longer samples get, so images would also have different sizes. The longer the audio the bigger the radius, obviously, but the central comparison area would still be same/overlapping, so different size files can still be compared! And circular shapes look often beautiful because symmetric.
TonE is offline   Reply With Quote
Old Yesterday, 01:10 AM   #27
dimentorium
Human being with feelings
 
Join Date: Jan 2020
Posts: 12
Default

Hi TonE,
interesting tool the HCE, did not know that one.I would guess its using comparable algorithms that I am using.
So overall I would kindly ask to not start a discussion which tool to use or is better, but te keep the discussion which use case is interesting or useful. That will help me to focus the few time i have to develop useful stuff.

I tried several tools and code repositories and ended in the setup I have which is for my workflow the best I have. Also I tried to read papers and publications from people where I believe they are way smarter than I and use what is available to not redevelop the wheel.
dimentorium is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 07:11 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.