EEL2 - string and table handling demystified

[papagirafe]

Hi folks!

As you most probably know, coding video preset is done in the EEL2. This is a powerfull programming language, quite easy to learn but characters strings handling is a total confusion. I am provinding here a piece of code to help understand the inner workings of strings. But first a bit of context:

EEL documentation

To add to the documentation:

Apart from regular variables known as "named spaces", EEL has 3 sets of globally shared indexed memory slots (aka tables) for your convenience:
1) the main one that contains about 8 millions slots to store numbers
that are adressable through this notation: base[index]
'base' and 'index' are regular name space variables.
so the memory slot number is 'base + index'
2) the string table that contains 1024 entries. You address a string slot by
assigning a value to a regular variable, ex: 'a=18' and strcpy() a value in it
3) gmem table: similar to #1 but shared among video presets and fx

//String and table operations demystified
// by papagirafe

// mutable strings operations
a=5;      // arbitrarily set string slot index to 5
strcpy(a,"memory slot #5");   //copy some stuff in there
strcpy(a+1,"memory slot #6 (=5+1)");
z=10;     //set base to 10th element in the main indexed mem space 
z[0]=5;   //store a reference to 'a' in main indexed address space
#z=z[0];  //copy string slot referenced by z[0] (i.e. 5) to named string #z
z[1]=a+1; // z[0+1] = next string slot = 6
#z1=z[1]; // confirmed! == content of slot a+1 
zz=z-5;     //let's try an index base lowered by 5  
#z2=zz[5];  //zz[5] == z[0] == string slot #5 = slot(a)
strcat(z[0]," confirmed");  //append string to string slot #5 through z[0]
#z0=a;                      //confirmed! we really modified slot refered by 'a'

//litteral number = direct access to string slot
#ms1000=ms1000=strcpy(1000,"memory slot #1000"); 
//but there is a maximum 1024 string slots, this operation fails: (empty string)
#ms2000=ms2000=strcpy(2000,"memory slot #2000, maximum = 1024");   
 
// immutable string operations
im="immutable";         //if initialized this way, string is immutable.
strcat(im,"  really???");
#im=im;                 //really immutable
#x5=x5=#x[5]="x5";      //named strings have independent index slots but become immutable
#check_a=a;             //indeed slot #5 refered by a is untouched
strcat(#x[5]," is mutable?");
#im2=#x[5];             // really not mutable
#x2000=x2000=#x[2000]="x2000";  //x[n] is not limited to 1024 slots
#str=str=str[5]="abcd";         //set a new immutable string with same index number as 'a' and 'x' 
#x5_is_untouched=x5_is_untouched=#x[5];
#a_is_untouched=a_is_untouched=a;

[ashcat_lt]

If you explicitly declare a gmem space, it can be separate from other gmems, and also has more slots. I have developed a few functions for array/table management which makes working with the memory buffers (local and gmem) a lot easier especially when trying to navigate and manipulate multiple multidimensional arrays. Other people have done very similar things, but their ways don’t really make sense to me for whatever reason, so I built my own. It’s not exactly exhaustive, but covers most basic functions and some specialty things which came up in my experiments.

[papagirafe]

Originally Posted by ashcat_lt

If you explicitly declare a gmem space, it can be separate from other gmems, and also has more slots. I have developed a few functions for array/table management which makes working with the memory buffers (local and gmem) a lot easier especially when trying to navigate and manipulate multiple multidimensional arrays. Other people have done very similar things, but their ways don’t really make sense to me for whatever reason, so I built my own. It’s not exactly exhaustive, but covers most basic functions and some specialty things which came up in my experiments.

Very interesting stuff you are describing there. So far, I only built a few test functions around arrays (like a range allocator) but I found alternative to gmem[] that allows multiple shared buffers in one script (albeit in read only mode).

[Justin]

1) EEL2 memory is a flat space of memory, with millions of entries (the exact size can be queried by __memtop()). You can access memory by using the [] operators. So you can do:

a = 1;
b = 2;
a[] = 100; // write 100 to memory offset 1
a[b] = 100; // write 100 to memory offset 3 (1+2)
b[a] = 100; // write 100 to memory offset 3 (2+1)
(a+b)[] = 100; // write 100 to memory offset 3 (1+2), these are all the same :)

There are not tables. You can emulate table syntax by setting up regions of memory as an array, but you need to make sure it doesn't overlap with any other memory and

2) Functions that work with strings use numbers to identify the string, which you can either pass directly or store in a variable. Examples:

a = 1; // use string index 1, 0..1023 are available for general use
a = #; // generate a unique string index at compile-time (this will always be >=1024). If you have multiple # statements in code they will all get different unique numbers.
a = "whatever"; // generate a (read-only) string index for "whatever". If you have multiple instances of "whatever" in your code, you may or may not end up with different numbers representing them.
a = #name; // generate a unique string index that is associated with "#name" at compile time. If you have multiple #name instances they will all get the same number.

...
strcpy(a,"foo"); // copy "foo" to whatever a points to

That's it. If you read/write these numbers from/to normal EEL2 memory, they work just the same (they reference the string).
Oh one more thing, there's a shorthand syntax (only applies to #strings):

#foo = anything; // shorthand for strcpy(#foo, anything);

[ashcat_lt]

The memory stuff trips a lot of people up, but it really is that simple. There is one big on dimensional array and we have to “manually” portion that out to hold however many arrays of however many dimensions as we want. We have to find ways to fold our multidimensional sets down to that one long string of slots and keep track of where they are.

There are different strategies for achieving that. If you analyze a few different JS delay plugins you’ll see at least a couple different ways. It’s actually not that hard but it does take some planning and housekeeping. And a lot of redundant code, and…

So I made a couple functions to make it a little cleaner and easier for my old object-oriented database driven brain to work with. I use namespaces and pseudoobjects to define my table/array and help find specific data indexes without having to type the same rather awkward math statement over and over. I’ve been meaning to post that work, but have plenty of lame excuses.

Namespaces of course can also be global, and even shared between JS and VP and I’d imagine even scripts, though I don’t much mess with that side. But I think it really is best practice when using gmem, to use your own explicitly declared gmem space just to keep things from “leaking” and causing kind of invisible or at least opaque conflicts. Even then, though, we need to be careful. The video peeker plugin and VP presets which implement it use their own gmem space, but if you want to have a second set, you have to give them their own new space. That’s easy enough in VP, but on the JS side, you have to actually make a new copy of the plugin which starts to get awkward and is why I made the 64 channel version work.


I haven’t worked a lot with strings. I wanted them to let me dynamically access namespaces and/or functions, but that totally doesn’t work. Otherwise I haven’t much seen the need until just recently when JS got that thing with the debug messages in the description line which makes…well, debugging…so much easier.

Edit - Well I’m gonna keep babbling for a while on somewhat tangential subjects.

My video delay presets don’t really use this memory array. In fact, it kind of goes the opposite direction. I’ve got a one dimensional array of frames that I am tiling into a grid on a much larger frame. For long delay times, we end up needing more than one of those buffer frames, so I end up projecting a one dimensional array onto a three dimensional space.

Of course, the individual frame itself IS a multidimensional array. In fact, it’s kind of a two dimensional array of four element arrays, and if you’ve got data which can fit in 16 bit boxes, you really can use it as such. I have never really wanted this, but it is there, and one begins to wonder if some of our gfx functions could help us with certain table processing functions. Like, can an additive blit is a lot more elegant than looping through two tables just to add their values. And what kind of whacky stuff could we do with xformblit. Isn’t there one where we can literally pass in code as a string???

But actually I kind of lied. I did do a thing which translates “tables” to pixels and back, and even used that to pass video from JS to VP, which…. We’ve talked about this before and I honestly haven’t gone any further on it, but there is the issue of synchronization between asynched threads. I’m think that if we take a clue from the video peeker - it writes a buffer of samples and a sort of “current time” index to gmem - and combine it with the oversized offscreen “buffer” frame from my video delay…

IDK for sure how that helps except that there is a frustrating disconnect between the gfx functions in JS and VP. (Why in the TF can’t I draw a circle in VP?!? F even a line might be nice once in a while) Also I suppose JS has a more direct connection to the audio stream. And if we could find a relatively easy way to adapt a JS plugin so that its display was available for our video projects…

[papagirafe]

Originally Posted by Justin

1) EEL2 memory is a flat space of memory, with millions of entries (the exact size can be queried by __memtop()). You can access memory by using the [] operators. So you can do:

a = 1;
b = 2;
a[] = 100; // write 100 to memory offset 1
a[b] = 100; // write 100 to memory offset 3 (1+2)
b[a] = 100; // write 100 to memory offset 3 (2+1)
(a+b)[] = 100; // write 100 to memory offset 3 (1+2), these are all the same :)

There are not tables. You can emulate table syntax by setting up regions of memory as an array, but you need to make sure it doesn't overlap with any other memory and

2) Functions that work with strings use numbers to identify the string, which you can either pass directly or store in a variable. Examples:

a = 1; // use string index 1, 0..1023 are available for general use
a = #; // generate a unique string index at compile-time (this will always be >=1024). If you have multiple # statements in code they will all get different unique numbers.
a = "whatever"; // generate a (read-only) string index for "whatever". If you have multiple instances of "whatever" in your code, you may or may not end up with different numbers representing them.
a = #name; // generate a unique string index that is associated with "#name" at compile time. If you have multiple #name instances they will all get the same number.

...
strcpy(a,"foo"); // copy "foo" to whatever a points to

That's it. If you read/write these numbers from/to normal EEL2 memory, they work just the same (they reference the string).
Oh one more thing, there's a shorthand syntax (only applies to #strings):

#foo = anything; // shorthand for strcpy(#foo, anything);

Thanks for the extra clarifications Justin. It is true that I abused the word "table" in there but for most programmers an indexed memory space or a one dimensional table are pretty much the same concept unfortunately.

[Justin]

Originally Posted by ashcat_lt

IDK for sure how that helps except that there is a frustrating disconnect between the gfx functions in JS and VP. (Why in the TF can’t I draw a circle in VP?!? F even a line might be nice once in a while) Also I suppose JS has a more direct connection to the audio stream. And if we could find a relatively easy way to adapt a JS plugin so that its display was available for our video projects…

Thing is, the yv12/yuy2 colorspaces are subsampled in their U/V, so things like lines and circles are a mess to do well... so we've punted on that and made scripts do it themselves

[papagirafe]

Originally Posted by Justin

Thing is, the yv12/yuy2 colorspaces are subsampled in their U/V, so things like lines and circles are a mess to do well... so we've punted on that and made scripts do it themselves

At some point I considered creating a vector font with a few characters representing basic geometric shapes (like circle) ...

[ashcat_lt]

My “easy” solution for circles is to just draw one huge circle off screen at init (which of course we have to fake in VP) and then use blit tricks to get them where I actually want them. I have what I think is a decent “line between” function that I was working on, too.

[papagirafe]

Originally Posted by ashcat_lt

My “easy” solution for circles is to just draw one huge circle off screen at init (which of course we have to fake in VP) and then use blit tricks to get them where I actually want them. I have what I think is a decent “line between” function that I was working on, too.

clever!