Shakeeb Alireza

In the case of gen patches which have a number of repeating patterns or a repeating network structure (like reverbs), it looks like it's easier to write the final version in gen code.

To test this, I converted the gen freeverb example that comes with Max to gen code. Does this look about right?

live.gain~

ezdac~

bpatcher

demosound

number

message

comment

flonum

gen~

codebox

There must be some recurring best practices to make such conversions (of large patches). How about examples of canonical conversions 'done right'? Or examples of open-source GenExpr libraries which contain generic composable functions that can be used in such patches?

In the case of gen patches which have a  number of repeating patterns or a repeating network structure (like reverbs), it looks like it's  easier to write the final version in gen code. 

To test this, I converted  the gen  freeverb example that comes with Max to gen code. Does this look about right?

There must be some recurring best practices to make such conversions (of large patches). How about examples of canonical conversions 'done right'? Or examples of open-source  GenExpr libraries which contain generic composable functions that can be used in such patches?

In the case of gen patches which have a  number of repeating patterns or a repeating network structure (like reverbs), it looks like it's  e

best-practices-converting-large-gen-patches-to-genexpr-codebox-code

Following from my recent post, I converted a schroeder reverb from a gen (patch) example on youtube into a pure codebox implementation.

Posting it here in case any one has feedback, corrections or comments:

sounding good. and the code looks great(succinct and organized, nice one!).

i like to try things out in actual context by playing music with it, and with gen~ stuff, changing params as fast as possible can often help detect any clicks, but as I shifted the size very quickly and sent sound in interrupted bursts through it, it remained smooth, so the sound feels nice and useable as well. here's a piece i made with it last night, continuously changing the size and such, i'd share the max patch, but it has some stuff i'm still working on for a rhythmic click generator not ready to be shared yet:

While I expect this is enjoyable work for the command-line codeophiles in our midst, there's no incentive in terms of efficiency to port stuff over. The 

object compiles the contents of patcher into GenExpr for operators and codebox contents alike, with the object producing the target code necessary to perform its calculations, and the 

 object generates and compiles native CPU machine code on-the-fly.. Everything is merged into a 

There are a few instances in which you need a codebox (there's a summary of those instances 

). Apart from that, work with what you know and are comfortable with. If you've already got an operator version of something that works, there's no advantage to codeboxing it, apart from personal preference. However, if the intellectual exercise excites you, have at it! Me? I'd rather patch 

Nice work, Raja.

While I expect this is enjoyable work for the command-line codeophiles in our midst, there's no incentive in terms of efficiency to port stuff over. The gen~ object compiles the contents of patcher into GenExpr for operators and codebox contents alike, with the object producing the target code necessary to perform its calculations, and the gen~ object generates and compiles native CPU machine code on-the-fly.. Everything is merged into a single representation.  

There are a few instances in which you need a codebox (there's a summary of those instances here). Apart from that, work with what you know and are comfortable with. If you've already got an operator version of something that works, there's no advantage to codeboxing it, apart from personal preference. However, if the intellectual exercise excites you, have at it! Me? I'd rather patch new gen~ stuff.

That's awesome, @Raja. Thanks for your feedback on the code and for using it in your patch (-:

@Gregory, you are absolutely right that there is no 'efficiency' advantage between a gen patch, its related automatically generated code, and a manual rewrite with functions as above.

For me at least, I find that the latter helps me to better understand the patch and its structure (especially if it's using well-researched elements), and the use of functions makes code shorter, more readable, more modular and ultimately quicker to extend which are all concrete benefits if one wants to ultimately create a library of composable functions to build such things.

@Gregory, you are absolutely right that there is no 'efficiency' advantage between a gen patch, its related automatically generated code, and a manual rewrite with functions as above. 

For me at least, I find that the latter helps me to better understand the patch and its structure  (especially if it's using well-researched elements), and the use of functions makes  code shorter, more readable, more modular and ultimately quicker to extend which are all concrete benefits if one wants to ultimately create a library of composable functions to build such things.

my pleasure, @Shakeeb, and Thank You for this patch... i found when extending to MC, i did have to put some smoothing over the size param(i just changed it to "schroeder(in1,size = slide(size, 2000,2000));" and that seems to do enough for me... but i also tried my own function involving a history object with smoothstep/mix in that same line instead of 'slide' and for whatever reason, i don't think gen~ likes certain history-based functions to alter the attribute/param of a function like that(?)... those attempts failed to compile and 'slide' was the only thing that worked there.. just noting an interesting find here..)...

here's a modified MC version of the original patch with an interesting MC-based effect, i set the live.gain object here with 'initial enable' at -40db so it isn't too loud(at 12 channels of MC), just make sure you see live.gain~ at that level (or set it below) before starting/hitting-play-on the playlist~ running the epno sample:

my pleasure, @Shakeeb, and Thank You for this patch... i found when extending to MC, i did have to put some smoothing over the size param(i just changed it to "schroeder(in1,size = slide(size, 2000,2000));" and that seems to do enough for me... but i also tried my own function involving a history object with smoothstep/mix in that same line instead of 'slide' and for whatever reason, i don't think gen~ likes certain history-based functions to alter the attribute/param of a function like that(?)... those attempts failed to compile and 'slide' was the only thing that worked there.. just noting an interesting find here..)...
here's a modified MC version of the original patch with an interesting MC-based effect, i set the live.gain object here with 'initial enable' at -40db so it isn't too loud(at 12 channels of MC), just make sure you see live.gain~ at that level (or set it below) before starting/hitting-play-on the playlist~ running the epno sample:

That's a sweet iteration @Raja. If I understand it correctly, the slow lfo modulates the size of each mc 'reverb-channel' over time to modulate the overall reverb space.

That pretty cool. Thanks for sharing. (-:

ya exactly, i wasn't too exact with the 'scaledexponential' settings, but i really enjoy, with the MC 'space'/interval ops, you can set the channels with spaced intervals that all give different effects, if you want to scope you can send a space-specific message to mc.sig~ like i do below, and these other ones i show in the pic below are also interesting effects with your schroeder-reverb's 'size' param:

Screen Shot 2020-12-19 at 3.53.55 PM.png

the slow lfo modulates the size of each mc 'reverb-channel'

ya exactly, i wasn't too exact with the 'scaledexponential' settings, but i really enjoy, with the MC 'space'/interval ops, you can set  the channels with spaced intervals that all give different effects, if you want to scope you can send a space-specific message to mc.sig~ like i do below, and these other ones i show in the pic below are also interesting effects with your schroeder-reverb's 'size' param:


Very nice @Raja. The mc part of max is totally new to me, another dimension so to speak.

I ran with your mc 'space messages' a little bit. Seriously cool (-:

I ran with your mc 'space messages'  a little bit.  Seriously cool  (-:

nice! i was actually wondering how to sculpt a drum sound from this reverb... but was only going for the kick(i heard it most prevalent when sweeping from negative to positive on the 'scaledexponential' message... i think in the negative range it suddenly causes the last channels to shoot up in size passed 100), 

nice! i was actually wondering how to sculpt a drum sound from this reverb... but was only going for the kick(i heard it most prevalent when sweeping from negative to positive on the 'scaledexponential' message... i think in the negative range it suddenly causes the last channels to shoot up in size passed 100), 
you managed to find kick, snare, and hihat! 😄

Looks like I spoke too soon about how easy it is to convert a large gen patch and its generated code into readable code with functions.

I am failing completely to do the same thing to 'creepyverb' (in the gen examples). It does not refactor (convert to functions) well, even incrementally. Probably something to do with History and Delay scope limitations when they are placed in functions.

 thread describes the difficulty I am facing:

I wonder if this is a limitation of the current implementation of the language. It would be good to know in any case.

 Looks like I spoke too soon about how easy it is to convert a large gen patch and its generated code into readable code with functions. 

A quote from TESTCASE in another thread describes the difficulty I am facing:

"for this to work, gen would need to 'know' that you have a delay in the gen subpatch and I don't think it does that kind of analysis. If you tried to do this in a codebox you could see the problem. If you had a function delay1 and another delay2 you would not be able to write to the delay lines in the the functions so you would need a history object. Anyway, I would suggest you need to organize this differently."

I wonder if this is a limitation of the current implementation of the language. It would be good to know in any case.

S

ya, it can be a bit tricky... (i think function-declarations don't allow that because history/delay/data/buffer actually have to create a space in hardware memory, not just a pointer that can be changed easily), you have to think around accessing memory from outside the function(can only pass the 'result' of a read/write op into the function(using arguments)).

Looks like I have to rethink this carefully.

I'm probably misunderstanding the implications of all of this: but it looks like I have to declare all of the History, Delay, Data, Buffer objects up front (let's say statically), instead of dynamically in functions, and then refactor at the edges of reads/write to these objects, which makes functions in gen-land 'second-class' citizens: they can contain History, Delay, etc.. objects, but can't provide access to them in a workable way, and of course they can't pass other functions to each other as parameters.

Looks like I have to rethink this carefully. 

their access is 'local' to the function only(but i think i was confused about your specific prob, you can ignore what i wrote above - was trying to wrap my head around the idea that one function with a reserved space for memory that's local to that function will not be able to pass that access directly...).

it sounds different when i rewrite your first function making sure to assign noise() to 'noise1' and 'noise2', too... just noticing something extra... not sure what it is though, haha:

    tap1 = delay.read(latch2 * 100, interp="none");

    delay.write((diff * latch1) + sig);

but can't provide access to them in a workable way

their access is 'local' to the function only(but i think i was confused about your specific prob, you can ignore what i wrote above - was trying to wrap my head around the idea that one function with a reserved space for memory that's local to that function will not be able to pass that access directly...).

it sounds different when i rewrite your first function making sure to assign noise() to 'noise1' and 'noise2', too... just noticing something extra... not sure what it is though, haha:
allpass(sig, control) {
	// may have to remove history
	Delay delay(100);
	noise1 = noise();
	noise2 = noise();
	latch1 = latch(noise1, control);
	latch2 = latch(noise2, control);
	tap1 = delay.read(latch2 * 100, interp="none");
	diff = tap1 - (sig * latch1);
	output = diff;
	delay.write((diff * latch1) + sig);
	return output;
}

^also, looking closer now, feel like this might be a typo? 

(you're assigning '1' to history_next there everytime(?))

"history_next = fixdenorm(int(1));" 
^also, looking closer now, feel like this might be a typo? 
(you're assigning '1' to history_next there everytime(?))

Thanks for pointing out the typos. I fixed them but the quality of the reverb is still poor. I took out the 1st allpass gen subpatch which has a very regular pattern and yet defies refactoring to illustrate the issue:

If you look at the code in the subpatch's codebox, order is significant, especially the delay.writes at the end, and yet this get's lost when I tried to convert these into functions:

Thanks for pointing out the typos. I fixed them but the quality of the reverb is still poor. I took out the 1st allpass gen subpatch which has a very regular pattern and yet  defies refactoring to illustrate the issue:

Having spent a bit of time more reading large well-structured gen codebox code (especially in commercial max 4 devices). It's clear that the assumptions and reservations expressed above need revisiting.

I've seen working examples of gen functions with History and Delays inside (in many cases filters) working without issue provided reads and writes were inside the function.

Another thing I did observe was that in all cases the core algorithm was not encapsulated in a main function and indeed the codebox itself was treated as the 'main' function. I don't want to read too much into this, but it's interesting nonetheless.

One thing is clear, though, the official docs should include some additional guidance on the degree to which encapsulation works or doesn't work in gen.

 I've seen working examples of gen functions with History and Delays inside (in many cases filters) working without issue provided reads and writes were inside the function. 

ya, sorry, this was what i was trying to say when mentioning to ignore what i wrote further above, i was confused about your issue and didn't quite check on what i was referring to properly. apologies!

It's clear that the assumptions and reservations expressed above need revisiting...
I've seen working examples of gen functions with History and Delays inside (in many cases filters) working without issue provided reads and writes were inside the function. 

No worries, Raja, you've been very helpful. In fact your comment above made me do some more research on the forum on this subject, which I believe is quite helpful in general.

I think we have been trying to explain an aspect of GenExpr code which Graham Wakefield (the designer of Gen) refers to as 'lexical instancing' -- (He expands on this concept in the forum three times, just search for 'lexical instancing'). I'll quote them partially here because they kind of make more sense together: 

 post, responding to the question "Please can you help me understand how functions work in FOR loops?"

As a direct response to this post, LARME makes the following statement:

To which Graham says "Correct". So basically one needs to make strict distinction between 'stateful' functions which have to be treated specially because of 'lexical instancing' and 'pure' or stateless functions.

 post, Graham provides an example of a 'stateful' function which creates its own state in each call:

I wasn't aware that one can pass history objects as arguments.. Good know (-:

 and most recent post on 'lexical instancing', Graham gives some practical method on how to pass state between functions, which I found quite informative.

Much clearer now. Now back to the drawing board.

No worries, Raja, you've  been very helpful. In fact your comment above made me do some more research on the forum on this subject, which I believe is quite helpful in general.

I think we have been trying to explain an aspect of GenExpr code which Graham Wakefield (the designer of Gen)  refers to as 'lexical instancing' -- (He expands on this concept in the forum three times, just search for 'lexical instancing'). I'll quote them partially here because they kind of make more sense together: 

In this post, responding to the question "Please can you help me understand  how functions work in FOR loops?"

It's "lexical instancing". That is, if you defined a function "foo", then each instance of foo() in your code implies an instance of any state foo requires (including History, latch, sah, phasor, accum, etc.). So calling foo(foo(foo(foo(x)))) creates four independent instances, but below [in a for loop] is only one instance, because the text "foo" appears only once.

Yes you can use functions in for loops, but you have to bear in mind the 'lexical instancing' issue regarding state. That is, a function that declares its own state (like History) or which uses stateful operators (like cycle, phasor, etc.) will share that state over each iteration of the for loop.  Which might or might not be what you wanted.

It's a little bit tricky because a function may not use History or phasor explicitly but instead call functions that do so.  Hence you need to make sure all functions in the chain of invocation is "clean" of state, am I right?

In this post, Graham provides an example of a 'stateful' function which creates its own state in each call:

I believe the equivalent genexpr is the following. This compiles and appears to have approximately the correct logic.

ffjk4027(clock) {
	History state;
	History buf;
	
	if (buf < 0 && clock >= 2.5) {
		state = !state;
	}
	
	buf = clock;

	if (state) {
		return 4;
	} else {
		return -4;
	}
}

(One caveat however is that the use of static variables in functions like the above code is that this variable will be shared between *all* calls of the function, which is not the case in Gen. Every lexical instance of the call to ffjk4027() in gen will create a new state/buf to go with it. If this isn't what you need, pass the History objects in as arguments to a function instead. But I'm guessing that the postfix "4027" means that this code was generated for single call, which means you don't have to worry.)

In the final and most recent post on 'lexical instancing',  Graham gives some practical method on how to pass state between functions, which I found quite informative.

a=biz(0,blam=4); // returns 4
b=biz(0); //returns 0 even after calling above line

...catches one of the  subtle things about genexpr, which is that operations are 'lexically instanced'. 
I think it might be easier to consider from the perspective of cycle. Roughly speaking cycle could be implemented like this:

mycycle(freq) {
  History phase;
  phase = wrap(phase + freq/samplerate, 0, 1);
  return cos(twopi * phase);
}

Now, if you want to make a chord of sinewaves, you can write this:

Both of these phasors have internal state (the History phase), each of which is independent of each other. Behind the scenes, each time you write `mycycle()` you are creating a `History phase` for that instance, which is retained over time -- it is "stateful" in a way that e.g. `add()` isn't.  

Your `biz()` function is also stateful in this way, because of the `Param blam`. So each time you write `biz()` in the code, you create a new instance of the `Param blam` associated with it. This `Param blam` remembers its value for each invocation of its particular `biz()` over time.  But if you write `biz()` twice, you get two `Param blams`, one each. 

If we didn't do it that way, then `mycycle(100) + mycycle(150)` would not create a chord, instead it would create one oscillator with a frequency of 250Hz. 

So, the state *inside* a function is "lexically instanced": there's a new indepenent instance of that internal state for each time you *write* a call to the function. 
If you want to share state between separate function calls, you will have to pass that state in to the function from outside. E.g.:

biz(poop, blam) {
  return poop + blam;
}
History blam(7);
out1 = biz(30, blam) + biz(30, blam);
Or, this:
biz(poop, blam) {
  return poop + blam;
}
superbiz(poop) {
  Param blam; // this param shared by both biz's:
  return biz(poop, blam) + biz(poop, blam);
}
// we only write `superbiz` once, 
// so there's only one instance of its internal state `Param blam`:
out1 = superbiz(30, blam=7);

Note that you can also pass buffers etc. into functions in this way. 

I think I have mostly figured out what went wrong in my earlier conversions outside of the usual typos: I had used an instance of History which I named history1 in one of my inner functions and then I used the same variable name in the outer function thinking that they would be separate because of the difference of scopes. However, this led to to a bug it would seem that gen and its 'Lexical instancing' does not protect the user in this case.

Here is the demonstration/proof of this 'scope bleed'

1. The working example is a working partial conversion to pure codebox gen code of creepyverb (from the standard gen examples):

2. The above example with a single renaming of the one of the history variables in the outer function (note the pop on opening and the loss of the reverb tail) which indicates that this something has gone wrong:

So clearly one has to be careful in the naming of variables, and I'm pretty sure that this issue will most definitely catch out programmers who expect typical lexical scoping rules.

Given the above issue, I would now agree with Gregory Taylor's earlier view that there is not much advantage to writing all gen code directly. I would now say that one is probably better off patching gen with minimal codebox use in order to avoid pitfalls as the one above.

I think I have mostly figured out what went wrong in my earlier conversions outside of the usual typos: I had used an instance of History which I named history1 in one of my inner functions and then I used the same variable name in the outer function thinking that they would be separate because of the difference of scopes. However, this led to to a bug  it would seem that gen and its 'Lexical instancing' does not protect the user in this case.

$ diff working.GenExpr error.GenExpr
78c78
< History history_16(0);
---
> History history1(0);
115c115
< mix_62 = mix(mul_61, history_16, damping);
---
> mix_62 = mix(mul_61, history1, damping);
168c168
< history_16 = history_16_next_109;
---
> history1 = history_16_next_109;

Best practices converting large gen patches to GenExpr / codebox code?