jit.gen, norm, sample: explain this behavior?
Hello,
I am finding some very interesting behavior in the jit.gen [norm] and [sample] interactions and I've spent way too many hours trying to understand and recreate it. This primarily revolves around adding integers to the [norm] coordinates before feeding them to [sample] which produces weird wrapping artifacts. In every other shader environment I've worked in this wouldn't have any issue, the integer component would be cleanly discarded and the pixels at x = 0.1, 1.1, 23.1, ... n+0.1 would all give the same result. Not so in Jit.gen!!
Attached is a simple test harness showing this behavior (see cellblock at bottom inside patch).
My understanding thus far: [norm] gives the pixel coordinate in range [0,1] (i.e. same as cell/(dim-1)). Taking [norm]*(dim-1)/dim gives the upper left coordinate in [sample] space. Feeding [norm] directly to [sample] causes the last pixel (x=1) to wrap back to (x=0). Calculating the pixel centers ([norm]*(dim-1)+0.5)/dim (top cellblock in patch) does not reproduce the behavior from [cell] and [samplepix] (see second cellblock in patch), but appears to retain the coordinates in the output. What have I done wrong here?
Manually sampling through the matrix (using the float box at the top of the patch) shows discontinuous results. As I sample across I get some sort of interpolation but then a sudden jump to the next pixel value.
When I sample beyond the dimensions of the matrix (i.e. wrapping should be occurring), it gets even weirder (bottom cellblock)! But it's also very cool and the artifacts in actual image processing are useful.
I don't want this behavior to be "fixed", I want to know what's going on inside [sample] that is causing this! Can anyone surmise or explain? Would we be able to see the math inside [norm] and [sample]?
Thanks!
This has been raised a few times before in the forums. It's a little messy, for a few reasons -- relating to common conventions of normalized coordinates with respect to CPU vs. GPU processing, pixel centers, border/bound modes, and so on. It seems straightforward but once you look at all the variances it gets pretty messy. The defaults used by sample() in jit.gen and jit.gl.pix might not be what you expect.
In jit.gen, if you want predicatable sampling, I strongly recommend against using sample(). Use samplepix() instead. The behavior of sample() really is a bit odd. For a cell index N, you want to get the pixel center as N+0.5, so samplepix(in1, N+0.5) works nicely. For example, samplepix(in1, cell()+0.5) is exactly equivalent to in1. To get this from a normalized index, samplepix(in1, 0.5+norm()*(dim()-1)) does it.
In jit.gl.pix, which runs on the GPU, the conventions are totally different, and there is no samplepix operator, but there good thing is that sample(in1, norm()) is equivalent to in1; or if you want to sample with cell indices you can do it with sample(in1, (N+0.5)/dim). Meanwhile, the output of cell() is a bit odd (complicated reason why), but you can get it to work with sample(in1, cell()/(dim()-1)).
BTW To know what is going on inside [sample] you can always send the message exportcode or full_source_code the jit.gen or jit.gl.pix, though it might not be super easy to read.
Thanks both for the concise question and answer. I was scratching my head for hours over this one