Antoine Villeret

i've found a pretty strange behavior in jit.op

when i tried to multiply a 1 plane char matrix by 1, it outputs only zeros, whereas the same works fine on float32 matrix

i've found a pretty strange behavior in jit.op
when i tried to multiply a 1 plane char matrix by 1, it outputs only zeros, whereas the same works fine on float32 matrix

jit-op-with-char

remember char data goes from 0 to 255, so to get the results you expect your operand should be 255, not 1.

however this still doesn't quite give the same behavior as multiplying by 1 (they are all 1 less than the original values). i assume the char data is converted to floating point and back, and the decimal points are truncated.

if you intend to do decimal based math operations on data, probably best to used either type long or float32 and convert them to char after the jit.op (which is as simple as sending the output of jit.op into a char matrix of same dim and planecount).

however this still doesn't quite give the same behavior as multiplying by 1 (they are all 1 less than the original values).  i assume the char data is converted to floating point and back, and the decimal points are truncated.

Can someone help me? I am trying to get my head around this. Here is what I understand:

When using [jit.op] on char matrices the matrix cell values are internally normalized (converted to 0.-1. float) before the calculation  and later mapped again to char values for output. 

When using a scalar value for the operant (e.g. [jit.op @op * @val 0.5]) the value is expected to be normalized "outside" [jit.op] --> makes sense as [jit.op]  cannot know beforehand what type of matrix it will receive.

In this process there are rounding errors and a limitation of precision due to the conversion as  mentioned in the ref page.  

When using an integer 1 as  value for  [jit.op @op * @val 1] on a char matrix it fails (result is aways 0)-> Is this a bug because ints work on long matrices?

Still internally the "rules" of the incoming matrix formats are applied during calculation. Example: Given a char matrix with a cell value of 100 it cannot be multiplied by [jit.op @op * @val 2.] to reach a cell value of 200, because 2 does not exists in 8-bit char, hence it is converted to 1 (normalized), which due to the conversion will result in a 0.996 according to the ref pages.

Here is an example patch to illustrate:



jit.op * with char ?