jit.matrix from list; parsing raw udp data
I'm trying to get raw UDP data from a tiny chip/camera (Xiao Seeed Studio ESP32 S3), and parse it in max as a single grayscale snapshot. I have the data coming into Max through sadam.udpReceiver, but I'm not sure how to put it into a jit.matrix and then to a pwindow.
Context: The chip camera is doing motion tracking of a dancer, and I have most of that coded through the Arduino IDE and sending parsed data to max. However, I also need to take a snapshot of what the camera is seeing, so it's positioned correctly. As it works now, Max sends the message "snap", and I get a bunch of (theoretically) grayscale numbers back into Max. That's working fine : well, at the very least I get numbers.
More context: I've tried workarounds, like setting up a web server on the ESP32 chip, but it doesn't like to multitask or doesn't like moving between grayscale and JPEG. It gets laggy, too.
I've attached a screenshot of the max patch after one "snap", as well as the Arduino sketch. Thanks to this great forum!
What the ESP32 is sending:
// ===================== // SNAPSHOT (GRAYSCALE BUFFER OVER UDP) // ===================== void sendSnapshot() { camera_fb_t *fb = esp_camera_fb_get(); if (!fb) { return; } uint8_t small[20 * 15]; for (int y = 0; y < 15; y++) { for (int x = 0; x < 20; x++) { small[y * 20 + x] = fb->buf[(y * 4) * 80 + (x * 4)]; } } udp.beginPacket(host, snapshotPort); udp.write(small, 20 * 15); // exactly 300 bytes udp.endPacket(); esp_camera_fb_return(fb); }
I should add I tried also a Node.script at one point, which I'm not good at. Below, Any other approaches? Thank you!
udp_receiver.js:
const dgram = require('dgram');
const maxAPI = require('max-api');
const PORT = 9002;
const socket = dgram.createSocket('udp4');
socket.on('message', (msg) => {
const values = Array.from(msg);
maxAPI.post(`packet size: ${values.length}`);
maxAPI.post(`packet size: ${values.length}`);
maxAPI.outlet(values);
});
socket.on('listening', () => {
const address = socket.address();
maxAPI.post(`UDP listening on port ${address.port}`);
});
socket.bind(PORT);
See [jit.fill] help file.
It's not enough to connect it to [jit.matrix]. [jit.fill] output is just a bang.
You have to give a name to the matrix and reference it and the plane to write to.
Example: [jit.fill buffer 0] -> [jit.matrix 1 char 16 16]
Thank you so much, Pedro!--that worked; I should have better understood that object. Now I am having an issue that it only displays the top left quadrant of what (I think) the camera is actually seeing, given previous tests. Other searches are suggesting this has something to do with the 256 length of what is being sent max (I get the same "quartered" results with sadam.udpReceiver as well as a node.script version).... thoughts? and again, thanks so much.
Hi again. A few suggestions:
zl objects default to a limit of 256 list elements. In order to process larger lists write something like [zl.len @zlmaxsize 4096]. That way, you get the real length of the list.
[jit.matrix] should be configured to the actual number of cells as the pixels of the image being described. What is the resolution of the camera? 40 by 30 as your jit.matrix implies?
Edit: I just saw you ESP32 code above and it looks like the resolution is 20 by 15, which gives you 300 pixels/list elements. Is that why you changed to 40 30 on the second screenshot?
Thanks once again Pedro--this was very helpful. It turns out that I misfigured the dimensions of the Seeed S3 camera, and that was baked into my ESP32 code. I changed the code, and also used node.script to get a better resolution image. Thank you for all your help!! Nick
final version:
