320x240 data reception via WIFI. Converting data for drawing on a TFT screen. RESOLVED
Hi,
I would like to convert r, g, b datas from a [jit.martrix 3 char 50*30] in order to have one data for each pixel

With that conversion I will have 50*30 datas and not 3*50*30 datas.
How should I do that?
if I you that in Arduino the code is like this
for (uint8_t j = 0; j < (30); j++) {
int r= msg.getInt(j*3+0);
int g= msg.getInt(j*3+1);
int b= msg.getInt(j*3+2);
uint32_t couleur = r << 16 + g << 8 + b;
Serial.print ( " " ); Serial.println ( couleur);
but I prefer to do that from Max to earn RAM.
At the end, the goal is to send data from Max to Arduino in order to assign different to 320*240 pixel screen.
I can only receive 1500 datas at once with OSC message with my esp32.
So I will have to split datas to 52 times to be able to send 1500 datas/frame, beacause 320*240/52 = 1477
I think it is not the good method.
For now I can send data from Max to Arduino in order to assign different color to 8*8 pixel screen, it works very well.
Thank you very much for your help and advices !
before you proceed with zl objects check zlmaxsize attribute which defaults to 256
thanks
I added this
zl.group 1500@zlmaxsize 4500. <-- 1500 pixel with 3 color
But as I said, I am not sure that sending 52*1477 is a good method to assign color to each pixel on, a screen of 240*320 pixels.
But I don't know others methods. so... convert r,g,b to one data will be a first step.
And I think I don't need a so good resolution. may be 60*80 pixel will be enough.
RGB type char easily fits into a 32 or 64 bit number by concatenation:
255255255
to avoid issues with leading zeroes you might want to add a 1 or a symbol at the beginning:
x003255003
1003255003

alternatively, for even shorter data, you could multiply the G values with 255 and the R values with 255*255 before you sum them; this will also create a unique number for each possible 8-bit color.
furthermore you could compress such a dataset by only giving the rgb value for the pixel where it first appeared, and when the same color is required for another pixel, only refer to the first pixel.
Thanks Roman.
But before converting datas (as a simple addition), I have to split each 3 first data from jit.matrix 3 char 50 30 in order to do your process. ..
And I can't do that easily.
And datas I will send, are not such as dataset. So no possible to compress
maybe i am still outside the context, but to get the RGB data from such a small matrix using counters to harvest values using "getcell" seems appropiate.
eventually transforming the matrix into one of dimensions 1500/1 - and then splitting it into 3 (of R, G, B channels using jit.unpack) - would be a good preparation (and more effective than using messages for all of this)

you wrote :
At the end, the goal is to send data from Max to Arduino in order to assign different to 320*240 pixel screen.
If you want to missuse arduino to set rgb for 240X320 pixel display using OSC messages,
it makes 320 x 240 x 3 = 230400 values.
where is that in your patch and that arduino code,
which actually prints to serial port ?
if you want to paint pixel by pixel, than closest go would be to use
240 rows containing 320 x 3 (960) rgb values and see how fast you can paint.
and that has to be parsed so in arduino code,
so that you can address pixels properly.
Hi ! Thanks for your help.
Here how I send 5 lines of 320 pixels. zl.group 60 on the right of the screenshot was just to check the method

But I'm sure there is a better method to send 240 lines but I can't do that. Can you help please?
Then. I will have to adapt the program you dit for me last time.
#include <WiFi.h>
#include <WiFiUdp.h>
#include "esp_wifi.h"
WiFiUDP Udp;
IPAddress local_IP(10, 10, 10, 10); IPAddress gateway(10, 10, 10, 1); IPAddress subnet(255, 255, 0, 0); // Router
const unsigned int localPort = 8888; // UDP Port
uint8_t buffer[960];
#include <FastLED.h>
#define NUM_LEDS 320 // here LED are 320 pixels on each of 240 row
#define PIN_CONTROLLING_STRIP 21
CRGB leds[NUM_LEDS];
void setup() {
WiFi.mode(WIFI_STA); esp_wifi_set_ps(WIFI_PS_NONE); WiFi.config(local_IP, gateway, subnet);
WiFi.begin("SSID", "12345678"); Udp.begin(localPort); // SSID & password
FastLED.addLeds<NEOPIXEL, PIN_CONTROLLING_STRIP>(leds, NUM_LEDS); FastLED.setBrightness(180);
for (int i = 0; i <= NUM_LEDS; i++) {leds[i] = CRGB(0, 0, 0); // turn off all
FastLED.show();}
}
void loop() {Udp.parsePacket(); if(Udp.read(buffer, 960) > 0) {doit();}
}
void doit(){for (uint8_t i = 0; i < NUM_LEDS; i++)
{leds[i] = CRGB (buffer[i*3+0], buffer[i*3+1], buffer[i*3+2]);}
FastLED.show();
}
I had an answer ready a month ago, by I forgot what was all this about after that long time.
When I get time again I will check it.
I only remember I wanted to convert RGB888 to RGB565
which is used in TFT libraries
here example using hex or dec representation
Here is a patch which sends rows of 320 x 3 RGB values.
speed between sending lines can be adjusted from 0.1 ms to as much as it needs.
each line has row number prepended followed by 960 values.

I really don't understand what you use, a TFT display ?
LED matrix ?
......
in case of TFT, you would use one of the arduino libs, and
send RGB565 instead of RGB888.
Hi ! and thanks again for helping me ;)
Now I have to receive 320*240 datas in a TFT_WIFI module with a 240 MHz CPU.
As I can send only with [sadam.udpSender] 1500 datas at once, I say to myself, I have to send
1200 datas in order to send 4 lines of 320 pixels and do it 60 times in order to have 240 lines of 320 pixels.
As you said to previous post, I convert rgb pixel to one color like below.
But I don't know how many datas I send at once ?

With the TFT_ESP32 module I test a program to change color to each pixel to the TFT screen. All pixels are modified each 1.5 sec.
So will it be possible to change color as fast as datas come from Max/msp ?
void loop()
{
for (int i = 0; i < 320 ; i++) // 320 pixels
{
for (int j= 0; j <240; j++) //240 pixels
{
color = random(0xFFFF);
tft.drawPixel(j, i, color);
}
}
}
sadam udp sends raw bytes only,
you can't send higher then 255.
You should switch to max udpsend object.
qmetro interval sets speed,
easy to calculate - 240 rows * 0.2 = 48 ms .
in theory .
but idea to use qmetro is to avoid too fast counting.
you should try metro and compare to qmetro.
if you go down to 0.1 ms then 24 ms.
but I don't know how fast can ESP32 receive.
test this for maximum send speed
this is updated patch, I think row numbering starts from 0, not 1

On local wifi network I receive this
without any drops from mac to mac.
Hi,
Thanks again
I changed your patch to send only one line of 160 datas by changing [counter 0 1 2 ]
The aim is to see if the ESP32 received well all the 160 or 161 datas.
I tried with 160 because It doesn't work with 320.
I can see in the monitor of Arduino IDE only four datas.
The three first are good and the forth is the the same as the first. And I have nothing after.

my Arduino program is
#define NUM_OF_DATA 160
void loop() {
OSCMessage msg;
size = Udp.parsePacket();
if (size > 0) {
while (size--) {
msg.fill(Udp.read());
}
if (!msg.hasError()) {
msg.dispatch("/m", assignDataParsed);
}
else {
error = msg.getError();
Serial.print("error: ");
Serial.println(error);
}
}
}
void assignDataParsed(OSCMessage &msg) {
for (uint8_t i = 0; i < (NUM_OF_DATA + 1) ; i++) {
row = msg.getInt(0);
pix = msg.getInt((i-1)/3);
color = msg.getInt(pix/1);
Serial.print( "r "); Serial.print (row); Serial.print( " p "); Serial.print (pix);
Serial.print(" "); Serial.print (i); Serial.print(" "); Serial.println (msg.getInt(i));
}
// tft.drawPixel(row, pix, color);
}
The Arduino monitor is
20:50:18.536 -> r 0 p 0 0 0
20:50:18.536 -> r 0 p 0 1 34997
20:50:18.536 -> r 0 p 0 2 30701
20:50:18.572 -> r 0 p 0 3 54227
20:50:18.572 -> r 0 p 34997 4 0
20:50:18.572 -> r 0 p 0 5 0
20:50:18.572 -> r 0 p 0 6 0
20:50:18.572 -> r 0 p 0 7 0
20:50:18.572 -> r 0 p 0 8 0
The monitor of max /msp print 161 datas
print: /m 0 34997 30701 54227 44189 11880 20689 34182 1222 58570 11487 6411 34800 46447 9986 4675 29435 24330 53587 51191 39163 32624 54168 65133 12215 61314 38224 15589 34234 41179 35072 52255 33505 21998 31017 34569 8989 58307 14378 30228 61870 38340 20773 56273 24887 60367 17361 48088 43627 31209 53649 5044 40877 586 48369 58745 4240 2365 27410 7521 10438 23127 9907 2003 18544 4493 34353 39567 52763 48185 51807 38885 37799 46337 10194 17263 13386 57899 21260 63547 62765 46385 25850 35479 36673 9258 8814 2758 44229 16109 1641 61851 1553 50056 7209 24323 37739 10037 46467 40018 7839 38782 47584 1300 57190 38469 53156 60815 27152 28790 6017 63719 18392 27715 49677 49804 59614 44225 52990 34830 55565 56501 38486 20252 34404 44163 5202 64490 22731 53962 20030 12795 40271 17733 51674 57084 45444 36209 38747 22557 35632 28164 13280 49175 36643 60195 10152 20797 9451 31684 25838 34573 41566 17943 20855 48113 8803 60665 51643 63426 40262
you again use OSC instead of raw udp ?
I thought you understood the difference along this so long thread.
Go back and rewrite that.
Hi and thanks for your help again and again.
I understood how to use row, pixel and color.
I made a program to get 160 datas (not anymore 320)
void loop() {
OSCMessage msg;
size = Udp.parsePacket();
if (size > 0) {
while (size--) {
msg.fill(Udp.read());
}
if (!msg.hasError()) {
msg.dispatch("/m", assignDataParsed);
}
}
}
void assignDataParsed(OSCMessage &msg) {
for (uint8_t i = 1; i < (NUM_OF_DATA + 1) ; i++) {
row = msg.getInt(0);
pix = i;
color = msg.getInt(i);
tft.drawPixel(row, pix, color);
}
}
From max/msp I have to send each line of 160 datas with [metro 30] if I want to see 160 * 240 pixels well changed.
It is slow because 35 ms * 240 rows = 8400 ms.
So, I think to send data from Max/Msp in order to reduce resolution as 1/100 ?
I will use a matrix like 80* 48 datas.
Do you know a way to reduce resolution with TFT library in order to transform a 320*240 pixels screen with only 80* 48 datas ?
you should as first test drawing pixels on that tft screen from max.
post arduino code which succesfully paints at least 1 row of that screen.
and post which screen you use, which arduino version , which tft library and so on.
raw udp would be more efficient than OSC which allways sends int32 ,
means 4 bytes per number.
if you use int16 RGB565 it reduces number of bytes needed to be sent.
only work you have to do is to learn how to parse that in arduino. ok?
In my previous post with OSC method, you can see I get pixel from Max and I can draw pixel to 240 row of 160 pixels.
Now I tried to get pixel with raw udp but datas red in Arduinio Ide are not good.
Maybe because I don't use int16 RGB565 in max/msp patch ?

Arduino code
#include <WiFi.h>
#include <WiFiUdp.h>
#include <OSCMessage.h>
#include <OSCBundle.h>
#include <OSCData.h>
#include <WiFiMulti.h>
WiFiMulti wifiMulti; // select wifi automatically
WiFiUDP Udp;
int numberOfdataReceiveAtOnce = 321;
int color;
uint8_t buffer[321]; //320 to test reception
const unsigned int outPort = 9999; // remote port (not needed for receive)
const unsigned int localPort = 8000; // local port to listen for UDP packets (here's where we send the packets)
OSCErrorCode error;
unsigned int ledState = LOW; // LOW means led is *on*. not use
//------------ END ESP and OSC setting
#define LED_BUILTIN 16
#define BUILTIN_LED LED_BUILTIN
#include <TFT_eSPI.h> // Hardware-specific library
#include <SPI.h>
TFT_eSPI tft = TFT_eSPI(); // Invoke custom library
void setup(void) {
tft.init();
Serial.begin(115200);
pinMode(BUILTIN_LED, OUTPUT);
digitalWrite(BUILTIN_LED, ledState); // turn *on* led
delay(10);
// select good wifi
wifiMulti.addAP("iPhone SB", "freeThewifi");
wifiMulti.addAP("Iphone de ben", "cacaprout");
Serial.println("Connecting Wifi...");
IPAddress local_IP(192, 168, 1, 107);// 145
// Set your Gateway IP address
IPAddress gateway(192, 168, 1, 1);
IPAddress subnet(255, 255, 0, 0);
WiFi.mode(WIFI_STA);
if (WiFi.config(local_IP, gateway, subnet) == false) {
Serial.print("echec de config");
}
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
Serial.println("Starting UDP");
Udp.begin(localPort);
Serial.print("Local port: ");
#ifdef ESP32
Serial.print("LocalportESP32: ");
Serial.println(localPort);
#else
Serial.println(Udp.localPort());
#endif
digitalWrite(BUILTIN_LED, HIGH); // turn *HIGH* means turn OFF led
}
void loop()
{
Udp.parsePacket();
if(Udp.read(buffer, numberOfdataReceiveAtOnce) > 0)
{
assignDataParsedToScreen();
}
}
void assignDataParsedToScreen()
{
for (uint8_t j = 1; j < 321; j++)
{
int row = buffer[0];
int pix = j;
int color = buffer[j];
Serial.print (" c "), Serial.print (buffer[j]);
tft.drawPixel(row, pix, color);
}
Serial.println();
}
I receive this datas without stopping
c 255 c 255 c 255 c 255 c 255 c 255 c 255 c 255 c 255 c 255 c 255 c 255 c 255 c 255 c 255 ....
as first remove
int row = buffer[0];
out of for loop
void assignDataParsedToScreen()
{int row = buffer[0];
for (uint8_t j = 1; j < 321; j++)
{int pix = j;
int color = buffer[j];
Serial.print (" c "), Serial.println(color);
tft.drawPixel(row, pix, color);
}
}
Hi!
By sending less data at once (I think less than 255) , I can receive data correctly!
Now I receive only 161 datas at once, not 321 as I wanted but it will be enough.
I give the entire Arduino code to receive 160*240 datas with raw udp.
From Max/Msp I send each line of 161 datas at 3 ms. If I try 2 ms, some lines are not drawn.
Last question please, how to convert int 16 bits to int 8 bits ?

include <WiFi.h>
#include <WiFiUdp.h>
#include <OSCMessage.h>
#include <OSCBundle.h>
#include <OSCData.h>
#include <WiFiMulti.h>
WiFiMulti wifiMulti; // select wifi automatically
WiFiUDP Udp;
const unsigned int outPort = 9999; // remote port (not needed for receive)
const unsigned int localPort = 8000; // local port to listen for UDP packets (here's where we send the packets)
OSCErrorCode error;
unsigned int ledState = LOW; // LOW means led is *on*. not use
//------------ END ESP and OSC setting
#define LED_BUILTIN 16
#define BUILTIN_LED LED_BUILTIN
#include <TFT_eSPI.h> // Hardware-specific library
#include <SPI.h>
TFT_eSPI tft = TFT_eSPI(); // Invoke custom library
int numberOfdataReceiveAtOnce = 161;
uint8_t buffer[161]; //321 is too much. Maye be because it is upper than 255
int row, pix, color;
void setup(void) {
tft.init();
Serial.begin(1000000);
pinMode(BUILTIN_LED, OUTPUT);
digitalWrite(BUILTIN_LED, ledState); // turn *on* led
delay(10);
// choose good wifi network
wifiMulti.addAP("Bbox-E69F0626", "XXXXX");
wifiMulti.addAP("SFR-d078", "XXXXXZ");// sandra
wifiMulti.addAP("SFR_A080", "XXXXX");
Serial.println("Connecting Wifi...");
if(wifiMulti.run() == WL_CONNECTED) {
Serial.println("");
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
}
IPAddress local_IP(192, 168, 1, 107);// 145
// Set your Gateway IP address
IPAddress gateway(192, 168, 1, 1);
IPAddress subnet(255, 255, 0, 0);
WiFi.mode(WIFI_STA);
if (WiFi.config(local_IP, gateway, subnet) == false) {
Serial.print("echec de config");
}
while (WiFi.status() != WL_CONNECTED) {
digitalWrite(BUILTIN_LED, HIGH); // turn *on* led
delay(500);
digitalWrite(BUILTIN_LED, LOW); // turn *on* led
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
Serial.println("Starting UDP");
Udp.begin(localPort);
Serial.print("Local port: ");
#ifdef ESP32
Serial.print("LocalportESP32: ");
Serial.println(localPort);
#else
Serial.println(Udp.localPort());
#endif
digitalWrite(BUILTIN_LED, HIGH); // turn *HIGH* means turn OFF led
}
void loop()
{
Udp.parsePacket();
if(Udp.read(buffer, numberOfdataReceiveAtOnce) > 0)
{
assignDataParsedToScreen();
}
}
void assignDataParsedToScreen()
{
for (uint8_t j = 1; j < numberOfdataReceiveAtOnce + 1; j++)
{
row = buffer[0];
pix = j;
color = buffer[j];
tft.drawPixel(row, pix, color);
}
}
I think you need to optimise your network as first,
and remove printing which slows down the whole process.
connect esp32 to fast and fixed local network.
If you reduce row to half size (320 - 160), you could maybe
continue using OSC protocol, which I think you know how to deal with.
sadam.udpSender can send raw bytes 0-255 only.
to pass uint16_t you need to insert
sadam.toBytes set to uint16_t
that will send color as 2 bytes.

you see here the conversion
......
But you don't mention what message makes you pixel show correct color ?????
what is a proper message your tft library needs to set a pixel to a color ?
I don't care how much "datas" you manage to print from received udp message.
what counts is if you manage to set a pixel to correct color.
without that info, it makes no sense to talk about OSC, raw udp or whatever else.
Hi,
tft library needs a uint32 to set a pixel color as we can see in tft library
virtual void drawPixel(int32_t x, int32_t y, uint32_t color),
I have ever tried to send unit16 from max/msp with OSC protocol but I could receive each line at 35 ms.
35 ms * 240 rows = 8400 ms.
So, to have all the data as quickly as possible, without a very good resolution (let's say 80*40 pixels), which method should I use?
I have no idea, maybe you need to check in esp 32 forums
if it is possible at all to stream video using RGB888
using wifi
Just made esp32 udp test.
sending 240 lines of 320 uint16_t values
using local Wifi and raw udp took < 300 ms
you can see transmission started 17:04:33.137

ended 17:04:33.424

end of each received lines and row number printed

P.S. all 240 lines were dumped from coll
Thanks a lot for trying to help again.
I can't receive buffer as you can. It works with 161 not with 321 and not with 642.
Maybe due to my Chinese stuff and as you said I will check on esp 32 forum.
For the moment I think I will send less data.
The best way to send data is raw udp but I can't send data with a value upper than 255.
So I will send color of 8 bit as uint8
With Arduino Code do you know a way to convert a color from uint8 to uint32 color ?
Please can you send you entire Arduino code ??
Last time I used tft was years ago using adafruit library , I don't remember which exactly.
it used uint16_t color resolution, but I did not stream any videos or such,
it was displaying colored text.
the difference is :
uint16_t color can be sent as 2 8bit bytes, if you use RGB888
then you need 3 8bit bytes.
Library you use must have somewhere example how to receive color
over serial or udp, I mean something looking like :
color = (r, g, b);
then tft.drawPixel(x, y, color); or even
then tft.drawPixel(x, y, r, g, b);
Printing of received data slows down everything.
if you need to transfer fast data, remove any trace of printing from the code.
don't make mistakes, like repeatedly output buffer byte 0 because it is inside of for loop.
I showed you that, but in your last posted arduino code
same mistake is still there.
solid wifi connection is needed , even with less data, you can get
dropouts because of unreliable wifi.
My test run on private router, which is not conected to internet.
Maybe you need to check if streaming video to ESP32 is possible at all,
if not there must be other means to do so, even using cheapest second hand
phone.
my arduino code receives 2byte color list and row number at end,
because it is easier to convert 2 bytes to uint16_t number like that.
big endian means MSB first.
example:
10 46 = 10*256 + 46 = 2606
#include <WiFi.h>
#include <WiFiUdp.h>
#include <OSCMessage.h>
#include "esp_wifi.h"
WiFiUDP Udp;
IPAddress local_IP(10, 10, 10, 10);
IPAddress gateway(10, 10, 10, 1);
IPAddress subnet(255, 255, 0, 0);
const unsigned int localPort = 8888;
char buffer[642];
int pix; int color;
void setup() {Serial.begin(115200);
WiFi.mode(WIFI_STA); esp_wifi_set_ps(WIFI_PS_NONE); WiFi.config(local_IP, gateway, subnet);
WiFi.begin("AIR", "12345678"); Udp.begin(localPort); // SSID & password
}
void loop() {Udp.parsePacket(); if(Udp.read(buffer, 642) > 0) {showit();} }
void showit(){int row = buffer[641]; Serial.println(row);
for (int j = 0; j < 640; j++) {int pix = (j/2); int color = (buffer[pix*2]*256 + buffer[pix*2+1]);}
Serial.println("done");
}

here another test, this time with 3 rgb values sent over using this:

and Arduino
#include <WiFi.h>
#include <WiFiUdp.h>
#include <OSCMessage.h>
#include "esp_wifi.h"
WiFiUDP Udp;
IPAddress local_IP(10, 10, 10, 10);
IPAddress gateway(10, 10, 10, 1);
IPAddress subnet(255, 255, 0, 0);
const unsigned int localPort = 8888;
char buffer[961];
int pix; int color;
void setup() {Serial.begin(115200);
WiFi.mode(WIFI_STA); esp_wifi_set_ps(WIFI_PS_NONE); WiFi.config(local_IP, gateway, subnet);
WiFi.begin("AIR", "12345678"); Udp.begin(localPort); // SSID & password
}
void loop() {Udp.parsePacket(); if(Udp.read(buffer, 961) > 0) {showit();}
}
void showit(){int row = buffer[960]; Serial.println(row);
for (int j = 0; j < 960; j++) {int pix = (j/3);
int r = (buffer[pix]); int g = (buffer[pix+1]); int b = (buffer[pix+2]); }
}
this time only row number gets printed
all lines received in ± 260 ms time
sometimes longer, but allways under 300 ms
Hi!! Thanks a lot!!
It looooks very good ;)
With your last Arduino Code I can receive well 961 data at once.
I found a function to draw color to TFT and I use it like that.
tft.drawPixel(row, pix, tft.color565( r, g, b ));
But you removed [qmetro] before your counter so I receive row not properly.
10:21:08.847 -> 22
10:21:08.847 -> 22
10:21:08.847 -> 117
10:21:08.847 -> 117

Can you put it again, please, I 'am afraid to do stupid thing..
Here the patch testing two different movies

your problem is
1 - you are not resetting counter, so it counts forever instead of wrapping 240 rows.
see here :

you have 2 options :
1 reset counter when jit.iter dumps "done" :

or maybe better use jit.iter cell outlet and remove counter.

..............
2- you must reduce playback frame rate to be able to really fill the screen.
allways check what goes out BEFORE plugging data stream into sadam.udpSender.