Long deactivated M4L device causes CPU spikes when reactivated

Dear Cycling74 Community!
I use a lot of M4L devices in Ableton 12 (also tested newer betas) on a very powerful M4 Macbook (64GB RAM, 30% utilized) to automate a music live set.

The problem I face is, that most of the time my m4l devices that are turned off by the ableton "device activator" (little round yellow on/off button) or a parent rack and stay off for some minutes. When turned on, they cause drastic CPU spikes (in the thousands) although in normal operation the cpu load is only at 7%. I can replicated the issue with an empty set and 32 of these devices: when I turn them off for 10 minutes and then on again, the spike happens. With 128 devices the spike already happens after 2 minutes. In both cases turning them off/on quickly has no effect on cpu.

Of course I did a lot of research. In ableton I tried different buffer sizes, tried do not activate too many devices at the very same time, etc. In Max I checked if there are loadbangs or other elements that trigger heavy stuff when the device gets turned on. Nothing changed anything. The devices are pretty big, they have about 10 ableton dials and GUI elements and I have many of them spread over the project.

All that makes me thinking, that there must happen some kind of "unloading" process or deep sleep happening under the hood of the devices when they are off for a "while". Maybe some stuff gets pulled on the SSD and off the RAM? Maybe reading all that within one sample it too much? As I couldn't find anything on that, I wanted to ask you, if you can point to me some documentation or the like, or if you have any tips here?

Computing wise I could leave all those devices on, the cpu then does not make problems. But I would like to save some resources and I would have to restructure years of work, that guarantee that only those parts are on, that I really need...

Thank you very much in advance!
Jakob