Packaging questions and review request

Ali Somay's icon

Hello, there was an external which I was working on, on and off and I think it is releasable now.

Before submitting it to Cycling 74 I want to get your feedback.

  • Created a ref page

  • Did a cross platform build

  • Created a home patcher and put the logo

  • Created maxhelp

  • Put in readme.md

  • Tested the external as much as I can

  • Code signing with my dev id for mac
    codesign -f -v -s "${DEVELOPER_ID}" --entitlements "${ENTITLEMENTS}" --timestamp --options=runtime "${PROFILE_EXTERNAL_PATH}"

Questions:

  • Currently I only have 1 external in the package I wish to publish but maybe in time I'll add more. Is this fine?

  • Should I notarize before submitting (probably yes but I wanted to also get your idea)

  • How frequently can I update my package if it is accepted after submission?

  • Is my structure ready for submission?

  • What entitlements should I include for a max external in general. It also reads and writes files through max api?

Thanks a lot!
Here is some more context:

Package structure and info:


Help:

Reference:


Package overview:

TFL's icon

I don't know what C74 standards are for getting a package into the official package manager, I guess the question of the maintainability is important here (what if you stop working on it? What if the Rytms future firmware version breaks everything?). But strictly from the authoring and documentation side of things, your work seems rock solid!

And by the way congrats for your contribution in reverse-engineering the Rytm and making an external from that! I would love to work on such thing for the Digitakt.

Ali Somay's icon

Thanks! It is good to hear some feedback!

Maintainability in theory is very important I agree but on the other hand in reality there is little chance at this point for them to change the sysex format in a way so it requires weeks of work to update this one also. But that is just a guess. Of course I can not promise that I'd maintain it fast in that case but another angle is I can imagine that it would be still valuable to have it for some and not update for a while if they're really using it in a productive way.

Me stopping working on it is always a possibility (because life..) but it is an open source project and I'd gladly hand it over to someone if there is anybody like that :).

If you'd like to check the project further:

Library https://github.com/alisomay/rytm-rs
External https://github.com/alisomay/petunia-externals/tree/main/externals/rytm

Please check https://github.com/alisomay/rytm-rs if you'd like to do something similar for Digitakt since you can reuse a lot of the techniques there and I don't believe that they change the sysex fmt for their every device. It should be quite possible.

Underlying libs for rytm-rs are in:
https://github.com/alisomay/rytm-sys
https://github.com/bsp2/libanalogrytm/

Ali Somay's icon

Also if anybody wants to evaluate the current state of the package here it is:

petunia.zip
zip


JBG's icon

Hi Ali,

This is a bit off topic, but I just noticed that your package is written in rust using the median bindings. Just out of curiousity, how was your experience working with this library? If you've previously worked with the C (max-api) or C++ (min-api) bindings, how does it work in comparison (missing features, error handling, concurrency, etc.)?

Ali Somay's icon

Ah sure,

I've written C (max-api) externals years ago but more to explore the SDK not complex stuff.
I've never used C++ (min-api) since it was not made at that time.

I'm kind of familiar with C (max-api) headers, probably not all but I check them out when I need to.
I find the source code of the headers more easier to read than the doxygen generated doc.

Rust bindings are usually done with bindgen. They're auto generated from the C headers.
In the Rust ecosystem we usually separate the binding packages (crates) and suffix them as -sys.

The max-sys crate mirrors all the C headers which you can find in the SDK so you can call those functions but they're considered "unsafe" since the Rust compiler (borrow checker) can not track the lifetimes of allocations through the FFI boundary (Foreign Function Interface) so you need to manage the memory manually like C.

People usually wrap the memory management logic and expose APIs to the users which adheres to Rust standards so at least people can think about the memory management in a Rust way or think about the memory management a bit less.

Median also does this (a little bit).

It also experiments with proc marcos (compiler plugins for Rust) where it generates code in compile time to make things easier for the user even closer to JS apis. On the other hand there are heavy trade offs with this approach and it is hard to get right. In median's case I'd consider it more like an experiment compared to a feature. I'd say if you'd like to do something serious you should avoid using those macros at the time of writing this.

Here is my opinion on that
https://github.com/Cycling74/median/issues/17

My approach is to use a hybrid of median safe wrappers with raw max-sys bindings since median safe wrappers are not complete and do not expose the complete features of the C SDK.

On the other hand it is a good starting point and for me writing in Rust is much much more ergonomic compared to C and on top of that you get all the goodies of crates.io, borrow checker, easy concurrency, testing suite, tracing and all the other very nice features which comes with Rust.

Error handling (if you know how to do it) is very powerful in Rust so combined with tracing you can really go a long way there and development experience is much easier. Also if you don't use raw max-sys apis due to Rust guarantees there will be no segfaults and raw pointer management. If you're are careful with max-sys you can also wrap the apis you need in a safe way.

In short words, if you're an experienced Rust programmer median wins in every way but the project (in my opinion) is in an experimental state.

The downside is the beginner or early intermediate Rust programmer can not use it easily if they want to do something relatively complex. In other words the entry barrier is high.

If one develops median more this entry barrier may get lower but this requires a lot of careful design work and somebody needs to devote time into it :)

JBG's icon

Thanks a lot for sharing all of this!

So basically, the entire max-api is exposed through the auto generated bindings in max-sys, and the fact that they are auto-generated also means that if the max-api is updated in the future (e.g. with array/string access), it would be easy to generate the necessary new bindings, right?

Either way, I'm glad to hear that you seem happy about the experience so far. I'll definitely start tinkering with this in the near future, and what you described here will really help out a lot when getting started!

Ali Somay's icon

To be really clear the conventional way is:

Repo 1
C/C++ Lib (open source fork of the upstream or directly upstream)
|
|
Repo 2 (has repo 1 as a submodule)
-sys bindings crate
|
|
Repo 3 (depends on -sys crate through cargo)
-rs crate

In rust ecosystem the dependencies are also built from source when you build your main crate. Cargo pulls their source and builds it in your machine.

So in Repo1, if your fork or the upstream (depending on your choice) is updated, in the next build of your -sys bindings, your build.rs script will build the C/C++ lib from the updated source and bindgen will re-generate the bindings from the updated headers in the updated source.

In median's case (to gain speed I guess..) this behavior is gated under a feature flag.
Check: https://github.com/Cycling74/median/blob/develop/max-sys/README.md
You can also check here to understand what that feature flag enables.
https://github.com/Cycling74/median/blob/168bd8e01971cb263c94c12cabc49d75c461f761/max-sys/build.rs#L171-L178

So for median you'll not get updates for free :)
You'd need to build it with a feature flag but that is also only one part.
You'd also need to update the sub module of the SDK and probably fork median and then you can open a PR to update upstream if you think that the new generations are valuable.

Because the sub module of the SDK is tracking a specific rev:
https://github.com/Cycling74/max-sdk/tree/f7495bf78e7f3f9ea08d117b732eee1993a05aa0

Actually this rev is a bit outdated even now.

It is not complicated but it just needs some maintenance. I think if more people use it then the maintenance will naturally happen by incoming PRs and discussions etc.. Naturally there is only so much time Alex Norman has https://github.com/x37v.

Great that you're going to try to use it also.
If you have any questions along the way don't hesitate.

JBG's icon

Alright, sounds simple enough. Again, thanks for sharing!

Ali Somay's icon

Notarized and submitted the package lets see :)

If you'd like to check the notarization script (which might be useful to some) it is here:
https://github.com/alisomay/petunia-externals/blob/main/justfile