Meeting with the Collab-Hub Team

Cycling '74: We've brought together a group of people that are working on a very interesting project. The project name is Collab-Hub, and it's a way for people working on multiple computer systems to work collaboratively. The three people involved in the the project are Anthony T. Marasco, Eric Sheffeld and Nick Hwang. To get started, let's learn a little bit about what Collab-Hub is. Nick, could you just give us an overview of what Collab-Hub can do?

Nick Hwang: Sure. Collab-Hub is a remote collaboration tool which is meant to help artists, musicians, circuit benders, game developers, or creative coders share data through a network. And it can be a remote network — so that they are working together to either make music, make art, make visual art, or make multiplayer games. The goal was to have them not limited to their own creative applications. We started out with Max (as a platform), and we've slowly added on different applications so that they can send data effortlessly through the network.

One of the things that we wanted to tackle early was not having users go through the arduous process of finding out their own IP address and their data ports. We're using a server-based network system so that they are relaying that information to a remote server, and then are able to share it - and decide where it goes. So they can say that they want to share with everyone that's participating, or only people within a particular room, or only to particular users along the way.

Cycling '74: That really sounds great, and it seems really important. I've seen a lot of different tele-artwork done over the years, and it always seemed like the art almost had to take second place behind the technology. It was such a technological challenge to do this properly, that it really foregrounded the technological part of it and perhaps made the art part a little less present, right? It's good to see a new kind of solution for this.

Now, I would like to know a little bit more about each of you, and how you are involved in the project. Let's start with you, Anthony.

Anthony Marasco: Yeah. So, I’m an assistant professor of Music Technology and Composition at the University of Texas Rio Grande Valley. Like Eric and Nick, I’m a composer, a sound designer. I do a lot with hardware and hardware is kind of my lane here with the team. As Nick mentioned, we originally started using Collab-Hub as a Max-to-Max data communication tool. But we realized that we wanted to make this a tool for all artists and all different mediums. And once we simplified it and kind of agreed on a universal communication protocol (and the server design and the way everything would talk to each other), that made it easy for us to bring in other applications to this environment. Hardware is where I’ve been leading a lot of the development focus.

So, for example, we have added an Arduino library for wireless enabled microcontrollers (like the ESP32, and I think Arduino has just released some of their own competing boards). Our library uses the same Collab-Hub communication protocol as our Max client does and we can have a Max patch wirelessly talk to a custom interface or robotic instrument from halfway around the world. And we’re also working on other hardware implementations for modular synthesizers. There’s an open source module called The Motivation Radio, which uses an ESP32 at its heart. I’ve written firmware to get that up and running as a bi-directional communication tool for modular synth users.

So, in performance, Nick and Eric have both controlled my modular synth all the way down here in south Texas. We’re also working on a client library for the Monome Norns, which is a really popular and easy to use computer music instrument with a large community of coders supporting it. The Norns library will let you synchronize your favorite scripts with a Max patch locally or across the globe - or a Pure Data patch or a Unity game. All of those things are going to be possible. And we want to make sure that hardware was not left out of this process because it’s very easy to get very insular and focus on communicating between laptop to laptop virtually, but the hardware maker scene is, I think, really important to focus on too.

Cycling '74: So true. Eric, how about you?

Eric Sheffeld: So, I actually just started a pretty new position here at Miami University. I'm split between the Music Department, and... it's a mouthful: the Emerging Technology in Business and Design Department. I am really excited to be able to have one foot in the creative world and one foot in the technology development world. The germ of the idea for Collab-Hub did stem from Max because of its support for Node.js. Our earliest projects and earliest workshops and stuff that we did were really just focused on that. So, I've mostly worked on the Max client side of things, and we sort of tried to make the Max client for Collab-Hub as familiar as possible to Max users. Therefore, it is broken up into sub modules that are loaded as bpatchers.

So, you can load only the functions of Collab-Hub that you need - because you might need to do things like room management. You can do things like adding and removing different control headers that you're broadcasting and letting the server know which ones you're using, and which ones you're not using. You can do chat and functions like that. So, we wanted to make sure that it wasn't overwhelming to see everything at once.

And all of the communication that comes in from the server runs inside the Node.js object inside Max. And all of the communications that come in from the servers are distributed to the normal Max GUI stuff. So all the umenus get populated, the dict.view objects get populated with all of the data from the server. That's been most of my work so far: working on the Max side of things.

Cycling '74: Great and what about you, Nick?

Nick Hwang: Yeah. I'm an assistant professor at the University of Wisconsin Whitewater, and a little bit before the start of the pandemic, Eric, Tony and I presented on a workshop at MOXsonic talking about potentials of using Node for Max. And shortly after that, we started developing the server thinking that we could develop a tool and not just a performance piece - a tool that could help people reach each other and send control data to each other around the world. And so that's when we started Collab-Hub. I've been involved in making the server and some of the other libraries, including the web interface. And we've noticed that using the web interface could be the potential to link many other libraries or applications as well. After the Max library and on top of the Arduino ones, we're building one that's a generic OSC library so that we can interface with many, many, many other applications as well.

Cycling '74: That makes a lot of sense. When did you all first start working on Collab-Hub? And when did you sort of lock down the protocol? Because in order to engage in hardware design, you have to be pretty confident that the protocol is pretty well locked down. So when did you start, and at what point did you feel like this was solid enough that you could actually start pouring it into hardware?

Anthony Marasco: I feel like it was once the pandemic started and we were all kind of settling ourselves in, probably by summer of 2020. We had one of our first big performances at the network music festival I think, right Nick?

Nick Hwang: Yes.

Anthony Marasco: And so at that point, we had a version of our server that we felt represented the core basics of what we're going to be doing with communicating. We had a five-member group which we still perform with regularly. It's the three of us plus Anna Weisling and Jeff Herriott. And we got into a groove — not just compositionally and performance wise — but also being able to say, "Here is how we should send different types of data.", "Here are the types of data.", and "Is a control data message meant to change a parameter?"

Is this an event message - one meant to just say, "Okay, start the piece, end the piece, move to the next section." We had kind of solidified that core structure. I think by that point, the performances we were doing warranted our confidence, and we thought other people would want that too. Then as the year went on and we continued to refine it. I think just to say broadly, the goal was just making it easier, making it more streamlined, because once you start opening this up to more clients, you have to agree on a universal method of doing this, which Nick did a lot to spearhead as to how you communicate with the server - whether it's from Pure Data, Unity, whether it's from your Arduino.

Those types of things all sort of came about after we performed with the tool and saw what we wanted, but also started asking around, to see what other people wanted - because we were also giving presentations on the tool as we performed with it at different conferences. And so, it was a nice a reactive process where we could see how other people asking for things that we hadn't considered or things that we'd noticed we needed. And I think by the end of 2020 or the beginning of 2021, I think we felt we had this system kind of locked down and all agreed upon. And that's when I started actively doing the hardware development, because at that point we could start using some reliable approaches to actually send data in the physical and the virtual world too.

Cycling '74: Got it. Now, Nick, you were involved in setting up the network design. This is all brokered through a central server, right?

Nick Hwang: Yes.

Cycling '74: How does that manage multiple projects potentially happening at the same time?

Nick Hwang: At the moment, on the macro level, we have silos that we call "namespaces" and we have a general open namespace. So, if anyone would download the Max suite today, they would be thrown into the general hub namespace, but then we've created some other namespaces that are siloed for other organizations. So, let's say that if you were the Ohio State University Laptop Ensemble - they would log into their separate namespace and within namespaces you can choose to group participants into rooms if they want to, and they can be part of their own room or join many rooms. But at the moment the server has those specific namespaces that can keep participants separate from each other.

Cycling '74: That makes a lot of sense, that also shows a lot of flexibility. This idea of having silos for organizations can be really useful in a university environment where you want to have collaboration, but within limits. Now Eric, I'm interested in the Max implementation of Collab-Hub, and particularly how difficult it is for somebody who just on a whim says, "Yeah, I want to work with my friend over in Boston, let's whip something up." How much of a ramp up time is there? And to what extent are the messages that you send back and forth similar to the message syntax we are used to in Max?

Eric Sheffeld: I tried really hard to make it exactly the syntax that you're used to in Max. And that sort of guides our client design for all the different platforms. But really we want to make sure that whatever you're using, that you're using a syntax you're familiar with in that program, you don't feel like you're learning something different. So, for example, in Max, the client script that runs in the Node object object is taking in a message that's like a header and a value. In Max we're used to using the route objects to move things around. Data in with a particular string or a particular number that like a header that we can use to direct the data to different places. And that's exactly what you send into the Collab-Hub client.

Let's say you send in the header "slider1", and then following that would be whatever values go with "slider1". And that's it. There are a couple of key words that someone might want to know if they want to do custom stuff, like change their username on the server, or if they want to create a new room. Our GitHub repo has a guide for all of those, exactly what the client expects to see so you can to send that message to the server. But even within that, the submodules have been pre-formatted to do all that for you. So if you want to create a room, there's already a button that says "create room", but you could also send a message that just says, "create room with the room name" - you could sort of customize it yourself if you'd like.

Cycling '74: Got it. And how complex of data can you work with? I mean, it is pretty clear that key/value combinations are built in. How would you be able to deal with more complex data - like dictionaries or Jitter matrices or some of these more complex data structures?

Eric Sheffeld: Nick is smiling because at some point he was testing sending general matrices, and I can't remember how large they got, but I want say you did at least 16x16. Nick, is that right?

Nick Hwang: 16x16, I was fooling around with 32x32 as well. So we were able to send some simplified matrices, 32x32, and be able to have them display easily onto a web interface or send it over to another Max patch.

Cycling '74: That's really important because matrices are so useful for two-dimensional data. Now, it sounds like a pretty robust set up right now, and one of the ways that you're expanding it is finding ways for different systems to utilize the Collab-Hub protocol. What is, what are some of the future plans you have for expanding what Collab-Hub itself is?

Anthony Marasco: What has come up in a lot of workshops and presentations we've given - at the NIME conference and at the Seamus conference and things like that - has been, "Will there be a local version of this?" One that you can spin up on your machine and not use the web-hosted server that's hosted globally, but instead be able just to say, I run this in my room co-located, to cut down on the natural web latency? And we have been working on an approach to get to that point, where we can have somebody run a server with a GUI attached to it so that they don't have to modify any parts of the server code, but they can do the same thing they do with our web interface or with the Max modules that Eric developed.

So I can create a room, or I can delete a room. I want to make some controls that people can subscribe to and observe, and they could do it all through a regular text field or a button - or something like that. So, that's something that we're working on now, so that people who are concerned about trying to use this halfway across the planet can say, "Well, okay, I can use the same Max patches, the same Arduino scripts, everything I was using when I was connecting to the internet. All I have to do is just change one little part when I hit connect, it's now connecting to all my devices within this concert hall." And there won't be a need to rewrite my scripts when I want to take a piece from one network to another.

Eric Sheffeld: I just want to clarify that, with the networking stuff, Collab-Hub uses Web Sockets as part of the backend. One thing that we've all experienced at one point is doing a network piece using OSC communication that doesn't always work at a university network or on an institutional network. So far, all the testing we've done — I mean, I don't want to give this with a hundred percent guarantee — but so far all the testing we've done is that Collab-Hub works because it's basically just web traffic. It's not going to be blocked by any of the institutional networks. So, even someone running this locally, aside from the ease and setup, they could very well benefit from Collab-Hub because they don't have to worry about any of the firewall issues on their campus network.

Cycling '74: That's really fantastic. So, one last question. This has been around for a little while now. What's the most ridiculous thing you've seen done with Collab-Hub?

Eric Sheffeld: Besides the things that we do...?

Cycling '74: Maybe it is something that you've done, I don't know.

Anthony Marasco: My journey into this was that I developed a system called Bendit_I/O, which I have to get back into developing a bit more, which is networked circuit bending. It was a series of boards with ESP32's at their heart, that basically just simulated switches and diodes and potentiometers. So you could replace those on your physical circuit bent device with a Bendit board, and they communicate with it and have them communicate to each other over the web. And so, I've been integrating that with Collab-Hub, or switching it over to run off a Collab-Hub.

There's a lot of potential for that sort of stuff, right? Because you could have circuit bent devices communicate with each other or communicate with more complex things like web graphics or artificial intelligence run in the web, all through Collab-Hub. Therefore, you have this very older, analog, classic tech communicating with more advanced, cutting-edge virtual technology. Nick, I think Rhumbline is a pretty interesting combination of a lot of stuff that we've done so far with Collab-Hub, right? Physical computing, virtual communication through the web and with Max, do you want to talk about that?

Nick Hwang: Sure. Rhumbline was a interactive installation that I was collaborating with Meg Schedel and Robert Costco and Brian Smith. And there's a physical presence in New York where they have motorized plectrums, imitating the sound of frogs. But the whole point of it was that we wanted to have people interacting with it throughout the world. And so, there's a web interface where people can pull up visually, there's a streaming part of it. They're streaming video from New York of the frogs playing themselves. And you can hear the sounds, but the participation factor is using a web interface where you can tap buttons that make the motors move. And so, you're using the web to control the frogs and then here it stream back at you from New York.

Eric Sheffeld: I wouldn't say this is ridiculous, but I think similar to what Nick said; I think one of the most magical things has been the interactions with the physical world. And I guess, aside from Rhumbline, I think the two other great examples of that are Tony's modular synth, which somebody across the world controlling the analog electronics and all of a sudden being able to collectively control that. And one of the things that when we've done our quintet performances, Anna mostly does the live visuals. And she's been doing a lot of stuff lately where part of the visuals is like a tiny black box theater on her desk. She actually has a camera and lights and things.

And at some point we were like - so how can we interface with her besides just controlling one of her parameters inside of Jitter, where we changed the brightness of something or a threshold of something. So she set up an LED strip around the top of her tiny modular black box theater, and each of us controls a bank of those lights. And I just find that really fascinating that we're getting to that point, and Tony's Arduino library will enable more of that stuff. But the fact that we're starting to have the virtual connecting to the physical world - I'm excited for other opportunities to do that as well.

Cycling '74: That's fascinating. Well, I want to thank you all for participating in this. I appreciate it.

The Collab-Hub team is presenting a workshop at Music HackSpace on September 19, 2021. You can learn more at the Music HackSpace website.