Passing Min Function as a callback to a function in another class?

If you can post the signature of the thing you're calling that takes a callback, I'll try to give specific advice. The likely bottom line is that you'll want it to be able to call queue<>::set() on your Min object in order to enqueue something in the Max main thread, and to do that your callback will need access to a pointer or reference to the instance of your Min object.

Passing around function-like things in C++ is a good deal more intricate than in C, because there are more types of thing that can behave as if they are functions, and they all have different types (as you've discovered). For instance, the types of a pointer to a 'normal' free function and a pointer to a member function with the same arguments and return type are different. See https://isocpp.org/wiki/faq/pointers-to-members for a discussion of this.

As well as pointers to functions and pointer to member functions, another common thing to come across in C++ is the idea of a function object, which is some struct that provides an overloaded call operator (operator()), e.g.:

struct MyFunctionObject{
void operator()(const std::string& s) { std::cout << s }
};

These are very common (see std::plus, std::less etc), partly because C++ doesn't treat functions as first-class objects, and very often we would actually like to pass a whole set of overloads around, which C++ doesn't have a language facility for. Function objects are so common, in fact, that in C++11 they added lambdas as syntactic sugar for defining them quickly. The equivalent to above:

auto myPrintter = [](const std::string& s) { std::cout << s };

Meanwhile, std::function exists as a library-level workaround to the problem that all these different species of Callable thing have different types, even if they have identical effects. It does this by being able to wrap a pointer to a function, or a pointer to a member function, or a function object into an instance of std::function (specialised on the appropriate signature) so that they all then appear to have the same type. This is useful when you want to store a bunch of these things in a homogenous container like std::vector, or where you want to have a function that accepts different kinds of callback, but don't want to write a function template. The trick it uses is called type erasure, and can be thought of as a type safe equivalent to dropping down to void*. However, it doesn't come for free (it may well involve a heap allocation, and will always introduce extra indirection): whether that actually matters depends on the context.

Min uses std::function in the constructor for queue<>, which expects some callable that takes some atoms and an int, and returns atoms (see c74_min_message.h). It also defines a macro called MIN_FUNCTION that provides the signature for a lambda with the same signature, so that when you declare a queue<> in your Min class, you can just write

queue<> myQueue { this,
MIN_FUNCTION {
do_stuff();
return {};
}
};

and the compiler will wrap the lambda up into a std::function for you. Calling myQueue.set() then actually does the work of enqueuing a call to your function.