dynamically loading java classes in a standalone


    Feb 21 2008 | 6:51 pm
    Hey gang --
    I'm building some standalones (Max 4.6.3, OSX 10.4.11, 2 GHz Intel
    MacBook), and I can get things to work by putting the max.jar file in
    "my.app/Contents/support/java/lib/". Now I have a large jar file,
    and I put it in "my.app/Contents/support/java/classes/". The problem
    is that the standalone is treating the jar file as a system jar and
    loading the entire thing into memory at start-up. I want to add the
    jar to the dynamic classpath so that it will only load the classes I
    need from the jar.
    I do this when running MaxMSP by setting the "max.dynamic.class.dir"
    in the "max.java.config.txt" to point to the large jar file. How can
    I set the dynamic classpath in a standalone?
    thanks!
    brad

    • Feb 21 2008 | 8:44 pm
      I don't really understand what you mean by saying that your entire
      jar is being loaded at startup. If you have any mxj instances in your
      standalone patch they should only load the classes that they have
      dependencies on. I am not aware of a way to get the JVM to force load
      a bunch of arbitrary classes.It loads any class when it needs it with
      the exception of the bootstrap classes.
      Regardless. I am not totally sure but if you include
      max.java.config.txt it is possible that mxj will find it. You might
      want to mess around with this.
      In the case that it does find it you will need to mess around with
      paths for the max.dynamic.jar.dir, max.dynamic.class.dir etc since
      they will need to be
      relative and I am not sure what they would be relative to.
      Sorry I can't be more helpful.
      t
      On Feb 21, 2008, at 10:51 AM, Brad Garton wrote:
      > Hey gang --
      >
      > I'm building some standalones (Max 4.6.3, OSX 10.4.11, 2 GHz Intel
      > MacBook), and I can get things to work by putting the max.jar file
      > in "my.app/Contents/support/java/lib/". Now I have a large jar
      > file, and I put it in "my.app/Contents/support/java/classes/". The
      > problem is that the standalone is treating the jar file as a system
      > jar and loading the entire thing into memory at start-up. I want
      > to add the jar to the dynamic classpath so that it will only load
      > the classes I need from the jar.
      >
      > I do this when running MaxMSP by setting the
      > "max.dynamic.class.dir" in the "max.java.config.txt" to point to
      > the large jar file. How can I set the dynamic classpath in a
      > standalone?
      >
      > thanks!
      >
      > brad
      >
    • Feb 21 2008 | 10:01 pm
      I am guessing at behavior I don't understand. What is the difference
      between the 'immutable classpath" and the "dynamic classpath". I
      have a jar file that's about 6 Mbtyes, and inflates to about 30
      Mbytes. If I put it on the system (immutable) classpath or just put
      it into one of the java classes/ or lib/ subdirs, it takes about a
      minute to instantiate the object that references a class in that
      jar. I can watch the memory usage slowly creep up to about 30
      Mbytes from the starting point as apparently all of the classes are
      being loaded and inflated in memory
      If, however, I explicitly put the jar itself on the "dynamic
      classpath" then the object that draws on that jar load immediately
      with no huge memory allocation. I don't know what is causing the
      different behaviors. I assumed it had to do with how the JVM was
      loading the classes in the jar.
      Also, if I unpack the jar into all the separate classes in the
      classes/ dir, it acts like when I add the jar to the "dynamic
      classpath"; i.e. it loads quickly.
      brad
      On Feb 21, 2008, at 3:44 PM, topher lafata wrote:
      > I don't really understand what you mean by saying that your entire
      > jar is being loaded at startup. If you have any mxj instances in
      > your standalone patch they should only load the classes that they
      > have dependencies on. I am not aware of a way to get the JVM to
      > force load a bunch of arbitrary classes.It loads any class when it
      > needs it with the exception of the bootstrap classes.
      >
      > Regardless. I am not totally sure but if you include
      > max.java.config.txt it is possible that mxj will find it. You might
      > want to mess around with this.
      > In the case that it does find it you will need to mess around with
      > paths for the max.dynamic.jar.dir, max.dynamic.class.dir etc since
      > they will need to be
      > relative and I am not sure what they would be relative to.
      >
      > Sorry I can't be more helpful.
      >
      > t
      >
      > On Feb 21, 2008, at 10:51 AM, Brad Garton wrote:
      >
      >> Hey gang --
      >>
      >> I'm building some standalones (Max 4.6.3, OSX 10.4.11, 2 GHz Intel
      >> MacBook), and I can get things to work by putting the max.jar file
      >> in "my.app/Contents/support/java/lib/". Now I have a large jar
      >> file, and I put it in "my.app/Contents/support/java/classes/".
      >> The problem is that the standalone is treating the jar file as a
      >> system jar and loading the entire thing into memory at start-up.
      >> I want to add the jar to the dynamic classpath so that it will
      >> only load the classes I need from the jar.
      >>
      >> I do this when running MaxMSP by setting the
      >> "max.dynamic.class.dir" in the "max.java.config.txt" to point to
      >> the large jar file. How can I set the dynamic classpath in a
      >> standalone?
      >>
      >> thanks!
      >>
      >> brad
      >>
      >
    • Feb 21 2008 | 10:42 pm
      On Feb 21, 2008, at 14:01 PM, Brad Garton wrote:
      > I am guessing at behavior I don't understand. What is the
      > difference between the 'immutable classpath" and the "dynamic
      > classpath". I have a jar file that's about 6 Mbtyes, and inflates
      > to about 30 Mbytes. If I put it on the system (immutable)
      > classpath or just put it into one of the java classes/ or lib/
      > subdirs, it takes about a minute to instantiate the object that
      > references a class in that jar. I can watch the memory usage
      > slowly creep up to about 30 Mbytes from the starting point as
      > apparently all of the classes are being loaded and inflated in memory
      The system classpath is the normal java classpath which is specified
      to the vm at startup. java -classpath C:/blah/blah etc.Once the vm is
      started it can't be changed. This relates to the security of the vm.
      The only way you can add a new class to the vm at runtime is by using
      a custom classloader.
      The dynamic classpath is a construct we came up with so that classes
      could be reloaded by the mxj classloader (our custom classloader) so
      you don't have to quit and start max everytime you change a class
      definition. This is useful during the development phase of an mxj class.
      Thus, any class that you want to reload dynamically after its
      definition changes needs to be in the dynamic classpath.For a
      standalone the classes won't really be changing so you should put
      your jar in:
      my.app/Contents/support/java/lib/
      alongside max.jar and see if that helps.
      The only reason that your code is working at all in the standalone is
      because as a last resort max will search the whole max search path
      for classes and jars it cannot find in the classpath. This would
      explain why it takes such a long time for your classes in the jar to
      load. The fact that the memory is increasing suggests that there may
      be a memory leak when max resorts to scanning the whole max search
      path for a class.
      It is not finding your jar in the classes directory since that
      directory is only for classes not jar files. However it is finding
      the classes in your jar file when it resorts to the full max searchpath.
      The c74/classes folder is by default part of the dynamic search path.
      We do not have a default dynamic jar path but one can be specified in
      max.java.config.txt.
      I am not sure how this plays out in a standalone since I am not sure
      how to specify a relative path to a dynamic jar directory in the
      context of a standalone. It would probably have to be relative to the
      max executable itself.
      In theory you could specify in max.java.config.txt these relative
      paths for any dynamic class and/or dynamic jar directories and
      included that max.java.config.txt in your standalone.It is just a
      question of finding out what they should be specified relative to
      which as i said above is probably the max executable itself.
      So to make a long story short.
      try moving your jar to the same directory as max.jar and see if your
      standalone behaves as expected.
      t
      > If, however, I explicitly put the jar itself on the "dynamic
      > classpath" then the object that draws on that jar load immediately
      > with no huge memory allocation. I don't know what is causing the
      > different behaviors. I assumed it had to do with how the JVM was
      > loading the classes in the jar.
      >
      > Also, if I unpack the jar into all the separate classes in the
      > classes/ dir, it acts like when I add the jar to the "dynamic
      > classpath"; i.e. it loads quickly.
      >
      > brad
      >
      >
      > On Feb 21, 2008, at 3:44 PM, topher lafata wrote:
      >
      >> I don't really understand what you mean by saying that your entire
      >> jar is being loaded at startup. If you have any mxj instances in
      >> your standalone patch they should only load the classes that they
      >> have dependencies on. I am not aware of a way to get the JVM to
      >> force load a bunch of arbitrary classes.It loads any class when it
      >> needs it with the exception of the bootstrap classes.
      >>
      >> Regardless. I am not totally sure but if you include
      >> max.java.config.txt it is possible that mxj will find it. You
      >> might want to mess around with this.
      >> In the case that it does find it you will need to mess around with
      >> paths for the max.dynamic.jar.dir, max.dynamic.class.dir etc since
      >> they will need to be
      >> relative and I am not sure what they would be relative to.
      >>
      >> Sorry I can't be more helpful.
      >>
      >> t
      >>
      >> On Feb 21, 2008, at 10:51 AM, Brad Garton wrote:
      >>
      >>> Hey gang --
      >>>
      >>> I'm building some standalones (Max 4.6.3, OSX 10.4.11, 2 GHz
      >>> Intel MacBook), and I can get things to work by putting the
      >>> max.jar file in "my.app/Contents/support/java/lib/". Now I have
      >>> a large jar file, and I put it in "my.app/Contents/support/java/
      >>> classes/". The problem is that the standalone is treating the
      >>> jar file as a system jar and loading the entire thing into memory
      >>> at start-up. I want to add the jar to the dynamic classpath so
      >>> that it will only load the classes I need from the jar.
      >>>
      >>> I do this when running MaxMSP by setting the
      >>> "max.dynamic.class.dir" in the "max.java.config.txt" to point to
      >>> the large jar file. How can I set the dynamic classpath in a
      >>> standalone?
      >>>
      >>> thanks!
      >>>
      >>> brad
      >>>
      >>
      >
    • Feb 21 2008 | 10:50 pm
      one more question.
      how large are all of those classes before you jar them up?
      it is very possible that the jar is being unzipped when the mxj
      classloader falls back on trying to locate the class in the max
      search path. as a matter of fact it is entirely possible that all
      jars in the max search path are being unzipped temporarily into
      memory ,including max.jar, since the classloader needs to crack open
      any jar file it finds in the search path to check if it contains the
      class it is looking for.
      This previous point would explain the memory blowup you are seeing.
      As mentioned before, the long time it takes for instantiation would
      be due to the fact that the whole max search path is being scanned
      for each class you are loading since your jar is not in the system or
      dynamic classpath and the classloader is falling back to its last
      resort scheme for classloading.
      t
      On Feb 21, 2008, at 14:01 PM, Brad Garton wrote:
      > I am guessing at behavior I don't understand. What is the
      > difference between the 'immutable classpath" and the "dynamic
      > classpath". I have a jar file that's about 6 Mbtyes, and inflates
      > to about 30 Mbytes. If I put it on the system (immutable)
      > classpath or just put it into one of the java classes/ or lib/
      > subdirs, it takes about a minute to instantiate the object that
      > references a class in that jar. I can watch the memory usage
      > slowly creep up to about 30 Mbytes from the starting point as
      > apparently all of the classes are being loaded and inflated in memory
      >
      > If, however, I explicitly put the jar itself on the "dynamic
      > classpath" then the object that draws on that jar load immediately
      > with no huge memory allocation. I don't know what is causing the
      > different behaviors. I assumed it had to do with how the JVM was
      > loading the classes in the jar.
      >
      > Also, if I unpack the jar into all the separate classes in the
      > classes/ dir, it acts like when I add the jar to the "dynamic
      > classpath"; i.e. it loads quickly.
      >
      > brad
      >
      >
      > On Feb 21, 2008, at 3:44 PM, topher lafata wrote:
      >
      >> I don't really understand what you mean by saying that your entire
      >> jar is being loaded at startup. If you have any mxj instances in
      >> your standalone patch they should only load the classes that they
      >> have dependencies on. I am not aware of a way to get the JVM to
      >> force load a bunch of arbitrary classes.It loads any class when it
      >> needs it with the exception of the bootstrap classes.
      >>
      >> Regardless. I am not totally sure but if you include
      >> max.java.config.txt it is possible that mxj will find it. You
      >> might want to mess around with this.
      >> In the case that it does find it you will need to mess around with
      >> paths for the max.dynamic.jar.dir, max.dynamic.class.dir etc since
      >> they will need to be
      >> relative and I am not sure what they would be relative to.
      >>
      >> Sorry I can't be more helpful.
      >>
      >> t
      >>
      >> On Feb 21, 2008, at 10:51 AM, Brad Garton wrote:
      >>
      >>> Hey gang --
      >>>
      >>> I'm building some standalones (Max 4.6.3, OSX 10.4.11, 2 GHz
      >>> Intel MacBook), and I can get things to work by putting the
      >>> max.jar file in "my.app/Contents/support/java/lib/". Now I have
      >>> a large jar file, and I put it in "my.app/Contents/support/java/
      >>> classes/". The problem is that the standalone is treating the
      >>> jar file as a system jar and loading the entire thing into memory
      >>> at start-up. I want to add the jar to the dynamic classpath so
      >>> that it will only load the classes I need from the jar.
      >>>
      >>> I do this when running MaxMSP by setting the
      >>> "max.dynamic.class.dir" in the "max.java.config.txt" to point to
      >>> the large jar file. How can I set the dynamic classpath in a
      >>> standalone?
      >>>
      >>> thanks!
      >>>
      >>> brad
      >>>
      >>
      >
    • Feb 22 2008 | 2:56 am
      Ok, I see I'm pretty confused about this. But the behavior I'm
      seeing is also confusing, and I can't think of why. It's my maxlispj
      object, and the BGabcl.jar file is 6.0 Mbytes. Un-jarred and
      inflated, it's about 30 Mbytes of classes (which is the amount that
      gets allocated when it does the "slow load" thing). Here's the
      behavior:
      In a standalone, if I put the BGabcl.jar file in either of these
      directories:
      my.app/Contents/support/java/lib/
      my.app/Contents/support/
      it loads quickly without the additional 30 Mbyte memory allocation.
      If I put it here:
      my.app/Contents/support/java/classes/
      it takes a long time to load and allocates about 30 Mbytes of memory.
      In the MaxMSP application, if I put it here:
      /Applications/MaxMSP 4.6/Cycling '74/java/classes/
      it takes a long time to load and allocates about 30 Mbytes of
      memory. But if I add this line to the max.java.config.txt file:
      max.dynamic.class.dir "/Applications/MaxMSP 4.6/Cycling '74/java/
      classes/BGabcl.jar"
      it loads quickly. Hence the source of my confusion about the dynamic
      classpath.
      One final weirdness, if I put the BGabcl.jar file here:
      /Applications/MaxMSP 4.6/Cycling '74/java/lib/
      with or without an entry in the max.java.config.txt file, it loads
      quickly (and works ok!), but I get these errors in the Max window:
      error: java.util.zip.ZipException: No such file or directory
      error: at java.util.zip.ZipFile.open(Native Method)
      error: at java.util.zip.ZipFile.(ZipFile.java:203)
      error: at java.util.zip.ZipFile.(ZipFile.java:84)
      followed by some Lisp loader errors, but the object gets instantiated
      ok!
      I've been using the object a lot, and it seems to work well even in
      the slow-load case. I really don't know what's going on with the
      loading differences, though.
      I know you've got a lot going on right now, so don't delve into this
      too deeply unless something seems obvious to you from the above
      description. If I put it in the right dirs, it works fine, and I'm
      happy.
      brad
    • Feb 22 2008 | 3:00 am
      This is what I think is happening, because the memory increase I see
      is about how large the un-jarred classes are.
      I think my confusion came from the dynamic classpath entry in the
      max.java.config.txt file making it work ok in the java/classes/ dir
      in the MaxMSP application hierarchy.
      brad
      On Feb 21, 2008, at 5:50 PM, topher lafata wrote:
      > one more question.
      >
      > how large are all of those classes before you jar them up?
      >
      > it is very possible that the jar is being unzipped when the mxj
      > classloader falls back on trying to locate the class in the max
      > search path. as a matter of fact it is entirely possible that all
      > jars in the max search path are being unzipped temporarily into
      > memory ,including max.jar, since the classloader needs to crack
      > open any jar file it finds in the search path to check if it
      > contains the class it is looking for.
      >
      > This previous point would explain the memory blowup you are seeing.
      > As mentioned before, the long time it takes for instantiation would
      > be due to the fact that the whole max search path is being scanned
      > for each class you are loading since your jar is not in the system
      > or dynamic classpath and the classloader is falling back to its
      > last resort scheme for classloading.
    • Feb 22 2008 | 6:33 am
      > it takes a long time to load and allocates about 30 Mbytes of
      > memory. But if I add this line to the max.java.config.txt file:
      >
      > max.dynamic.class.dir "/Applications/MaxMSP 4.6/Cycling '74/java/
      > classes/BGabcl.jar"
      >
      > it loads quickly. Hence the source of my confusion about the
      > dynamic classpath.
      >
      yeah. dont do this. the class directory is for classes not jars. you
      could do:
      max.dynamic.jar.dir "/Applications/MaxMSP 4.6/Cycling '74/java/myjars"
      and put BGabcl.jar in it.
      I cant say what the behavior of mxj is when you set a dynamic class
      dir to a jar. suffice it to say that it is undefined.
      > One final weirdness, if I put the BGabcl.jar file here:
      >
      > /Applications/MaxMSP 4.6/Cycling '74/java/lib/
      >
      > with or without an entry in the max.java.config.txt file, it loads
      > quickly (and works ok!), but I get these errors in the Max window:
      >
      > error: java.util.zip.ZipException: No such file or directory
      > error: at java.util.zip.ZipFile.open(Native Method)
      > error: at java.util.zip.ZipFile.(ZipFile.java:203)
      > error: at java.util.zip.ZipFile.(ZipFile.java:84)
      >
      there might be something weird in the way your jar is constructed?
      manifest out of sync? confusion about having that weird dynamic
      classpath entry point to a jar? i cant say.
      > I know you've got a lot going on right now, so don't delve into
      > this too deeply unless something seems obvious to you from the
      > above description. If I put it in the right dirs, it works fine,
      > and I'm happy.
      awesome.
      ciao
      t