Adam Glazier

I have tens of thousands of large XML files that are comprised of audio analysis data. Is it possible to use net.loadbang-SQL to store and search these files?

I've yet to use a database on this scale, so it's hard for me to tell if I am barking up the wrong tree.

I have tens of thousands of large XML files that are comprised of audio analysis data.  Is it possible to use net.loadbang-SQL to store and search these files?

I've yet to use a database on this scale, so it's hard for me to tell if I am barking up the wrong tree.


xml-and-net-loadbang-sql

On 24 Mar 2008, at 06:52, Adam Glazier wrote:

> I have tens of thousands of large XML files that are comprised of 

> audio analysis data. Is it possible to use net.loadbang-SQL to 

The net.loadbang SQL machinery is for relational database access; it 

doesn't provide any direct help for working with XML files, although 

it might well make sense to use a DB to store fragments of the 

You can do XML processing from Java (wheeling in Dom4j or other bits 

of XML library); alternatively, Groovy and Python probably have more 

immediate XML support, although I've not looked at this yet:

http://www.loadbang.net/space/Software/net.loadbang.groovy

http://www.loadbang.net/space/Software/net.loadbang.jython

Nick Rothwell - nick@cassiel.com - www.cassiel.com

--- open-source goodies for MaxMSP: Python, Groovy, Nixie Tubes,

--- rotatable text bricks, databases: all at www.loadbang.net

On 24 Mar 2008, at 06:52, Adam Glazier wrote:
> I have tens of thousands of large XML files that are comprised of  
> audio analysis data.  Is it possible to use net.loadbang-SQL to  
> store and search these files?

The net.loadbang SQL machinery is for relational database access; it  
doesn't provide any direct help for working with XML files, although  
it might well make sense to use a DB to store fragments of the  
decomposed XML.

You can do XML processing from Java (wheeling in Dom4j or other bits  
of XML library); alternatively, Groovy and Python probably have more  
immediate XML support, although I've not looked at this yet:

	http://www.loadbang.net/space/Software/net.loadbang.groovy
	http://www.loadbang.net/space/Software/net.loadbang.jython

Nick Rothwell - nick@cassiel.com - www.cassiel.com
--- open-source goodies for MaxMSP: Python, Groovy, Nixie Tubes,
--- rotatable text bricks, databases: all at www.loadbang.net


So are you saying that if I parse out the XML and populate a net.loadbang-SQL database, it can handle searching and sorting tens of thousands of entries?

After looking at some test results from net.loadbang-SQL, it looks like all the entries are being stored in one text file. Would it be extremely slow with such a large set of entries? It would potentially be around 1gb of data.

After looking at some test results from net.loadbang-SQL, it looks like all the entries are being stored in one text file.  Would it be extremely slow with such a large set of entries?  It would potentially be around 1gb of data.


> So are you saying that if I parse out the XML and populate a 

> net.loadbang-SQL database, it can handle searching and sorting tens 

Yep. But how well it does that depends on a lot of things...

> After looking at some test results from net.loadbang-SQL, it looks 

> like all the entries are being stored in one text file. Would it be 

> extremely slow with such a large set of entries? It would 

The text file storage is used by HSQL. I believe it does a lot of RAM 

caching while the database is active, but I've not stress-tested it 

that much. For more heavyweight data storage you probably want to move 

to something like the MySQL option, and also spend some time on the 

If your main concern is freeform searching, rather than normalised 

data representation, there are probably better solutions out there 

> So are you saying that if I parse out the XML and populate a  
> net.loadbang-SQL database, it can handle searching and sorting tens  
> of thousands of entries?

> After looking at some test results from net.loadbang-SQL, it looks  
> like all the entries are being stored in one text file.  Would it be  
> extremely slow with such a large set of entries?  It would  
> potentially be around 1gb of data.

The text file storage is used by HSQL. I believe it does a lot of RAM  
caching while the database is active, but I've not stress-tested it  
that much. For more heavyweight data storage you probably want to move  
to something like the MySQL option, and also spend some time on the  
design of your schema and indexes.

If your main concern is freeform searching, rather than normalised  
data representation, there are probably better solutions out there  
than a relational DB.

I really appreciate the advice! It helps clear up questions about where I should be looking and what the differences are.

I really appreciate the advice!  It helps clear up questions about where I should be looking and what the differences are.


XML and net.loadbang-SQL