Searchtweet - Design patches that respond to twitter posts.

Please note that we are no longer taking submissions for Tools and no longer supporting the Tools section, so you may run into broken files and URLs. You can contribute to the Cycling '74 Community by submitting a Package or using the forums instead.

Update: Version 1.2 is available to download, adding support for custom Twitter API keys. This was requested by some users to increase reliability. Version 1.0 and 1.1 do not work any more due to changes in Twitter's API.

Searchtweet is a mxj external that searches twitter with a given keyword and returns a desired maximum number of most recent posts, with username, date and twitter text body routed to separate outlets. The object returns a specified number of most recent twitter posts that contain a given keyword, #hashtag, or @usertag.

Advanced search queries also work:
https://support.twitter.com/articles/71577-using-advanced-search

Searchtweet uses the twitter4j library (http://twitter4j.org) programmed by Yusuke Yamamoto.

Note:
Unless custom API keys are entered, the plugin will default to its own authentication keyset. This may limit the number of searches per given time. Eg. You must optimize your search frequency to avoid hitting the API rate-limits. (Ex. Do not connect a [metro 10] to the inlet, your IP will be banned in no second)

For more info:
https://dev.twitter.com/docs/faq

Have fun,
Gokce

searchtweet1.2.zip

zip

Tool Files

http://cycling74.com/files/2011/04/searchtweet1.2.zip

bennyfromtheblock

This has got me thinking! :)

tkzic

This is fantastic. Thank you.

RT White

Excellent object and tool! Forgive my naivete as the world of APIs is pretty new to me, but is there a way to limit the output of this device to a single language? The patch I'm working on is mostly pertinent in English.

Thanks for any help.

Gokce Kinayoglu

Hi RT, and all,
The plugin works using the basic search functionality of the twitter API. I do not know if there is a way to limit searches based on country or language through specific search keywords.
Also, the twit4j offers a push-based update functionality which I did not implement in this plugin. If there are any exciting projects that needs that kind of a more "realtime" update feel free to contact me in person or here.

RT White

Thank you Gokce! I will let you know if I get anything interesting working.

David Benson

Hey all,

Just for a clarification, if you are using the standard search parameters on Twitter you can search by language and by region. If you are using the Twitter Streaming API, however, the query language is different. I have yet to dig into this patch, but I wanted to see if I could be of service.

For the standard search terms, using lang:en should limit your searches to english only.

For geolocation use: geocode:40.7484918,-73.98428659999999,25km. You will need to use a third party service to determine the Latitude and Longitude of your location (The one I posed is Manhattan) and you can use the km parameter to alter the distance.

I like to use http://geocoder.us/ for finding a latitude and longitude. Just enter an address and it should output what you want.

Hopefully that helps a little to answer your question, @Gokce.

Gokce

Thanks for the information David! This is amazing.

The search parameters indeed work well with the script! There is a small trick. Max will interpret the commas in the message as separators. The message sent to the script needs to be formatted as shown below with backslashes before each comma, such as:

search geocode:40.748433\,-73.985656\,1km 1

Put that in a message box and wire it to the mxj directly. (Do not use the search box in the example patch.) This should give the most recent tweet sent from an area within 1km radius around the Empire State Building in New York. Tested, and here's the result I received:

"fangirlDANosaur"
"RT @danisnotonfire: I'm currently having that classic tourist fangirl moment where I realise I know all the buildings in Manhattan from every movie/game ever. "

Looks like she is in New York. Exciting! Now go play!

Gokce

Gokce Kinayoglu

Thanks for the information David! This is amazing.

search geocode:40.748433,-73.985656,1km 1

"fangirlDANosaur"
"RT @danisnotonfire: I'm currently having that classic tourist fangirl moment where I realise I know all the buildings in Manhattan from every movie/game ever. "

Looks like she is in New York. Exciting! Now go play!

Gokce

Thierry

Hi David and Gokce,

Thanks for your precisions about language filtering and geocode. I would like to filter the tweets according to their language. But, as "search geocode:40.748433\,-73.985656\,1km 1" works perfectly for me, "search voyage lang:fr 1" returns the following errors :

java.lang.UnsupportedOperationException: does not contain an int
at com.cycling74.max.Atom.getInt(Atom.java:572)
at searchtweet.search(searchtweet.java:34)

Did I miss something ?
Thanks a lot if you would have an idea...
-Thierry

PS: Gokce: I left a question on your FB page about language filtering, before having found this page, sorry :)

Thierry

Hi David and Gokce,

I finally found the right syntax to filter by language, which is: search "voyage lang:fr" 10
It needed quotes.
Thanks, Best
-Thierry

C74 Ginger

Hi David and Gokce,

Thanks for your precisions about language filtering and geocode. I would like to filter the tweets according to their language. But, as "search geocode:40.748433,-73.985656,1km 1" works perfectly for me, "search voyage lang:fr 1" returns the following errors :

java.lang.UnsupportedOperationException: does not contain an int
at com.cycling74.max.Atom.getInt(Atom.java:572)
at searchtweet.search(searchtweet.java:34)

Did I miss something ?
Thanks a lot if you would have an idea...
-Thierry

PS: Gokce: I left a question on your FB page about language filtering, before having found this page, sorry :)

KAIBE

Hi Gotkce and all, is there any advice you can give me on a similar kind of thing I'm doing here? https://cycling74.com/forums/text-input-for-maxmsp

Gokce Kinayoglu

Hi Kaibe,

I suggest you to load the plugin and play with it before starting out with your project. Once you are able to get the text output that you need from searchtweet, the rest is figuring out how to "parse" it. There are several functions that does exactly that in MSP and there are also some external libraries for string parsing and search. Good luck!

Gokce

DRKOSS

Super cool - thanks for posting!

DRKOSS

Hi - looks like something has gone sour with the api - now i get this message:

TwitterException{exceptionCode=[26d84cad-002e486e], statusCode=410, retryAfter=0, rateLimitStatus=null, version=2.2.0}

any ideas - much obliged!

shane

DRKOSS

Aha - twitter changed the API!

We will begin 410ing inbound API v1 traffic now. We will gradually ramp up over the next several minutes. More info: https://dev.twitter.com/docs/faq#17750

Have a show using this object in a few weeks, so if anyone has an idea, i would be very grateful and have no other choice but to buy you some beers in NYC!

Gokce Kinayoglu

I will take a look at this - hopefully soon!

DRKOSS

Cheers!

wraithers

Hey, I too noticed that my project is dead due to the API change. This is going to kill my graduation project pretty much =/

I downloaded the latest stable version of twitter4j, updated the twitter4j.jars in the libs folder, added twitter4j.examples to a Java project in Eclipse, following http://compusition.com/web/articles/maxmsp-eclipse, and added only the following to SearchTweets.jar:
import com.cycling74.max.*;
and
... extends MaxObject {

If I just copied the SearchTweets.class (not java) file to the Max folder, it just returned "Could not load class 'SearchTweets'"

So I also tried to to create an object in Max: [mxj twitter4j.examples.search.SearchTweets] and I'm getting the following in the Max window:
(mxj) Class twitter4j.examples.search.SearchTweets is not a subclass of com/cycling74/max/MaxObject

Do I need to work on adding INLETS and OUTLETS? Did I miss out anything else I should add?
What can I do to get this working? Any help would be supremely appreciated!!

Gokce Kinayoglu

Guys do not panic. I will check this today.

Gokce Kinayoglu

Done :). Please download and *test* the new version. User discretion advised! Let me know how it works.

DRKOSS

First off - thanks so much for even taking a look - i'll never forget my 12th grade math teacher's sign on her desk - a crisis on your part is not a crisis for me!

very cool - and please let me know if you are ever in NYC - you have some libations coming your way.

cheers!

wraithers

You, sir, are aces.. and spades.. and I can't really explain how much this means to me! Let's just say you saved my college Graduation Project - and also me, from breaking my head with Java and Eclipse.. Hehe..

DRKOSS put it very well, and I agree 100% :)

I'm all the way over in India, but if you do ever pop over here, I will be glad to oblige a trip around the city to see the sights and sounds of fair Bangalore, followed by drinks.

Now to get the final bits of my project wrapped up, cheers!

Gokce Kinayoglu

You are welcome. It just took slightly more effort than I expected because twitter added an application authorization requirement which was not there before. But you should really thank twitter4j and its author, who seems to have kept his java library up to date till now.

greendtea

Can anyone confirm if this is still working? I can't seem to get it to even load up on max 6 windows 8.1

A blank max window opens up and the patcher does not load.

Christopher

Hmm, doesnt seem to be working for me. Here's what's in the Max window:

java.lang.IllegalStateException: Authentication credentials are missing. See http://twitter4j.org/configuration.html for the detail.
    at twitter4j.TwitterBaseImpl.ensureAuthorizationEnabled(TwitterBaseImpl.java:200)
    at twitter4j.TwitterImpl.get(TwitterImpl.java:1833)
    at twitter4j.TwitterImpl.search(TwitterImpl.java:282)
    at searchtweet.search(searchtweet.java:38)
java.lang.IllegalStateException: Authentication credentials are missing. See http://twitter4j.org/configuration.html for the detail.
    at twitter4j.TwitterBaseImpl.ensureAuthorizationEnabled(TwitterBaseImpl.java:200)
    at twitter4j.TwitterImpl.get(TwitterImpl.java:1833)
    at twitter4j.TwitterImpl.search(TwitterImpl.java:282)
    at searchtweet.search(searchtweet.java:38)
java.lang.IllegalStateException: Authentication credentials are missing. See http://twitter4j.org/configuration.html for the detail.
    at twitter4j.TwitterBaseImpl.ensureAuthorizationEnabled(TwitterBaseImpl.java:200)
    at twitter4j.TwitterImpl.get(TwitterImpl.java:1833)
    at twitter4j.TwitterImpl.search(TwitterImpl.java:282)
    at searchtweet.search(searchtweet.java:38)

greendtea

Yeah something must have changed with the API, it's not working for me anymore either. It's a shame, I was really hoping to utilize it in my thesis project for school.

nnimar

Try downloading the latest version of Twitter4J (http://twitter4j.org/en/index.html) and replace the old by the new in your java folder, it works again for me !

Sam Sturtivant

Is this still working? I keep getting an error in my max window even after changing the Twitter4J java files to the new ones.

nnimar

It's working for me but indeed you have some warnings from log4j the first time you run it.

Gokce Kinayoglu

Just updated the pack with the new twit4j libraries. Tested in Max5 and seems to be working fine again. NNMAR what are the error messages you are getting?

mmake

Hi Gokce, thanks for your work on this. I get a 'Could not load class 'searchtweet' -error. Is there suppose to be a searchtweet.class in the zip file? You have it in the maxobjects download, but i guess that is the old one that doesn't work anymore? Thanks again.

nnimar

Thanks GOKCE, when I do the first search, I get log4j:WARN No appenders could be found for logger (twitter4j.internal.http.HttpClientlmpl)

Then it goes on with Please initialize the log4j system properly and see the logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Matthew Lewis

The readme says "copy the searchtweet.class under [maxmsp]/Cycling '74/java/classes folder"... there is no searchtweet.class file included in the current zip file?

Gokce Kinayoglu

Thanks Matthew, I must have erased the class file accidentally while rushing the update. Now all files are back in place.

Matthew Lewis

Works great now, thanks!

mmake

Thank you for the update!

anghelcristiana

Hey ! I have followed all the steps and tried to run the patch: it opens, but when I search for something, I get this error:
java.lang.NoClassDefFoundError: com/google/appengine/api/urlfetch/HTTPRequest
at twitter4j.AlternativeHttpClientImpl.handleRequest(AlternativeHttpClientImpl.java:43)
at twitter4j.HttpClientBase.request(HttpClientBase.java:53)
at twitter4j.HttpClientBase.get(HttpClientBase.java:71)
at twitter4j.TwitterImpl.get(TwitterImpl.java:1968)
at twitter4j.TwitterImpl.search(TwitterImpl.java:293)
at searchtweet.search(searchtweet.java:48)
Do you know what this means? It's probably something really stupid that I'm missing, but I'm just starting out and I could really use some help.
Thanks!

liseliselise

Hi there, I've installed the latest version of twitter4j (4.0.2) and replaced the files in the java>lib folder.. I still get all these errors:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Relevant discussions can be found on the Internet at:
    http://www.google.co.jp/search?q=506c3b98 or
    http://www.google.co.jp/search?q=105d1087
TwitterException{exceptionCode=[506c3b98-105d1087 63e3f388-fb44fc20 63e3f388-fb44fc20 63e3f388-fb44fc20], statusCode=-1, message=null, code=-1, retryAfter=-1, rateLimitStatus=null, version=4.0.2}
    at twitter4j.HttpClientImpl.handleRequest(HttpClientImpl.java:178)
    at twitter4j.HttpClientBase.request(HttpClientBase.java:53)
    at twitter4j.HttpClientBase.get(HttpClientBase.java:71)
    at twitter4j.TwitterImpl.get(TwitterImpl.java:1556)
    at twitter4j.TwitterImpl.search(TwitterImpl.java:247)
    at searchtweet.search(searchtweet.java:48)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid c
ertification path to requested target
    at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(Unknown Source)
    at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Unknown Source)
    at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Unknown Source)
    at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(Unknown Source)
    at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(Unknown Source)
    at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Unknown Source)
    at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at java.net.HttpURLConnection.getResponseCode(Unknown Source)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
    at twitter4j.HttpResponseImpl.(HttpResponseImpl.java:35)
    at twitter4j.HttpClientImpl.handleRequest(HttpClientImpl.java:142)
    ... 5 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(Unknown Source)
    at sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
    at sun.security.validator.Validator.validate(Unknown Source)
    at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.validate(Unknown Source)
    at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
    at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
    ... 20 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
    at java.security.cert.CertPathBuilder.build(Unknown Source)
    ... 26 more
Failed to search tweets: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I haven't seen anyone post about these specific errors here.
Thanks to anyone who can help me with this!

bob123

Hi Gokce, first of all thank you so much for the searchtweet object! It is the backbone of my duo "Twittercore", we would have missed a lot of fun without it :) We are using it since 2012 for our performances, working great so far.

My questions is: We performed tonight and after one set got a 429 error from the Twitter API in the max window. Couldn't search for tweets any more. Waited for some time, no luck. Tried again later from a different IP address and it seemed to work again. We also performed yesterday for way longer then today without problems. If I understood correctly twitter allows 180 requests in a window of 15 min now and I'm very sure we didn't even come close to a 100.
Do you have an idea what that could be? Does searchtweet function like a regular user in the eyes of the API and can we get banned? How about two people using the object at the same moment on different locations, do their requests count like they are coming from the same person? Say both do a 100 in 15 min, does that count as 200 then for twitter because we use the same object in max?
I'm using max 5.1.9 / searchtweet 1.1b, twitter4j 4.0.1
I'm on a mac, OS 10.9.4

We have quite a big gig in Cape Town next week (we live in Amsterdam, Netherlands) and are pretty nervous about running into problems suddenly.
Thank's again an hope you can help us!

Spa

hi Gokce,
would you mind releasing the source so it can be implemented with more items.
Thanks

Gokce Kinayoglu

Hi guys,

Sorry for not responding earlier. I do not get notifications when there is a post on this thread. I will look into this problem soon. About releasing the source code my answer is, the code is so simple that anyone who understands how mxj extensions are programmed and compiled can rewrite it in minutes because it is basically wrapping the twit4j library (!). For the others I do not recommended that they touch the code and create rogue versions. :)

The 429 error means "Too many search requests". this behaviour is highly anticipated and the only way to deal with this is to have the code use custom search "keys" for each plugin user. Now the same key is used practically for all users of the plugin. Therefore it seems that anybody who uses this app shares the same search quota. This results in intermittent errors.

I will look into this in the next week and hopefully come back with a better version altogether - that allows realtime (JSON) feeds from twitter, instead of having to ping the object each time one needs a result.

Thanks for your patience in advance ;) !

Matthew Lewis

If you eventually could add the option to enter our own search key, that would be great. thanks.

Gokce Kinayoglu

Liselise the mxj comes with the necessary Twitter4j classes. You do not need to seperately install Twitter4j. Perhaps you have a version mismatch. The current version (1.1b) of searchtweet comes with twitter4j version 4.0.1 .

Spa

Well Gokce,
there's simple and simple.
so a simple source code will have gave me a base to understand the wrap... and expand it in eclipse
I'm not very fluent in java...
Waiting for your version with user key, perhaps with source...
Thanks for the work

Gokce Kinayoglu

Sorry guys, it has been an incredibly busy at work the last few months. I intended to set aside some time to write a live feed plugin, but it never happened. I'm now working on a version which will accept custom keys as Matthew Lewis suggested. It should not take too long. Just writing to let you know I did not forget about you :)

Gokce Kinayoglu

There you go. Version 1.2 is up there, now allowing custom Twitter API keys. Better very late than never!

Benjamin Saphier

Hi Gokce,
I'm trying to create a max patch to "sonify" data from twitter. What I'd like to do is find the number of times a specified hashtag or keyword is mentioned, within a specific timeframe, and use that number in my Max/MSP project.

The searchtweet object, as is, doesn't allow me to get the information from twitter that I'm trying to use.

I'm wondering if you could point me in the right direction. Or maybe you know how I would alter the searchtweet java code to get this data from twitter?

I hope I'm making sense, and would really appreciate any help or words of advice!!

Gokce Kinayoglu

Hi Benjamin,

I'm not aware of a twitter-specified keyword frequency counter, but you can probably do this manually.

Searchtweet accepts a desired maximum number of results as one of its parameters. If you submit a large max-number, let's say 50, if will return the last 50 times that keyword was mentioned. If the keyword has been mentioned less than 50 times ever since the beginning of time, then it will return less number of results. Depending on the time period you care about (the last hour, the last day, the last minute etc.) you can filter the returned messages by the time value returned in each post and do a count.

Eric

Hi Gokce,

First of all thank you for your great tool. I have a probably newbish question though, I don't understand how to output "mxj tweet body text" in multiple messages, unpack or iter have no effect, all I get is the last tweet when I want to get all 10 into a list or 10 individual messages.

This is also a problem, if I put a regexp to fiilter out all retweet from the search result, I just get a blank message. if the last tweet is a retweet.

Thank you.

Eric

Hi Gokce,

Thank you for this wonderful tool.

I don't understand how to display a constant stream of information though, by example, having "search $1 50" it will display the 50 last tweets in console, but in the message box it'll only display the last one. I've tried zl , unpack, and text but none of those worked out. What i'd like to have is the 50 last tweet showing one per one in the message box, zl.stream and zl.queue did not work for that. There must be something I'm missing and this is probably a newbish question.

Thanks a lot.

Max Wortman

Hey, there!
Is there a newer version of Searchtweet available?
No matter what I change the search term to it keeps bringing up the same tweet...

I've got a bucket set up to see if the tweets have the same amount of characters... If so, it sends a retrigger to the initial search bang. Every time I open the patch I get stuck on a certain tweet at some point...

Please help!
If this project has been abandoned, are there any other alternatives built off of this beautiful external? : )

Screen-Shot-2016-05-04-at-1.10.47-AM.png

png

Gokce Kinayoglu

Eric: Searchtweet will just burst out all the responses in a row. The beauty of Max is you do not need to store these responses in an array or something, unless you absolutely need to. if you attach an object that does anything with the output, all tweets will be processed in a row. You can try this with a print message. The display will show the latest received input as it is supposed to. I suggest looking at zl.group if you need to store multiple messages in a buffer. Good luck exploring Max!

Max: Your solution to do a retrigger everytime a tweet with the same number of characters is received is a baaaaad idea. :D This would just cause searchtweet to hit the twitter API million times in a second. No wonder you stop getting new responses after a while. Also, good luck exploring Max! ;)

Tony

Has the Twitter Api changed again as my once working searchtweet is not working anymore?!

Gwen Taualai

Hi Gokce,

Thanks you for this awesome tool.

I've been playing around with searchtweet and trying to get it to give a bang every time a particular hashtag is used. The hashtag I'm using is very popular (#i), once every 20secs or so.

I understand that searchtweet will give me the latest tweet out of a list, but how do I get it to give me each tweet as it comes in? I'm just hitting the API limits within minutes. Is this at all possible to be done in a live way?

Thanks for your help,

Gwen

DRKOSS

Aha - twitter changed the API!

We will begin 410ing inbound API v1 traffic now. We will gradually ramp up over the next several minutes. More info: https://dev.twitter.com/docs/faq#17750

Have a show using this object in a few weeks, so if anyone has an idea, i would be very grateful and have no other choice but to buy you some beers in NYC!

Linden

this is great! i wonder if some tweets get shortend?? it seems messages get truncated after 140 types or so... am i missing something?

Billie-Grace Dunk

Hi - wondering if this still works for anyone else? I am having issues getting it working with error messages in max console along the lines of 'trustAnchors parameter must not be empty'. New to world of coding and not sure what this means. If someone could help that would be fantastic!

Clau

Hi Gokce,

first of all thank you for this fantastic tool!!

I have an installation using this tool and last time I presented was May 2017 and everything was working fine, I didn't get errors at all, but now, I updated my system to high sierra and updated Java and the patch is not working any more , I get a long list of errors... (see below)
could you please give me some advice? I'd really appreciate!
thanks in advance!

Clau

Gokce Kinayoglu

Hi all, looking at the errors posted here it seems like the problem arises from a Java 8 issue, or certificate incompatibility with the new Mac OS. Have any PC users recently tried the plugin?

Also, because I am not able to actively maintain the project, I decided to open up the source code to the community. Here's the github link:

https://github.com/gokcekinay/searchtweet

ghostique

thanks bro!

Published

Author

Gokce Kinayoglu