Drinking from the Streaming API

Thursday, 28 February 2013

Today we’re open-sourcing the Hosebird Client (hbc) under the ALv2 license to provide a robust Java HTTP library for consuming Twitter’s Streaming API. The client is full featured: it offers support for GZip, OAuth and partitioning; automatic reconnections with appropriate backfill counts; access to raw bytes payload; proper retry schemes, and relevant statistics. Even better, it’s been battle-tested in production by our internal teams. We highly recommend you take advantage of the Hosebird Client if you plan on working with the Streaming API.

Using Hosebird

The Hosebird Client is broken into two main modules: hbc-core and hbc-twitter4j. The hbc-core module uses a simple message queue that a consumer can poll for messages. The hbc-twitter4j module lets you use the superb Twitter4J project and its data model on top of the message queue to provide a parsing layer.

The first step to use Hosebird is to setup the client using the ClientBuilder API:

// Create an appropriately sized blocking queue
BlockingQueueString> queue = new LinkedBlockingQueueString>(10000);
// Authenticate via OAuth
Authentication auth = new OAuth1(consumerKey, consumerSecret, token, secret);
// Build a hosebird client
ClientBuilder builder = new ClientBuilder()
   .hosts(Constants.STREAM_HOST)
   .authentication(auth)
   .endpoint(new StatusesSampleEndpoint())
   .processor(new StringDelimitedProcessor(queue))
   .eventMessageQueue(queue);
Client hosebirdClient = builder.build();

After we have created a Client, we can connect and process messages:

client.connect();
while (!client.isDone()) {
 String message = queue.take();
 System.out.println(message); // print the message}

Hosebird Examples

We recommend you learn from the examples on GitHub or contribute your own.

If you want a quick example, set these properties in hbc-example/pom.xml:
SECRET
SECRET
SECRET
SECRET

Then you can run this command on the command line:
mvn exec:java -pl hbc-example

This will connect to the sample stream API and print 1000 JSON items from the API.

Acknowledgements
The Hosebird Client was primarily authored by Steven Liu (@steven) and Kevin Oliver (@kevino). We’d also like to thank the @TwitterAPI team for their thoughtful suggestions and help.

On behalf of the Hosebird team,
- Chris Aniszczyk, Manager of Open Source (@cra)