Welcome Waxy/CrowdsMachine Users

Didn’t expect you so soon, great to have you here :)

Had some server issues earlier, but adding some memory to Crowds’ slice seems to have fixed it now. If you get any problems, or have any feedback, please feel free to post a comment here or email me at niryariv@gmail.com. Hope you enjoy Crowds Machine!

HelloWorldChat In Stanford Peace Dot Directory

HelloWorld Chat is a project I’ve been working on with my friend and designer extraordinaire Aviva. Basically it’s a “chat online with a stranger” site like Omegle & co, but we try to match between users from different countries (based on their IP address). Once we reach a certain level of traffic, we could start adding more sophisticated matching rules and connect between users of specific nations, based on current world events and so on.

We haven’t done much in way of publicizing the site, so it was great to be accepted into Stanford’s Peace Dot directory. Peace Dot only adds one organization per day to the directory, and we’re excited to finally get in :) They even do a little welcome video for each site, here’s Standord’s BJ Fogg welcoming us:

Thanks, Peace Dot!

Technically, HelloWorld is a simple Google App Engine based chat, written in Python. The focus was less on the chat code and more on making the application easy to customize and adapt for various sites. The look can be completely changed by editing two HTML files, and being App Engine based makes deployment extremely simple and cheap.

Sites built on top of HelloWorld might be used to foster dialog between sides in a conflict, aid in reconciliation phases, or help people going through their own struggle with a disease or addiction talk with others in the same situation. The anonymity allows users to speak frankly where it may often be difficult or dangerous to do so, and hopefully some may later move on to talk via Skype or Email – our goal is to be the easily accessed and risk free gateway to a dialog.

While forums and multi-user chats have important roles, a one to one conversation has a unique value of its own. Talking one to one makes slogans and cliches a lot harder to hide behind. It is a lesson I learned myself in the summer of 2006, when I happened into an IM conversation with a Lebanese person while both our parents’ towns were under fire by the other side. That experience was the inspiration for HelloWorld, I hope this tool would allow many more such conversations to take place.

Fall in New York

Generally beautiful, New York City is even beautiful-er when autumn foliage comes around. So it was cool to discover this Flickr set of scanned leaves from Morningside Park – literally 3 blocks from where we live.

(Via what about the plastic animals, via Outside.in)

Google App Engine Quotas on > 430k Requests/Day

[UPDATE: Posted this yesterday for >140k requests/day, updating now with >430k requests..]

TheRealURL had a particularly busy Thanksgiving Day Black Friday for some reason. Just for curiosity’s sake, here’s the quota usage for a GAE application under a relatively high load. The quotas that aren’t pictured are all on %0. Billing is now turned on (for the Incoming Bandwidth) but limited to up to $1/day – and was never used yet.

Bear in mind this is a pretty minimal app, of course, but still – not too shabby.

UnShorten WordPress Plugin Uses TheRealURL

I recently found that Jon Rogers, a developer from the UK, released the UnShorten WordPress plugin, which uses TheRealURL to unshorten links displayed by the Twitter Tools plugin.

That’s pretty cool, TheRealURL was designed as a web service with exactly this type of use in mind.

Between this and other users TheRealURL now serves over 40,000 requests a day. It’s nice to see Google App Engine handling that with barely using any of the various daily quotas (except for incoming bandwidth… Need to check on that one):

So Beautiful.

Jenny Pollak’s foggy window shots.

(Found via FFFFOUND)

Obscure

i love working on an obscure book. people cling to ideas, because they’re supposed to be vouchers for a million dollars. no, write an obscure book. build something outside all that pressure. i guess treehouses for kids qualify.

_why, Sep 2004

Where’s _why? I do hope all is OK.

TheRealURL Adds Page Titles

I needed this for a project I’m working on, so I added a new feature to TheRealURL unshortening service: JSON/P requests now return the page title (scraped from the HTML <title> tag) as well as its original URL.

For example, http://therealurl.appspot.com/?format=json&url=bit.ly/a returns:

{ "url" : "http://www.apple.com/", "title" : "Apple" }

The plain text format remains as is – nothing but the unshortened URL – so I don’t think there should be any issues for existing API users. Response times don’t appear to be affected much either. If you do get any issues, please let me know in the comments or at niryariv@gmail.com.

The Long Poll: AJAX Push(like) Chat with Comet

Recently I’ve been working on an AJAX based chat application (in development..). The obvious way to do it is send an XMLHttpRequest every few seconds to check for new messages. Unless it’s a particularly animated conversation most requests won’t return any new content, so I added a simple Conditional-GET like system based on the chat’s text size. Here’s the client side implementation:

function refresh_chat() {
	$.ajax({
	  	url: "/chat",
	   	data: "format=xhr&chat_id={{chat_id}}&cur_len=" + chat_content.length,
		  complete: function(xhr){					
				if (xhr.status == 200) render_chat(xhr.responseText);
				setTimeout("refresh_chat()", 5000)
		  }
	 });	
}

And the server code that handles it:

cur_len = self.request.get("cur_len", 0)
if len(chat.content) == int(cur_len):
	self.error(304) # return 304 Not Modified
else:
	self.response.out.write(chat.content) # return new content

That’s basically the standard approach. Pretty simple, works ok (could be optimized a bit, for example return only the actual new content etc). It’s not exactly an elegant design, though. Trying to use HTTP, designed as a Pull protocol, for an application that requires Push results creates this system of frequent server requests with empty responses, kind of like the “Are we there yet?” conversations with kids on long road trips.

Jack Moffitt’s JSConf talk introduced me to the concept of Long Polling, aka Comet or (with a lot added) BOSH, as a way to simulate HTTP Push. Rather than have the client sending a lot of short, frequent requests and the server responding to each as fast as possible, long polling turns it around: the server holds the requests as long as it can, returning a response only when it has new data or a timeout limit was hit. So, instead of sending request every 3 seconds, for example, you can send one every 30 seconds.

Client side code remains almost the same:

function refresh_chat() {
	$.ajax({
	  	url: "/chat",
	   	data: "format=xhr&chat_id={{chat_id}}&cur_len=" + chat_content.length,
		  complete: function(xhr){					
				if (xhr.status == 200) render_chat(xhr.responseText);
				setTimeout("refresh_chat()", 1000);
		  }
	 });
}

But on the server side, there’s a bit of new logic to keep checking for new content while the server holds the response:

cur_len = self.request.get("cur_len", 0)
end_by = int(time.time()) + 30

while int(time.time()) < end_by:
	if len(chat.content) != int(cur_len):
		return self.response.out.write(chat.content) # return new content
 	
	time.sleep(1)

self.error(304) # return 304 Not Modified

If you have any experience building web applications, you’ve spent a lot of effort making sure servers respond quickly to requests. Delaying the response is counter-intuitive, which in itself makes Comet useful to know, if only for its new perspective. However, this also makes production use a bit complicated, since most web server stacks are optimized for maximum requests/second rather than long concurrent requests. Content-rich sites often use separate servers for big media content for this reason, and Comet also has its own server (er “HTTP-based event routing bus”) in Cometd.

FeedVolley: Messages From Iran

I just put up a quick hack I made with FeedVolley (more about FV here), that aggregates Twitter (and other media) feeds coming from inside Iran: Messages From Iran

I don’t know about news value, but it’s pretty cool to be able to refresh that page now and then and get a snapshot of the current mood and happenings, in these possibly historic times there.

It was also cool to find another use for FeedVolley, which I neglected a bit recently ;) I added some page caching on top of the existing feed caching, to allow it to handle some traffic (Slicehost’s 256MB slices seem to start sending swap alerts as soon as traffic rises above negligible). The sources are basically the ones listed here, with a few additional ones I’m trying to find. In fact, if you really want to keep a close watch on what’s going on, you may want to watch the FriendFeed stream – the FeedVolley page is really just an HTML skin to make the feed look a little nicer (hopefully).

(Favorite tweet so far: “@jonobacon IRC is blocked. Tell our regards to Ubuntu Global Jam from Iran. I’m twitting the #iranElection story from a Kubuntu machine :)“. Makes me think of starting to use Twitter again..)