I recently found that Jon Rogers, a developer from the UK, released the UnShorten WordPress plugin, which uses TheRealURL to unshorten links displayed by the Twitter Tools plugin.

That’s pretty cool, TheRealURL was designed as a web service with exactly this type of use in mind.

Between this and other users TheRealURL now serves over 40,000 requests a day. It’s nice to see Google App Engine handling that with barely using any of the various daily quotas (except for incoming bandwidth… Need to check on that one):

Jenny Pollak’s foggy window shots.

(Found via FFFFOUND)

i love working on an obscure book. people cling to ideas, because they’re supposed to be vouchers for a million dollars. no, write an obscure book. build something outside all that pressure. i guess treehouses for kids qualify.

_why, Sep 2004

Where’s _why? I do hope all is OK.

I needed this for a project I’m working on, so I added a new feature to TheRealURL unshortening service: JSON/P requests now return the page title (scraped from the HTML <title> tag) as well as its original URL.

For example, http://therealurl.appspot.com/?format=json&url=bit.ly/a returns:

{ "url" : "http://www.apple.com/", "title" : "Apple" }

The plain text format remains as is – nothing but the unshortened URL – so I don’t think there should be any issues for existing API users. Response times don’t appear to be affected much either. If you do get any issues, please let me know in the comments or at niryariv@gmail.com.

Recently I’ve been working on an AJAX based chat application (in development..). The obvious way to do it is send an XMLHttpRequest every few seconds to check for new messages. Unless it’s a particularly animated conversation most requests won’t return any new content, so I added a simple Conditional-GET like system based on the chat’s text size. Here’s the client side implementation:

function refresh_chat() {
	$.ajax({
	  	url: "/chat",
	   	data: "format=xhr&chat_id={{chat_id}}&cur_len=" + chat_content.length,
		  complete: function(xhr){
				if (xhr.status == 200) render_chat(xhr.responseText);
				setTimeout("refresh_chat()", 5000)
		  }
	 });
}

And the server code that handles it:

cur_len = self.request.get("cur_len", 0)
if len(chat.content) == int(cur_len):
	self.error(304) # return 304 Not Modified
else:
	self.response.out.write(chat.content) # return new content

That’s basically the standard approach. Pretty simple, works ok (could be optimized a bit, for example return only the actual new content etc). It’s not exactly an elegant design, though. Trying to use HTTP, designed as a Pull protocol, for an application that requires Push results creates this system of frequent server requests with empty responses, kind of like the “Are we there yet?” conversations with kids on long road trips.

Jack Moffitt’s JSConf talk introduced me to the concept of Long Polling, aka Comet or (with a lot added) BOSH, as a way to simulate HTTP Push. Rather than have the client sending a lot of short, frequent requests and the server responding to each as fast as possible, long polling turns it around: the server holds the requests as long as it can, returning a response only when it has new data or a timeout limit was hit. So, instead of sending request every 3 seconds, for example, you can send one every 30 seconds.

Client side code remains almost the same:

function refresh_chat() {
	$.ajax({
	  	url: "/chat",
	   	data: "format=xhr&chat_id={{chat_id}}&cur_len=" + chat_content.length,
		  complete: function(xhr){
				if (xhr.status == 200) render_chat(xhr.responseText);
				setTimeout("refresh_chat()", 1000);
		  }
	 });
}

But on the server side, there’s a bit of new logic to keep checking for new content while the server holds the response:

cur_len = self.request.get("cur_len", 0)
end_by = int(time.time()) + 30

while int(time.time()) < end_by:
	if len(chat.content) != int(cur_len):
		return self.response.out.write(chat.content) # return new content

	time.sleep(1)

self.error(304) # return 304 Not Modified

If you have any experience building web applications, you’ve spent a lot of effort making sure servers respond quickly to requests. Delaying the response is counter-intuitive, which in itself makes Comet useful to know, if only for its new perspective. However, this also makes production use a bit complicated, since most web server stacks are optimized for maximum requests/second rather than long concurrent requests. Content-rich sites often use separate servers for big media content for this reason, and Comet also has its own server (er “HTTP-based event routing bus”) in Cometd.

I just put up a quick hack I made with FeedVolley (more about FV here), that aggregates Twitter (and other media) feeds coming from inside Iran: Messages From Iran

I don’t know about news value, but it’s pretty cool to be able to refresh that page now and then and get a snapshot of the current mood and happenings, in these possibly historic times there.

It was also cool to find another use for FeedVolley, which I neglected a bit recently ;) I added some page caching on top of the existing feed caching, to allow it to handle some traffic (Slicehost’s 256MB slices seem to start sending swap alerts as soon as traffic rises above negligible). The sources are basically the ones listed here, with a few additional ones I’m trying to find. In fact, if you really want to keep a close watch on what’s going on, you may want to watch the FriendFeed stream – the FeedVolley page is really just an HTML skin to make the feed look a little nicer (hopefully).

(Favorite tweet so far: “@jonobacon IRC is blocked. Tell our regards to Ubuntu Global Jam from Iran. I’m twitting the #iranElection story from a Kubuntu machine :)“. Makes me think of starting to use Twitter again..)

I love this NY Times article not just because it reminds me of my old motorcycle which I enjoyed fixing almost as much as riding (and spent almost as much time doing, anyway), but especially for the arguments it makes about the irrelevance of much of conventional “wisdom” about careers, education and happiness in today’s world. It feels great to read your own views and thoughts put in a coherent, clear form. The Case for Working With Your Hands.

rmmbr is a service I built for myself and recently opened for general use. I wanted a way to send myself notes about things I need to do, items I want to research etc. If just write this on a scrap of paper I usually seem to forget about it, so using Textmarks‘ web-SMS gateway (which is pretty nice btw – reliable and easy to interface with) I wrote a little web app that lets me text these notes to my inbox.

The examples on rmmbr’s home page are pretty much how I use it – I come across some term I want to research or am reminded of something I need to do and send an text message like rmmbr <whatever> to 41411. When I get back home an email is waiting in my Gmail inbox with the text sent, a Google search link for it, and if it’s an address or a date Gmail might also attach its “map this” or “add to calendar” links as well.

To start using it, you need to let rmmbr connect your phone # with your email address by texting rmmbr signup <your@email> to 41411. You’ll get a confirmation email and can start sending yourself messages. Just remember to always start the message with “rmmbr” – it still gets me sometimes..

I don’t know how useful it would be for anyone with an iPhone or a Blackberry, but if you’re like me and prefer to keep your cellphones small, cheap and low maintenance, you might find this a useful solution for your text-to-inbox needs. Enjoy :)

[Update: formerly rmbr, the app/sms code is now rmmbr, to prevent confusion with rmbr.com - which is a different product, not related to rmmbr in any way]

WordPress just added special TED support – I’m not a fan of TED’s exclusive/trendy approach, but it’s a good opportunity to post this excellent talk by James Howard Kunstler.

The lesson of design which considers its users, environment and resources vs. design that is isolated, contrived and wasteful holds true to software as well, from the UI to the server.

I discovered JSONP just recently, following Chris‘ comment. Though I initially didn’t intend to support JSON, JSONP made enough difference that I rewrote most of the TheRealURL code (all 20 lines of it) to support it. Since it took me some time to figure out JSONP initially, perhaps a quick guide might help those who follow.

JSONP allows you to make an HTTP request outside your own domain, which enables consuming Web Services from JavaScript code. It relies on a JS quirk: while XMLHttpRequest is blocked from making external requests, there’s no such limit on <script> elements. What JSONP does is add a <script src=> element to the DOM, with the external URL as the SRC target.

To serve JSONP simply return the JSON data inside a function. e.g., this JSON:

{ "hello" : "Hi, I'm JSON. Who are you?"}

Becomes:

some_function({ "hello" : "Hi, I'm JSON. Who are you?"})

(The reason is that the latter is actually code that will run inside the created by the JSONP client, so it needs to be executable code rather than plain JSON data)

some_function is provided by the calling client, usually in the ‘callback’ parameter. So, a query like this:

get_jsonp?callback=getthedata

Should return:

getthedata({ "hello" : "Hi, I'm JSON. Who are you?"})

On the server side, this means adding some code similar to this:

// assume $json holds the JSON response
if ($GET['callback'] != '') $json = $GET['callback']."( $json )";
return $json;   // my PHP is rusty but you know what I mean

On the client side, modern JS frameworks include JSONP support (or you can DIY). For example, in jQuery <= 1.2 adding &callback=? to the query string in getJSON method’s URL sends a JSONP request.(jQuery transparently replaces the ‘?’ with a unique string). Here’s how you get the unshortened URL for ‘bit.ly/a’ using therealurl:

$.getJSON("http://therealurl.appspot.com?format=json&url=bit.ly/q&callback=?",
	function(data){ alert(data.url) }
);

That’s about it. JSONP probably won’t feature in the next Beautiful Code edition and obviously you need to watch the URLs you’re accessing so you don’t get malicious JS code executed, but, until cross site XHR is resolved, JSONP can get the job done.