plaintext

Recently I’ve been working on an AJAX based chat application (in development..). The obvious way to do it is send an XMLHttpRequest every few seconds to check for new messages. Unless it’s a particularly animated conversation most requests won’t return any new content, so I added a simple Conditional-GET like system based on the chat’s text size. Here’s the client side implementation:

function refresh_chat() {
	$.ajax({
	  	url: "/chat",
	   	data: "format=xhr&chat_id={{chat_id}}&cur_len=" + chat_content.length,
		  complete: function(xhr){
				if (xhr.status == 200) render_chat(xhr.responseText);
				setTimeout("refresh_chat()", 5000)
		  }
	 });
}

And the server code that handles it:

cur_len = self.request.get("cur_len", 0)
if len(chat.content) == int(cur_len):
	self.error(304) # return 304 Not Modified
else:
	self.response.out.write(chat.content) # return new content

That’s basically the standard approach. Pretty simple, works ok (could be optimized a bit, for example return only the actual new content etc). It’s not exactly an elegant design, though. Trying to use HTTP, designed as a Pull protocol, for an application that requires Push results creates this system of frequent server requests with empty responses, kind of like the “Are we there yet?” conversations with kids on long road trips.

Jack Moffitt’s JSConf talk introduced me to the concept of Long Polling, aka Comet or (with a lot added) BOSH, as a way to simulate HTTP Push. Rather than have the client sending a lot of short, frequent requests and the server responding to each as fast as possible, long polling turns it around: the server holds the requests as long as it can, returning a response only when it has new data or a timeout limit was hit. So, instead of sending request every 3 seconds, for example, you can send one every 30 seconds.

Client side code remains almost the same:

function refresh_chat() {
	$.ajax({
	  	url: "/chat",
	   	data: "format=xhr&chat_id={{chat_id}}&cur_len=" + chat_content.length,
		  complete: function(xhr){
				if (xhr.status == 200) render_chat(xhr.responseText);
				setTimeout("refresh_chat()", 1000);
		  }
	 });
}

But on the server side, there’s a bit of new logic to keep checking for new content while the server holds the response:

cur_len = self.request.get("cur_len", 0)
end_by = int(time.time()) + 30

while int(time.time()) < end_by:
	if len(chat.content) != int(cur_len):
		return self.response.out.write(chat.content) # return new content

	time.sleep(1)

self.error(304) # return 304 Not Modified

If you have any experience building web applications, you’ve spent a lot of effort making sure servers respond quickly to requests. Delaying the response is counter-intuitive, which in itself makes Comet useful to know, if only for its new perspective. However, this also makes production use a bit complicated, since most web server stacks are optimized for maximum requests/second rather than long concurrent requests. Content-rich sites often use separate servers for big media content for this reason, and Comet also has its own server (er “HTTP-based event routing bus”) in Cometd.

WordPress just added special TED support – I’m not a fan of TED’s exclusive/trendy approach, but it’s a good opportunity to post this excellent talk by James Howard Kunstler.

The lesson of design which considers its users, environment and resources vs. design that is isolated, contrived and wasteful holds true to software as well, from the UI to the server.

[UPDATED on April 21st, 2009 to reflect the JSON/P additions. Since it's <24 hours after the initial release, I hope it won't cause anyone problems.]

The Real URL began as a joke – after discovering, while working on another project, over 80 URL shortening services, I figured there must be room for a service that un-shortens all these URLs. (The web is overflowing with hype and blog posts/articles complaining about it just add to the noise, so it’s better to make your point by building something. My favorite example is the Twittering Office Chair).

Turns out there are already several out there: (eg, trueurl) but I built it anyway, since I had a slightly different approach in mind. The Real URL is meant to be used as a web service rather than on its own. It returns the “real” URL in either raw text, JSON or JSONP format – examples and details are on the homepage. (I added JSON mostly for JSONP, per Chris’ comment – admittedly I didn’t even know it existed ;) This enables cross site JS requests which might actually make The Real URL useful.

While I do want The Real URL to be solid and reliable in the long term, I don’t want to spend much time/money keeping it up. It’s a sustainability issue – building a system that will work reliably over a long time while requiring minimal care and resources. I made a few design decisions to that end:

• Keep it simple (always a good idea). Real URL does only one thing and is accessible in only one way (the homepage demo uses XHR to access the service, to keep it so). It now supports text/JSON/JSONP, but it’s just the same output formatted differently. Sometimes you give up some elegance to make the product useful. As in the following item:
• Deploy with Google’s App Engine. Initially it was nice, super-minimal Sinatra code. Unfortunately Google App Engine doesn’t support Ruby yet and there’s no service that offers comparable cost/stability ratio, so I rewrote in slightly less minimal Python for GAE.
• Use App Engine’s domain (therealurl.appspot.com). Buying a domain and keeping it renewed isn’t a big deal, but it still requires some attention – especially if you happen to hit a nice domain name which people try to grab or piggyback on. Sticking with appspot.domain minimizes this issue. (if the need rises I might add a “real” domain later on, but in any case therealurl.appspot.com will remain active)

If you find a use for The Real URL this or have an idea for one, please comment here or email me at niryariv@gmail.com. Let the street find its own uses etc ;)

From Stewart Brand’s How Buildings Learn series: “Planners the world over take inspiration from Venice. But Venice was never planned”

This points to a fundamental mistake everyone makes, whether you’re planning a city or trying to relive a vacation you enjoyed – you try to match an example or experience, by building top-down what was built bottom-up, orchestrating what happened unexpectedly, reverse engineer Venice.

This is likely to fail: even if you have to resources and skill to copy Venice, your copy by definition is located in another place, with another weather, a different economy, inhabited by different people and so forth. So your copy of Venice, however precise, will always feel contrived and forced unto its environment and people instead of growing organically to meet them.

In software, these copies of Venice usually happens when you build an application without much user feedback in the process. You set your goal to an existing product which you admire, but don’t involve the users in the process. You often end up with a features users don’t require, an interaction which doesn’t make sense to them or even a whole product which doesn’t serve a real need.

I like 37signal’s Design Decisions posts, which explain the thinking behind seemingly small details in their apps. It makes sense: building the core functionality of most web apps is relatively straightforward, the real quality (and, ultimately, most of the effort) is in the details. So, here’s my take on Feedvolley’s design.

I’m not aware of any site that does the quite same thing as Feedvolley, so the first challenge is to get users to understand what it’s about and how to use it. My favorite way to learn to use something is to play around with it (not recommended with firearms, bikes and similar BTW), so the goal was to make Feedvolley’s interface invite users to do just that. That means making it as easy as possible to accomplish something, and then make it rewarding to keep playing with what was created.

To make starting out easy, the homepage is a minimal form with fields for feed or HTML page URL (one of the features that make RSS/Atom a good Web API is the fact it’s often auto-discoverable) and email (more about that in a moment). A default theme is pre-selected.

Another way to create a page is by clicking the “Create a page like this” link located on top of every user-created Feedvolley page. This lets users start with an existing page and modify the content and HTML to their needs. That’s one of my favorite web app buttons – it invites a viewer to become participator, and lets users start with something similar to what they want, and just modify it as they learn the system.

You might have noticed there is no registration step here. Personally, I hate having to register to a website in order to do anything. Feedvolley (like Notifyr) uses email as its user authorization system. Each user gets an edit link with a unique token string. This token is also kept in a cookie, so users aren’t forced to go to their email in order to start editing. “Create a page like this” also serves as backup for this system. In case a user lost the edit link, she can simply go to her page, create a duplicate and continue editing that.

This may not be perfect, but seems optimal for most users. If some users ask for a more rigid authorization system, we can always add it later on as an option.

Once a user created a page, the next step is to make further work on it possible, and worth the time. This is where the “Customize” link comes into play: users can set a page’s URL path and have complete control over the HTML, JavaScript and CSS in the theme. To make themes as easy as possible, Feedvolley’s markup closely follows that of Tumblr (Tumblr’s templates inspired the Feedvolley concept, in fact. They do great work over there). In “work with the existing environment” spirit, this lets users easily adapt existing Tumblr themes for Feedvolley and also use Tubmlr’s docs, which also saved us some documentation pains :)

As far as design goes, I’m with the “it’s done when there’s nothing more to remove” (as opposed to “nothing more to add”) school. So, I’m pretty happy with the result in Feedvolley. Some challenges remain: how to make it more obvious that a page can be customized once it’s created, for example. It’s not perfect yet and we’ve already incorporated some user feedback into it – if you have any comments or requests, please leave them here or email me directly: niryariv@gmail.com.

I don’t usually read Fast Company & co, but I found an article in the recent issue pretty inspiring. It’s about a landscaping company called Whole Systems Design which tries to build stuff in a way that takes into account the environment where it’s located. Environment not just in the “compost your pizza leftovers to offset CO2 of your vacation” meaning, but in the “stuff that already exists when you start building” sense. It’s a simple and intelligent approach to design – create something that works with what’s already there. From the article:

[company's founder, Ben Falk] calls mainstream environmentalism, with its “nihilistic,” minimize-human-impact approach, one of the “largest hurdles we face toward being a good community member of the earth again.” He cites one of his influences, Berkeley architect Christopher Alexander, who asked, “Can a building be just as natural as a tree?” Adds Falk: “Or a beaver dam? … It goes beyond ‘Let’s pretend we’re not there.’ We’re here. Let’s make a good impact.”

This links for me with the release of Phusion Passenger, aka “mod_rails for Apache”. I don’t know how well it works (Dreamhost now offers it, which seems a good sign), but if it works well this may be the most important thing that happened for Ruby on Rails since it was launched.

Rails’ main weakness, in my opinion, was that it did not take into account what was already before it, namely the Apache web server (and also stuff like existing code/DB schemes, but that’s less critical perhaps). It was very slow under Apache, and instead of fixing this the Rails community set out to replace Apache with something else – first lighttpd, then Mongrel, now nginx. But it turns out Apache leads its category, by a large margin, for a reason and that creating a good web server, though it might seem pretty straightforward to begin with, is actually pretty hard.

With mod_rails, Rails deployment may now be as easy as PHP deployment, which will give RoR a chance to compete in the shared server space, which is often looked down upon and yet is far bigger than the dedicated server domain, as seen here (this is based on PHP IP vs Domain stats – very rough but generally the right idea):

It is said that The Velvet Underground were hugely influential even though few actually bought their records, because most of those who did listen to them ended up starting bands. For some time, I was thinking Ruby on Rails is going to end up similarly – myself and many other developers got our initial exposure to MVC and ORM and Convention Over Configuration from Rails, but we often ended up building applications in Rails-influenced frameworks like CakePHP or Code Igniter that implemented these ideas in languages that could be easily deployed with reasonable performance (the key word here is easily. You can get performance with Rails, but it requires leaving your core problem for some time, to specialize on deploying RoR). Now, it may not be this way – provided people are still working on improving the docs :)

Ubuntu’s Brainstorm app is very cool. Basically it lets users post suggestions, bugs and requests for Ubuntu, and rate/comment other users’ posts. It enables users to get involved in the process and lets developers know what users are missing most and get ideas for future additions. It’s a new interface for discussion, something new on top of forums, bug tracking apps etc.

(Dell’s Ideastorm, predates Brainstorm, but the concept will probably work better with Open Source projects where users are more accustomed to this kind of feedback process. Ideastorm’s own top items are almost all Open-Source related. Having less of a life also makes some difference I suppose ;))

You can get the source at Ubuntu’s repository but unfortunately there isn’t a downloadable standalone package for this yet. It would be great to have an simple install of this, I think almost any product and company could benefit from something like this.

This is why open source software is growing so fast: it’s a positive feedback loop, its products feed its own growth. Good to see Ubuntu growing so nicely, with Vista & Leopard we definitely need an alternative.

This gives you (almost) complete access to Metafilter’s database: you can simply run any SELECT query on their DB that you like – export as CSV/Tab delimited file too. Very very cool. Wouldn’t it be nice to see something like this on, say, NYTimes or Amazon.com?

I spent some time recently polishing the old “Amazon price feed” project, and gave it its own domain: Prixfeed.com (pricefeed.com was, naturally, taken). Basically what it does is take an Amazon product (by its page URL or ASIN code) and returns an RSS feed of its price. This lets users keep a watch on the price, getting updated whenever it changes.

Always liked 37signals‘ Design Decisions posts, so here are a few I made with Prixfeed, mostly to keep it simple and minimum-effort as possible (BTW, where it says “RSS Reader” I refer to the software reading the RSS feed, not the person using that software):

• One URL:

  http://prixfeed.com/?ASIN=<Amazon ASIN or product URL> returns the RSS feed.
  (Or, with URI Templates: http://prixfeed.com/?ASIN={ASIN} )

  http://prixfeed.com/ without params will display a form where the user can enter an Amazon ASIN code or product URL, which then directs to the its RSS feed.

• No cron jobs: Prixfeed simply checks a product’s price whenever the above URL is called (naturally, some caching is needed here to guard from buggy/malicious clients overloading it). This means data refresh is timed and triggered by the RSS readers, removing the need to build a backend update system or keep track of what data needs to be – ie, which products are currently monitored. Which leads to:
• No database: so no need to keep track of who’s tracking which product, but with a non-RSS system (eg. email alerts) you’d still have to store prices in order to guarantee you’re only sending out a message when the price has changed. RSS, though, has a guid element which lets an RSS reader determine whether an item was read before. By making that GUID a function of the price, Prixfeed makes sure the RSS readers will display only new prices as new items.
• Machine readable: it’s easy to programatically create a Prixfeed URL request for an item price, but I wanted to make the response machine-parseable as well, while keeping it human-readable. One way would be to have a URL param that will return an XML response, but I like the minimalism of a single URL so I decided to use a Microformat-like approach. The response’s description element contains HTML which includes attributes like id="price" etc that allow automatic parsing of the content (more details here).
• Small output: each message contains the text “The Amazon price for <item> is now $x” and a purchase link. I like minimum friction products – there should be some proportion between the value you’re getting from something and the investment it requires you to put in, both when entering information and reading it. Since the value here is less than life-changing, I figured I better keep the time investment minimal as well.

That’s it then – once again I manage to write a blog post with more lines than the code its about! Go on and RSS thy Amazon purchases!

Kyle and Nate launched BricaBox this week. I think they’ve done an impressive job – you talk of the Web leveling the playing field but it’s still quite amazing to witness firsthand what two talented people can build in only 8 months: a sophisticated product that feels as solid and looks as good as anything else out there, and a brand name that in the right circles is already pretty well known.

BricaBox as a concept is very interesting. Essentially I see it as another step in the evolution of web publishing: initially you’d put some stuff online by authoring HTML pages, later on it was made simpler by CMS software (like PHPNuke etc) which let more people join in. Blogs, especially services like Blogger & co that let users start a blog without any technical knowledge, took this further ahead, increasing the number of participators by orders of magnitude and bringing online publishing to a point where basically anyone with a net connection can take part in it.

BricaBox expands on this by providing a simple blog-like interface for publishing information that isn’t text-centric – location based, like a list of cupcake sellers, item based like the NY Tech organizations directory, and so on – there are a lot of building blocks to play with, and I expect the most interesting formats haven’t been discovered yet, and will come from the users. I think a key point about blogs is that many people do not use them in the original diary-like “web log” pattern, but as a simple way to get text online. Blogs trade fine-grained control for ease of use, and clearly most people find it a good deal – enough so that many tech people who are capable of setting up their own CMS prefer to use hosted blogs instead. BricaBox taps into that, taking a similar usability-first approach.

Another interesting difference from blogs is the built in participation functionality – BricaBoxes are built so that readers can easily become contributors and add their own content to them. For example, if you happen to read my own humble (isn’t “humble” a really nice substitute for “lame”?) BricaBox of “places to watch soccer in NYC” you can not only add comments and ratings, but also new locations I didn’t know about. This feature, I think, taps into a very powerful aspect of the Web, the instant ability to move between read and write modes.

There’s a lot of potential to BricaBox. It’s going to be interesting to watch this grow – and I think it will, and quite fast.