A site IA visualized in aiSee

One of the most overlooked facets of many major sites is the actual organizational hierarchy. Too many designers and programmers just segment things into a few quick sections without doing any deep thinking on the issue, which can lead to major issues later.

In that vein, Audette Media has an excellent tutorial on the basics of Information Architecture for SEO. If your digital inventory of content and media isn’t well organized, it won’t just be hard to find for humans (leading to a higher bounce rate), it will also be shorted by search engines, leading to a loss of precious search engine position.

The gist of the article suggested that the team is struggling to work out a solution to making the text of proposed government action (particularly legislation up for signature) available and commentable. This got me thinking about how I might try to meet the challenge.

The way I see it, they have to present the text in a way that is:

1. Easily indexed, bookmarked, and searched
2. Easily commented on, with particular emphasis of reputation and identity of the commenter
3. Easily edited, with the changes legally required to be tracked

This immediately got me thinking of the best interfaces I’ve seen to work under these constraints, namely the Django Book and Wikipedia.

The Benefits of Granular Edits and Comments

The interesting thing about the Django Book is that it offers commenting on by section (usually a paragraph or group of paragraphs on one subject).  The benefits of this approach as opposed to commenting on an entire 1000+ page document, or even a single page, are obvious, and the structure of bills lends itself naturally to this approach. In fact, the bills, by nature, are already broken up into logical sections, making manual or programming section-splitting unnecessary.

What might this look like in practice? I’ve marked up a few fake pages¹ to serve as examples:

Tracking Changes

According to the Post article, the law mandates that changes to documents be archived, and it makes sense to make them available. Thankfully, web 2.0 culture has a few handy solutions to this, the most obvious being a closed wiki-style revision system for each section.

Upon clicking a link provided in each section, the user could see all past revisions of the section, and the date of the change(s). Technologically this is fairly trivial to do, in fact there are even entire database systems evolving around this kind of versioned content concept.

Handling Identity and Authority

One of the biggest challenges in providing open content is avoiding the youtube effect. Informed commentary is hugely beneficial, but low-value content like jokes, youtube style “OMG lOl teh lAW”, etc, are inevitable. This is a particular problem for an official government site, since the content must be kept to a minimum standard, but the slightest allegation of censorship would be discrediting.

To battle this, we can take a page from the playbook of Amazon.com and their “real name” system. Any normal user could post, but users willing to go the extra mile² would gain the ability to post without review, and have special icons or backgrounds showing their status.

An Example of A Review with Amazon's RealName Verification

In this case, users would presumably want to attach additional weight to certain classes of commenters. At the bare minimum White House employees, Congressional/Senatorial staff members, etc. could post with considerable additional authority.  Allowing users to filter un-verified commenters would be beneficial and simple to implement as well.

On a wider level, extending the system to those who can prove having a Doctorate, Law Degree, or Elected Position would be informative. This particular idea seems to suffer from serious scaling problems, but Amazon seems to handle it, and there are only 435 House members and 100 Senators, so an initial 550 or so accounts is doable in a short amount of time.

Searching, Printing, and Tracking

The final step to full a ‘web 2.0′-ing of the process involves improving search access, and adding functions allowing the computer savvy to remix the content.

Searching should work by bill or by date, and allow optional searching of comments and previous revisions.

Each subsection could offer an RSS feed, as well as overall feeds for each bill and for comments. This would allow programmers and journalists to easily watch for edits to a bill, as well as comments made by reputable sources.

Imagine being able to watch a bill and see a google reader alert noting that it was changed this morning in response to a comment-criticism by prominent judge Richard Posner, or that a major piece of pork was added to an appropriations bill. The implications for government transparency are staggering.

It’s probable that 90% of users won’t use a single one of these functions, but they would make the job of the vocal minority who do (journalists, bloggers, etc) considerably easier, which means wider dissemination and offers benefits for everyone.

Those who can; do. Those who can’t; blog.

I’d like to point out that I have nothing to do with the administration or their web team. Additionally, I’ve never seen all these features together in a production system, nor do I have to build it or pay for it.

This makes daydreaming about features notably easier =).

That said, everything I’ve gone over is not only technically feasible, it’s already in production in extremely large-scale sites, and most of it is already implemented in various free software projects, so it can be done.

Technology has an opportunity to bring the people closer to the U.S. government than ever before, but implementation is really make or break for government involvement, not to mention the public’s perception of the government (see: Internal Revenue Service, The).

Have any other ideas for how the interface to the bill system might work? Leave a comment below and let me know.

1. Click an Image to see the full size version. I’ve broken a few UI best practices here with respect to coloring links for example’s sake. Caveat Emptor
2. I believe Amazon bases theirs on credit card information, but something state-ID based would be plausible as well

The navigation is a bit unwieldy, but it is a neat take on trying to connect article facts and user comments on disparate (but related) stories and site content.

More information on some nodes is provided by popups, although clear citations and links seem to be lacking, perhaps they’re just getting intercepted by my pop-up blocker; hard to say.

Regardless it’s worth checking out, and even if this example falls a bit flat, it’s still nice to see a paper trying to ‘connect the dots’ in novel ways.

Sample Screenshot of the Gaza Coverage Visualization

It’s a flow chart of the Ice Cube song Today Was A Good Day (NSFW Language). Classic rap makes for some amusing grammar and linguistic feats, and seeing it mapped out here is a good laugh, not to mention an amusing stretch of the flowchart metaphor.

So check out the original at flickr, and here’s to more cool stuff in 2009!

Axis maps blog is sharing an interesting technique for adding another dimension of data to maps by using transparency.  Somewhat like a cartogram, they’ve taken each county in the US and modified it based on population density, but instead of stretching it to a new size, they’ve increased the alpha transparency to make more populated areas brighter and less populated areas faded out.

It looks like a challenge to get the color mapping right, especially in light of data that has a few outliers compared to a large area of uniform low density like population.  Perhaps a logarithmic scale would be more useful for data like this.

Regardless it’s a novel highlighting method, and definitely one to keep in the toolbox in the future.  I could also see this being useful for highlighting in non-map visualizations such as treemaps.

There are a few more examples over at the original post, so check it out!

Perhaps every category of data is composed of sub-categories, or a change in one data point has a major effect on surrounding data, but regardless, the standard library of charts and graphs doesn’t offer much in the way of making hierarchical data clear, so here are a few alternatives:

Nested Categories:

Treemaps:

Recently “Treemap” style charts have come out of their origins in academia into common use.  A personal favorite of mine, they allow aggregate categories to show through without losing the smaller constituent data.

Each category is sized according to what percent of the total it takes up, and child categories can be placed inside parents in a similar manner. Interactive versions often allow for ‘drilling down’ deeper into the data by clicking on a category to see its members full screen.

Treemaps seem to work best when the total number of categories at each level is fairly small (otherwise the hundreds or thousands of categories become tiny, undifferentiated squares), and each item fits neatly into a single sub-category.

As usual, Many Eyes leads the pack with excellent treemaps, and even offers a rather unique treemap over time.  Other solutions include a free (non-commercial) desktop program, java library, jQuery plugin, flex component, and Excel add-in from Microsoft.

Multi-Level Pie:

If the categories are a little more concise then those in a treemap, or deeper data is required at a glance, multi-level pie charts may be the solution.  As the concentric rings go ‘out’, each item is sized with respect to its contribution to the inner parent category, allowing for deep hierarchies to be understood at a glance.

Neoformix has a great illustrated example of building a multi-level pie chart, and Fusion Charts offers the best flash version that i’ve used in their powercharts package. HDGraph applies the concept to hard-drive space use, historically a big driver of multi-level visualization.

Bubble Diagrams:

Bubble diagrams, which are generally used to show multi-dimensional data (x, y, size, color) instead of hierarchical data, can be adapted easily, either by coloring all members of a similar group the same (for example Gapminder assigning members of each continent the same color), or by nesting the bubbles within another, to create an effect somewhat similar to a treemap.

Many Eyes offers a fairly simple version, mainly for highlighting relative sizes, as does Microsoft Excel, while Gapminder offers more complex abilities. I’ve yet to see anyone take on the nested-bubble approach, but feel free to contact me if you know of a solution.

Organizational Hierarchies:

Vertical Hierarchy:

Most commonly seen in the ‘organization chart’, vertical hierarchies are common for showing chains of membership or authority, but rarely have an added dimension to show actual numeric values.

The best examples use pictures effectively, and tend to represent data with each level’s members being of similar importance.  The JIT toolkit offers a javascript based approach to this, as do any others.

Horizontal Hierarchy:

In contrast to vertical hierarchy, these charts normally serve to hide large amounts of data that may be useless to a user, and allow fast drilling very deep into a hierarchy. Horizontal charts work best when the data is very deep, but each category only has a few subcategories, so the user can drill down very quickly.

My favorite horizontal hierarchy examples comes from the spacetree at the JIT.

Relational Data:

Sometimes data in hierarchies takes a more abstract form, where each element influences one or more elements, or just overlaps multiple categories.  These types of data sets tend to provide an additional challenge to visualize, since laying them out optimally involves a bit of heavy-lifting mathematically.

Node-Link Diagrams:

Coming from a branch of mathematics, these diagrams offer different takes on data that is highly interrelated: one element may link to hundreds (or thousands) of additional data points, or none at all.

Node-Link charts (also called Network Diagrams) are best employed when data is related, but not necessarily in a clear hierarchy.  Different dimensions of data can be shown by the size of each node, color, or even position.

A network diagram projected in 3D to highlight hierarchies

Some toolkits even allow the links to be different lengths, thicknesses, or colors to show an additional dimension of how two points are related.

A huge number of solutions exist for different variations on network diagrams, a few selections include efforts from constellation, fusion charts’ power charts, many eyes, JIT, and jsviz.

JIT also offers an innovative hyperbolic network graph, allowing connections that are further down the chain to be hidden from the user.

Traditional Charts:

Stacked Bar:

Probably a simplest and most familiar chart type on the list, at a glance this chart only allows two ‘levels’ of depth, (parent category (Bar), and parent percentage (Inner bar)), but interactive versions allow the user to click on a bar or bar component and ‘zoom’ to see categories that make up that bar.

This chart is excellent for comparing simple, broad categories with few constituents, and benefits from general familiarity, but fails to impart more than two levels of depth at a glance, and it’s often non-obvious that user interaction is possible.

Most common charting packages offer stacked bar charts, including Excel, and flash packages such as Open Flash Chart, FusionCharts, and AnyChart.

Stacked Shapes/Area:

Similar to stacked bar charts, many charts offer stacked Area or Shape (Pyramid, Cylinder, etc) charts.

Although these can be useful (particularly the pyramid chart, when one or two options in each ‘bar’ take up an overwhelming percent of the total, but the lower percents must still be visible), they suffer from the pitfalls of stacked bar, plus the added difficulty of resolving the relative sizes of 3D objects.  Oftentimes relative proportions that are intuitive in rectangular dimensions (“This box is twice the area of that box”) are lost in other shapes or in 3D (“How much more volume is in the bottom half of the pyramid than the top half?”).

Again, these are supported by most common spreadsheet and graph libraries, but beware of obscuring data just to add eye-candy.

Common Challenges & Future Developments

One of the largest problems with presenting hierarchical data is size, once the members of a tree grow beyond a few dozen, putting them all on screen at once can be overwhelming, but knowing which elements to hide can be a challenge as well.  This makes effective use of color, as well as choosing the right visualization for the job extremely important.

Hopefully in the future we’ll see more development of toolkits and frameworks for quick-deployment of interactive charts, and better support for deep hierarchical data.  Thanks to strides taken by a few of the javascript toolkits, flex/flare, and IBM’s Many Eyes, most of the solutions listed above are becoming accessible to the average developer, although custom deployments still abound.

With the uptick in machine readable statistics being made available at corporate, as well as national and international levels, we’re sure to see a rise in large, deep data sets in the future, and the corresponding need to understand that data intuitively.

A Standard 2008 Election Map (Upper), and a Cartogram skewed by Population (Lower)

Cartograms, or visualizations of an area skewed by some variable, are a powerful tool to control for disparities over a large area, especially with respect to politics.

A relatively large but sparsely populated area will dominate a standard projection, whereas a cartogram allows populated areas to be warped to show their true influence (See sidebar).

Applications of Cartograms:

Socio-economic data is the most obvious use case for cartograms, particularly data from the United Nations and National Elections.

The cartogram strategy can be applied to just about any area visualization, and works particularly well where there are major disparities between area size and overall effect.

Algorithms and Tools:

A general outline of the cartogram creation process, as well as some excellent alternatives to the standard ’skewed border’ approach can be found at “Scaling Counties in a Checkerboard State” over at style.org.

Actual implementation of cartograms seems to often follow the algorithm first published here. Desktop versions in Java (with code) are available here and here.  There’s a rundown of more methods at indiemaps.

A Rescaled Election Cartogram. Can You Identify the State?

Another post at style.org looks at alternate skewing methods, namely squarifying the areas in question, and then expanding them by a fixed ratio with respect to the data.

Non-Standard Cartograms:

Some of the most beautiful work in cartograms comes from the SASI group and Dr. Daniel Dorling.

Indiemaps blog has a number of posts on cartograms, including the promise of python source code, and some great visuals.

I think cartograms are an untapped resource for commerical data, especially for geographically disbursed figures like sales or conversion rates. Hopefully in the future we’ll see more tools centered around this resource, as well as a decent online generator.

There’s a post by Joseph Hellerstein worth a read over on O’Reilly Radar: The Commoditization of Massive Data Analysis.  It’s more enterprise focused then small-normal business focused, but that’s just a consequence of the target audience.

His primary point is becoming especially pertinent to web companies and smaller developers: The convergence of dropping hardware prices and machine-readable APIs is making the storage and processing of vast amounts of information practical.

We are at the beginning of what I call The Industrial Revolution of Data. We’re not quite there yet, since most of the digital information available today is still individually “handmade”: prose on web pages, data entered into forms, videos and music edited and uploaded to servers. But we are starting to see the rise of automatic data generation “factories” such as software logs, UPC scanners, RFID, GPS transceivers, video and audio feeds.

It’s already reasonable for a site on a commodity web host to store every user and search interaction, or a database of tens of millions of data points, and in the future it will only get easier. The question is, what tools will we use to make sense of all of this?

His analysis reduces the field to SQL (via Oracle) and MapReduce (via Hadoop), but once we look beyond the enterprise, tools like Erlang (or functional programming in general) and the emerging CouchDB show promise, not to mention some of the cloud computing entries from Amazon and others.

On the visualization side of things, tools like Processing and the Prefuse Toolkit are seeing quick uptake, as well as more focused commercial tools like FusionCharts.

Whatever the toolchain turns out to be, those of us with an interest in understanding information have the opportunity to be on the forefront of the change, and if we don’t gain expertise in the available options early, we risk being left behind.

Well Formed Data is offering this post full of offbeat election and presidential data visualizations.

It’s definitely worth a look, I’m particularly enamored with the New York Times’ Presidential Physique Graph, although the data density is a bit low for being so large.