The Non-human interface

Introduction
In this essay I hope to cover the state of current human-machine interfaces, what's wrong with them, what can be done to fix them without confusing people or introducing too many new learning curves, and where I think they should go in the future. This essay is not meant to be definitive, or the absolute authority on the topic, but rather my views on the subject, as a computer programmer and someone who has to not only use, but build the interfaces that let people work with machines on a daily basis.

In the beginning was the console
Try to imagine, if you can, and time and land far away and long ago. A time when there were no mice. Color was a myth and legend, and "clicking" was an annoying sound, not a verb. This was a time on the console, when all human interaction with computers was done with a keyboard. There were no windows, no pointers, no drag-and-drop. Just you, and a little blinking cursor. Type a command, and get a response. Plain and simple. If there wasn't a command for what you wanted to do, then you would program one yourself, or chain together lots of other ones to produce the results that you wanted.

These days are not entirely gone. I still use a console for the "down-deep" control and speed that you get from it. There is just no faster way to bring up and down network interfaces, log into a remote machine, move files around, rename things, etc. In fact, a lot of things that we do with computers today in the point-and-click world, actually take less time in the console than they do in a windowed GUI. If you know how. And that's the trick.

The console faded away as a primary user interface not because there was some driving desire to have pictures on a computer screen. By and large, at the time computers were starting to be used by non-corporations, they didn't even have the power to show pictures, or do anything useful with them. And why would you want to? Computers are for business computations, and writing letters, and nothing else. Right? Well, times changed. Apple came out with a home computer that was the first to not need a great work of Japanese literature, and a 100-man support staff to operate, and all the rules changed. The ever vision-full, and megalomaniacal Steve Jobs came up with a way of doing all the things that computers do, but in a way that didn't require the user to take college courses on how to use them. If you want to move a file, why not just grab it, and put it where you want it? What's with all the 'mv -f ~/file.doc /home/otherguy/' crap? At least, that was the idea. Consoles disappeared because it kept a very large demographic away from buying very expensive products: The everyday person.

Now that computers can be used just by "clicking", things are going to change! People will do what they want, when they want, and it will "just work"! Well, it's a nice thought. The reality is that to the people who were used to consoles, it was clunky, and the people who weren't, it was foreign. And worst of all, it was all the old console users who were making the very thing that was designed to destroy their usefulness. It only seems logical that they would build it in such a way that programmers would still be needed. Who the hell ever thought of a spreadsheet, anyway? And does anyone who hasn't taken courses on them use them for anything other than the most basic math? What about a calculator? The modern computer interface was born out of a business need to sell more stuff to more people, and designed to keep those people out of power over what they had purchased, so that the people who made it would (a) still be needed, and (b) you could sell even more (support contracts, more and more "powerful" -- read 'complicated' -- software, etc). And how it was done, was really very smart.

The curse of the metaphor
Every graphical computing interface we've seen so far is based on some metaphor. Some way of representing data so that it resembles something that we already use. The most common of these is the "desktop" metaphor. Files are sorted into folders that we keep around our desk, and so it is with computers. We put documents out in front of us to work on them, and move them aside when we don't need the; so it is with computers. Throw in copy-and-paste, some searching, and you've got the desktop metaphor. The mouse becomes your hand in this virtual office space, and the keyboard you pen. In theory, it seems elegant. In theory it works. In reality, however, anyone who uses a computer on a regular basis can tell you that it somehow falls short.

Metaphors have been a part of human culture for long before computers, however. Even the idea of writing is just a way of making permanent something that we would normally just say. And even speaking is just a way of communicating something that we think. There is nothing magical about it, though, unlike computers where most people think it's all voodoo. It's just what you do. And computers are trying to be that as well. But there are a few places that they just don't do too well. In the real world, if you write a letter, and walk out of the room without pressing some magical button, the letter doesn't burst into flames never to be seen again. With computers, though, if you don't press "save" then everything that you were working on is gone. The metaphor breaks. The solution? Ask them if they are sure they want to quit! Of course! Because we all know that some acid-induced animated paper clip always asks me if I want to save the letters I write. Perfect! Wait a second...

The need for clicking "save" came when drives were measured in Megabytes, at most. You couldn't hold on to every little thing, because you would just run out of space. Now, however, even laptops have 80+ Gigabytes of storage. Do you have any idea how much stuff that would take to fill (I'm talking text and images here, not someone's 3-month long playlist)? The need for that part of the interface is gone, and yet it lingers around. This is a minor example of a way in which computers become confusing. You have a metaphor ("It works just like your desk, with some extra stuff to make it easier!"), and then punch holes in the metaphor for reasons that aren't valid anymore ("Oh, I forgot to mention; if you don't tell me specifically that you want to keep something you've been writing for the last hour, I'll assume its not needed and get rid of it for you"). But there are plenty of other places that these metaphors fall apart, and make things more confusing and harder to use than normal. Try this: use your computer, doing things that you normally do with it, and think about what metaphor it is trying to convey. Then, put your mind into the space that it would be in for its real-world counter-part, and see where it falls short. Drive letters. Web "pages". Having to wait for one thing to open, just so you can see what some other thing is. Turning on and off your "desk"...

But these metaphors are not without their place. They help keep the learning curve less steep (or try to). Certainly dragging a file to a new location is simpler than learning the commands for the console. I just think that they go too far in some cases, and not far enough in others. All the extra stuff cluttered with your computer is hardly a good representation of a desktop, but it's still called that. Books on the screen turning into web "pages". What the hell relation does a web site have to do with page in a book anymore? I know that when I read a good book, I just love the animations at the top offering to connect me with singles in my area. My point is, either make it work like its real-world counter, or completely change it. People will figure it out, and this is how:

The mind's power to ignore
The brain is a mighty, and wonderful organ. Unknown to science or philosophy are all its wonders, but known is its... Okay I've got nothing... My point is, your noggin has some pretty cool things that it can do that really help with all this mixed metaphor stuff. There are two things in particular that I am talking about: (1) Forget and (2) Integrate. Both are related.

One of the most powerful things that they brain does is classify things, including what to expect of things that are already classified. Once you have those things tucked safely away into their cubbies, you will either integrate it wholly into your life, or forget it entirely. Language: You hear it long enough, until it all just clicks, and then *bam* it permeates pretty much your entire life. The same old billboards on the way to work: You see them enough, until *bam*... You just kinda' stop seeing them. A good example of these two things working together is driving. How many things do you think go on when you turn a corner while you are driving? You slow down, and turn on your blinker, check your mirrors and your blind spots (hopefully), turn the wheel (just enough, mind you. Not too much or not enough), accelerate through the turn, etc etc. There's a lot going on. And if you are like me and drive a stick-shift... Oh jeeze! Now, how many of those things do you think about when you do them? Probably not much. It all just gets lumped into "turn".

This pattern classification and integration action can be put to use when you use a computer as well. You already do. Typing. Clicking on things with accuracy. Remembering to click "save"... Because of this, I think that interfaces can provide much more control and integration than what they do, and you will just filter out what you don't need long before it ever gets to any part of your brain that does any kind of thinking. The trick is to just make it consistent, so that your brain can classify it faster. How much work does it take any of us to find what we need on Google? You can already weed out what is valid and what is not, even without reading a lot of it, and certainly without clicking on any of the links to investigate. And even if you do, you already have an idea of what you're going to find. How many of our parents can do the same thing? That goes into the fact that the more you integrate something into your life, the more you completely ignore it. I bet that your parents could use a slide-rule without thinking. Could you?

If you make the interface consistent, and intelligent, then the brain will do the rest. Pretty drop-shadows, wavy effects, sounds, etc. All have their place. They keep the eyes focused where they need to be. The provide feedback about what is and is not allowed, etc. But they shouldn't be the interface in and of themselves. They should augment it to make it easier to use for longer stretches of time, but it shouldn't be the final goal of a GUI developer. The final goal should be an interface that works with the mind to make it simple to classify things, so that you will naturally figure out what it takes to use it quickly, and efficiently.

I just got a copy of the latest and greatest OS from Apple; Tiger. I still haven't played with everything, but I've poked around a good bit. They do a lot of things right, in my mind (and some things wrong. I'll get to that), that I think will help give you an idea of what I'm talking about. One of the greatest things in this new system is Spotlight, their search technology. All the files on your harddrive are indexed, and that index is kept up to date at a system level (so it is always fresh), and you can search against that index in some pretty cool ways. On top of that, people can make plugins that change how certain files are indexed (so that your mp3's can have their lyrics indexed as well as their names) so that you can get more intelligent information. Files can also be tagged with extra information that make the search better. What does all that mean to the user? Let me explain:

The idea of having a lettered drive, or even the / directory is not one that is inherent to computers. On your harddrive are just big strings of 1's and 0's, and nothing else. Nothing that even denotes files or folders. All that stuff is made up by your operating system, because way back when someone said "Hey, let's make a metaphor like your desk. You put files into folders! Genius", and then they shoe-horned all those 1's and 0's into something that made that work (kind of). Then the obvious happens; you put more folders inside of folders, or have 20 copies of things lying around, or hang on to things that you would never hang on to in the real world, etc. Pretty soon, it looks nothing like a desk-top filing cabinet, and more like a bunch of crap that you can't find anything in. Spotlight takes care of that. You tell if what you want, and it gets it. Want everything that meets some criteria in one place, regardless of what you do with it (how many times have you explained to your parents or grandparents the idea of saving a document someplace, and not that it just goes "on the drive"?)? Then attach a search to a folder (Smart Folder), and from then on, that folder will gather everything from all over the drive that meets your specifications (I want every word document that was made in the last week, and is tagged as a work document). Done. Simple. On top of that, they added an interface to those actually quite powerful controls that an average person can figure out, and pretty quickly you have the perfect way of organizing your files: Forget about it, and let the computer do it. As it should be.

Now, the downfall of Spotlight is that there is still the normal file-folder paradigm in the system. So it gets confusing sometimes. Is this a folder or a smart folder? I tagged that file as a document, how come it's not in my documents folder? You get the idea. I think that if they would have replaced the finder with spotlight all together, and said screw the old ways, then it would be perfect. Then the smart folders are just saved searches (search for your searches, right?), and you just get what you need, when you need it, in whatever way makes the most sense for you to get it (by date, name, size, author, geographic coverage, lyrics, etc etc etc). Let your brain do the heavy lifting. Imagine a computer that you tell it "Get me that thing that I downloaded the other day. You know, that one that showed the picture...", and it would. That's spotlight, if the Finder wasn't stuck on to it.

Another cool thing in Tiger is Automator. You have a library of blocks of simple code with some simple options that you string together to make complex programs. You don't have to be a programmer to use it, and any time you install a new program, you get more things that you can use as those blocks (of course, Spotlight is there to search for the block that you want, in case you can't find it in the list). Most of the things that people do with computers are tedious tasks that a person can't do by hand. That's what they're for! Unfortunately, what happened is that we got rid of all the annoying tedious stuff with computers, and then just made things that can only be done with computers, but are also tedious. Automator helps with that, by letting you make programs once that do all those things for you. Great! As it should be! That's what computers were promised to be in the '80s and we never got. They're on to something here.

All fandom of Apple aside, I think that those sorts of simple things are what an interface should be based on, not have those things based around an interface. The problem is in data representation vs manipulation. Which brings us to our next bold header:

Data view, vs Data function
As I mentioned before, what you see on the screen, and what you do with your computer has no relation to anything that the computer is actually doing. I know that you all have heard "Computers are just ones and zeros", or "All they do is add", but I don't think that the actual point of all that is really made. You are reading text right now, and your computer has no idea what the hell text even is. It doesn't know what a letter is. There is no magical combination of 1's and 0's that it looks for to produce some glyph on the screen. There are only streams of 1's and 0's that go in, get added together, and the results get spit back out again. Then other, smaller computers take those new streams of 1's and 0's, and do some other stuff with it. Even the software that does all this is just 1's and 0's that stream in and out. There is no smarts, no intelligence. No secret key that it looks for or that has meaning. No meaning, nothing. It really is a modern miracle that anything with computers even works at all!

However, none of that means anything to a human, and so we come up with the metaphors that I talked about earlier to help it make sense. There are two parts to this: How the data is presented to you, and how it is made available to change that data (and those two things are linked together to make sure that what you are seeing is a good metaphor for what is going on behind the scenes). A simple example is writing a letter. You see some text on the screen, and blinking cursor, some buttons for formatting etc. That is the presentation, or "view" part. You click where you want to type, and then type with the keyboard, which adds to the letter, which is the manipulation part. The two are linked together so that you see what you are typing as you type it; that is the connection between the two.

In most people's minds, these two things blur together, and there is no real concept of any of that being different. But think for a minute about what I said about none of it having any meaning. None of it is anything important to the computer. None of it is anything that resembles what you think it is. As a result, the interface that is presented is many-times removed from what is happening. The reason that I point all this out is to show that, if we can go from no meaning what-so-ever, all the way to the other side of the spectrum (amazingly complex tools and data types), why does it still suck? We can make it do whatever we want, that much is obvious. How come nobody has?!

Part of the problem, now, is that it is systemic. It is built in at this point. Most people think that when they type something on their computer, that the computer actually stores those letters and numbers someplace, but it doesn't. It just kind of has a stream of 1's and 0's, and collides it with another stream of 1's and 0's, and that seems to produce something useful. However, because people have been led to believe that the computer stores and retrieves things in the same way that we think (those metaphors, "made to provide a virtual version of real-life events"), they expect it to work in ways that are like their minds, but it just doesn't, and can't because of the way that the metaphors have been built up, and layered on top of one-another. This leads to endless frustration, annoyance, un-knowing of how to perform a task, training the mind to do things that it shouldn't have to do, etc etc etc. That's why I think that the metaphors should either be perfect representations of their real-world counter-parts, or should in no-way resemble anything related to the real world. In the first case all the limitations of how that works will be evident, and people will know how it functions. Of course, then what is the point of a computer, if you are going to have to do it the same old way? In the latter case, I think, is where things need to go, and to some extent, are.

The way of the future
Here are some thoughts on interfaces for common tasks computers are used for, and how they might be altered to work more with the brain, and not force the user into assumptions about how it works that are inherently flawed. I want to cover Browsing the web, writing a letter, opening and editing that letter later, sending that letter to someone, and playing games. Obviously there are more things that computers are used for, but I think that these things will cover a pretty good range, and present ideas that can be stretched out across other areas of computing. Plus, it's what I thought of off the top of my head.

Before I get going with this, I just want to re-stress the fact that these are just my ideas. I don't have degrees in any of this stuff, I just have to work with it daily, and make it daily. If you don't like the ideas that I present here, then just don't use anything like what I am proposing! ;-)

When you browse the web, generaly one of two things happen: Either you search for what you want, and then look through the results until you get something useful, or you already have a destination in mind, and you just type that address in (or have it book-marked), and off you go. I'm going to focus in this case on the searching for things, because the interface is not all that important if you already know what you are doing, and where you are going. I'm also going to use Google as the example search engine, just because it's google. When you search for something, as it stands now, you type in what you are looking for, and it goes and retrieves everything that it thinks is relevant, sorted using some really complex calculation based on number of links, blah blah blah. What you see is a list of links, and little broken previews of what that link will give you. Even with that limited information, you discern what is useful and what isn't, and only click on links that you are already know are pretty good candidates, and if you need to search again, you refine your search by putting things in quotes, or adding +'s and -'s, etc. Until you find what you are looking for.

What I propose for a system that will work more with the mind, and allow for faster, more accurate searching is a bit different. I think that you should be able to type in what you are looking for (the same way that you do now), but instead of just getting a list that you have to page through, you get a stream of links, and their descriptions (the descriptions, for this to work, would have to be a bit longer, but not much). As you move your mouse down the page, it scrolls down the list, and as you move up the page, it moves up the list. When you find something that you think is promising, you move to the right over the top of it. That brings you to a screen that has even more information about that page (in terms of a larger preview, maybe even with pictures, etc), as well as links that are similar to that page (you know when you click on "more like this"? The functionality is already there). If that gives you what you want, click, and you go there. Otherwise you can use the new similar links as the new scrolling stream of links, or go back (to the left) to the original stream, and continue your search. If you want to re-state your search, just start typing and the search box comes up, and you can modify as needed. This way the place that you are trying to find is more readily available, and your brain will figure out in what direction you have to move to find something that is useful. That's what your brain is good at (abstracting, forgetting, and integrating), and the computer should play to that.

When you write a letter to someone (old school), you pull out a piece of paper, and start writing. In the computer world, it gets more complicated. The only thing that makes it go quicker is that it can be sent electronically, and most people can type faster than they can write. When you write something by hand, it doesn't matter what it is, the process is the same. Weather you are writing a letter, a term-paper, a poem, it doesn't matter. All that is worked out by the way that you use it (you send a letter, you turn in a term-paper, etc), but on computers different tasks (really the same task) are handled with different programs. Why? It's the same paper isn't it? It's the same computer isn't it?

I don't have too much grief over the idea of opening a program first, before you start typing (although I think that can be gotten rid of, too, with some simple pattern matching so that the computer can figure out what it is that you want, and open the appropriate programs... All that's for another essay, though). What I have a problem with, is having to have one program open for writing an email, one for writing a paper, another for making a post to my blog, etc etc. I'm just typing in all of them! Why not have one interface, with some smarts built in (with overrides for when it's just not that smart), that figures out what you want to do with it. If you start typing "Dear Susie", why can't it know that you are writing a letter, and load up the controls for an email, and fill in Susie's email address for you? That doesn't seem too hard (and as a programmer, I can tell you it's not). Or if you are just typing normally, load up the controls for margins, tables, formatting, etc. You are obviously writing something long. Or, if you are doing a bunch of regular indents, using certain keywords, and broken, not-quite-right sentences, then there is a good change you are writing code. Load up the correct controls for that.

In this sort of paradigm, programs are not entire monolithic entities, but more like operating system plugins that the OS just loads as they are needed. Break them up into small plugins that fit into larger packages, and it is totally reasonable to load only what the user needs when the user needs it (no waiting for Word to load EVERYTHING that word can do, just to make a note). Seamless, easy for the user, and they don't have to think about it. Just use the damn thing, and it will fill in the tedious crap. Isn't that what computers are supposed to do?

Going back to the subject of writing this letter, I mentioned before my disdain for having to click "save" to hang on to something. I think that you should just be able to type, and close the program, and it hangs on to it (now, this is a good example of different needs for different places. On the web that wouldn't work because of how the internet works at its core. Maybe someday, though). It gives it a name that it thinks is appropriate (and it doesn't matter, because the user will never see the name anyway.. Keep reading), tags it with some information about who wrote it (you log into your computer, don't you?), when it was written, any anything else that either the user has set up, or that the OS plugin thinks it needs (what kind of file, indexed information that is pertinent to that kind of data, etc). Then, when you want to find it, and keep working on it later, you use the same sort of interface that I described for Spotlight. Just tell the computer what you want, and it finds candidates that make sense, complete with previews, and quick opening of that data. If the computer caches, during the search, the plugins needed to open that file, and when you click on that file, it just pushes that bit of programming to the user (again, not that hard), it would be seamless. You would just tell the computer "Get me that letter I wrote yesterday", and it would come up with every letter you wrote the day before, with previews so you know what you are looking at. Find the right one, click on it, and *bam* it opens back up, right where you left off, ready to go. Click send, and off it goes. The computer gets out of your way, and lets you do what you have to do, when you have to do it. No more work before you can get to work.

The last bit that I want to go over is computer games. A lot of people think that computer games are silly, or just not needed, or whatever. I would agree to a certain extent. However, computer games have been the area of programming research that pushes computing power forward for the common person. Those nice shadows on the edges of your windows? Thank game developers for figuring out how to do that in real time. Compressed music (mp3's, etc) that plays quickly, while letting you do other things. Games. There are a ton of examples, not all of which come to mind right away. And even if you don't buy that, you can't deny how many games are sold, and how many people play them. It's a big deal with computers, and it's not going away. As such, why shouldn't games portray whatever it is that they are trying to portray, while at the same time being easy to use, and more natural? Why not have them have the same goals as every other computer program? For the sake of this essay, I'm just going to be talking about First Person Shooters.

FPS games, such as Doom or Half-Life, typically allow the player to do quite a lot of things. Run around, look in all directions. Look in different directions than the one you are moving in. Manipulate the environment around the player, etc etc. A good example is Half-Life 2, with one of the most advanced physical simulation systems that has ever been in a game. You can just about do whatever you want with the world... But not with your character. You can stack boxes, knock down walls, use bodies as weapons, but you -- the player -- still moves like a giant chunk of wood that can get smaller (a bit), and shoot back. You go forward, backwards, side to side, jump, duck, and shoot, and that's it! You move as one giant thing, not as a person. But what do you expect? Four keys and a mouse is all you get to control things. Not much to work with, right? Well, I think you can get even simpler than that, but still allow for more freedom within the game world.

I say, get rid of the four keys, and just have 2 (yes, two) mice. One for the right-side of the body, and one for the left. Most gamers have mice with at least 5 degrees of freedom within the mouse (2 buttons, and a clickable scrollwheel -- the wheel goes two directions). Have two of those, and you get 10 degrees of freedom, not counting actually moving the mouse around your desk. To the user, there would just be some pre-programmed combinations that did specific things (walk, jump, shoot, etc), and a training level. In the training level, it would show the user the basics, and then ask them to do the basics, as well as some more complicated things. This would server a two-fold purpose: 1) The user would have a chance to make those movements natural, and let the brain start ignoring them, and 2) it would let the computer "learn" what the user's take on those movements are. Everyone is a little bit different, so just have the computer figure out the fuzzy edges. It's been done before, and in real-time, so why not again? Do it in a training level where you aren't under any pressure to win, and don't care how it goes, and then store what is learned by the computer to be loaded later. Pre-computed weights on the base-movements. Then, when you hit the game world for "real", moving realisticly, controlling every part of your character's movement and interactions just as naturally, and with as much freedom, as you do in real life. Imagine an action game with real action, not pre-recorded crap that gets old after an hour.

I'm not sure if that last one is as clear on paper as it is in my mind, and to explain it fully would take at least another full essay this size, so I don't think that I will be doing that right now. Some other time, perhaps. I hope, though, that I at least got across the idea that the metaphors that we are using for computing are broken, they don't work with the brain, and making things that would work with the brain would be really easy, and make computers fullfill the promise that was mad so long ago. Maybe at some time I'll throw in some things that I cut out about just exactly what it would take, or maybe even some demo programs that will give you a better idea of what I mean. But I think that I've rambled enough. Ask questions, and I'll answer! :-)

Thanks for letting me bore you,
--Thomas