If KDE is to be recognized has the best free desktop environment, the KDE com munity must learn to integrate usability testing in its development process. Basic usability testing is not difficult, it isn’t expensive and the results are invaluable. This test conducted in early July 2002 with four participants outlines of some of KDE 3.0’s shortcomings including inconsistencies in KFileDialog and the difficulties of working with Konqueror’s embedded viewers.

Goals

The goal for this test was twofold. One: to acquire experience conducting usability tests on systems not designed by me, and two: to begin sorting out KDE’s bigger usability problems using actual test data.

Until recently, my experience was limited to testing my own material on the web. Testing KDE, a large-scale international project, was a new step for me. Since this is not the subject of this article, I can jump right to the conclusion here and tell you that testing websites and testing desktop environments is not that different and that, in both cases, a little testing goes a long way to identify problematic areas. And since neither point is particularly new I will not dwell on this any longer. On to the KDE stuff…

KDE has been around for a while and has received a lot of attention from the free software crowd. In the past few months we have begun to see commercial initiatives centering their efforts on bringing GNU/Linux out to the general public, some fairly reasonable, others downright moronic. In these projects, and I only want to think about the reasonable ones here, KDE is often chosen as the default desktop environment. I understand this to be a positive indication of KDE’s current level of usability, stability and overall maturity. But with all this attention, KDE is destined to end up in the hands of users who have little interest in filing bug reports and who will most likely not have the confidence or patience to explore, change, customize or hack their computer environment. Making free desktop software truly usable by all is a new challenge for the free software community of which KDE is just one (fine) example. The first thing that needs to happen is to gather sufficient information to make informed decisions based on actual observations.

Test Description

This first test was meant to be very broad. All I wanted to measure was how new KDE users would go about accomplishing basic tasks such as moving files, checking mail and browsing the web. Some of the assumptions I made about the user’s environment included that the user would not have to setup his/her workstation him/herself and that the user would not have to go through KPersonalizer’s step by step setup. This is the kind of environment you would find in the corporate sector where newly hired staff just logs in to a workstation already set up by the company’s system administrator.

I used KDE 3.0.2 compiled from source with the following packages added: kdemultimedia, kdenetwork, kdepim, and kdeutils. I left KDE the way it was after compilation: default KDE look-and-feel, default mouse focus, default desktop icons, default wallpaper, default panel icons, default everything. The hardware was unremarkable: Pentium II, 256Mb of RAM, 17″ LCD monitor; KDE was responsive but not unrealistically fast.

I created five tasks for users to complete:

1. Open a web browser. Go to <URL>. Bookmark this page so that you may visit it later. Download <PDF file> on the page to your home directory. Close the web browser.
2. Create a new directory on the desktop. Call it “myStuff.” home directory, locate a directory named “documents.” Move all the files in “documents” into the newly created “myStuff” directory. Delete the old “documents” directory.
3. Change your current desktop background to use an image as wallpaper. The image file to use for wallpaper is called “myWallpaper.xpm” and is located in your home directory, under “images.” You’ll want the image tiled over your desktop.
4. “KMail” is an e-mail client, a program to read and send e-mail messages. Locate KMail, open it, and check your mail. (KMail has already been configured with a test account). You will have received a message from <somebody> containing one attachment. Read the message and save the attachment in your home directory. Reply to the message; simply acknowledge that you received the e-mail. Close KMail.
5. Locate the file you just saved in step four. This file is compressed (“zipped”); uncompress it. One of the newly uncompressed files is called “Readme.txt.” Open it to edit its content. Below “This file was created by <somebody> on <date>” add “This file was modified by <your name> on <today’s date>.”

Test Results

Overall, test participants went through tasks 1, 2, and 4 easily. 3 and 5 caused some problems. While some participants noted that KDE looked different from Windows, none seemed bothered by the differences and the look-and-feel of KDE. Users identified all the elements of the interface without any trouble including KDE’s Konqueror and KMail icons. Most users seemed to understand the K menu’s presence and function intuitively and they used it much more than I had anticipated. This is, I think, a reassuring observations for many who worry that it is too confusing to a new user. The layout of the menu itself could probably use a little work but that is a subject for another set of tests. KMail performed particularly well as all users went through basic e-mail tasks without any hesitation.

Small Issues

Completed Download Confirmation

Users had no trouble downloading a file from the net but the absence of a confirmation that the download was complete was surprising to some. This is especially problematic with small files or on fast networks when no progress bar is ever displayed. It is my understanding though that KDE 3.1 will provide a “Keep this window open after the operation is complete” checkbox which is a step in the right direction to solve this problem.

System Tray Items Tooltips

All elements on the panel and its default applets have tooltips with the exception of the System Tray. Users exploring the system took advantage of tooltips and I was able to observe that they served their purpose quite effectively. Users have obviously learnt to rely on them. Making sure that System Tray items also include tooltips would make exploring KDE safer.

While we’re on the subject of tooltips… The K menu’s tooltip reads “Start Application” which left at least one user perplexed. The K menu’s tooltip definition could probably be rewritten to say something a little more engaging and should at the very least say “Start Applications,” in the plural.

Bigger Issues

Personalizing the Desktop

Most test participants were able to switch desktop background but all had trouble with the task. Their first instinct was often to use the K menu, select “Preferences,” then “Personalization.” They looked long and hard in those “Personalization” modules, convinced that what they saw as desktop customization was there. The second most looked at module was “Desktop” under “Look-and-Feel.” Some eventually found the “Background” module, some gave up the “Control Center” altogether and tried their luck with the Panel’s “Preferences…” option.

Right-click Paste

One user, when asked to move files from one directory to another (task 2), selected all icons, right-clicked to select cut from the context menu, then selected the destination folder, right-clicked again and selected paste from the context menu. A Windows user, he was expecting that this would move all files to the directory he had selected. What happened instead was that his files were moved to the selected directory’s parent directory.1

File Dialog Window

When users were asked to download a PDF document in their home directory, all of them did so without any major problem. At the same time, all showed some hesitation when dealing with KDE’s “Save As…” dialog window and specifically with the location shortcut box on the left. This box was the first thing that all users directed their attention to indicating, I think, that its visual and practical appeal is undeniable. Yet the choice and the ordering of its items and the current default are confusing.2

The box is built as follows: the first item is “Desktop,” and points to /home/<user>/Desktop; the second item, “Documents,” which is selected by default, points to /home/<user> as does the third item, “Home Directory.”

Save As dialog window showing “Documents,” the default directory, selected.

Throughout the test, I asked users to use the “Home directory” as the base for all their file management tasks. I made this choice because /home/<user> has always been the primary “building block” of file management a Unix system presented to its users. Generally speaking, KDE respects this convention: there is a “Home” icon on the desktop, a “Home (Personal Files)” entry in the K menu, etc..

Save As dialog window showing “Home Directory” selected and the current path still unchanged from “Documents.”

But the current file dialog window, of which “Save As…” is only one instance, is inconsistent in this regard. Though “Documents” and “Home Directory” are in fact the same, they appear to the users as separate. All test participants had to click back and forth between the two a couple of times to come to the conclusion that either both locations were the same or that one of the two buttons didn’t work properly as “nothing happened” when switching for one to the other.

Single-click vs. Double-click

Every single new KDE user I have ever casually observed was caught by surprise by KDE’s “active” single-click default policy. Testing confirmed this observation. All test participants have task after task made the same mistake of double-clicking when a single-click would have been sufficient. Unfortunately, I neglected to ask test participants how they felt about this particular point —I’m still new at this!— but I know from observing them that it did slow them down. For instance, it made copying and pasting files difficult by creating superfluous windows to close and forcing them to concentrate on a task they were otherwise comfortable with. In one case, one user nearly saved his files in the Trash because of this. As he double-clicked the Desktop icon of the “Save As…” dialog box, the system interpreted his action as a single-click on the Desktop icon followed by a single-click on the Trash icon, the latter occupying the space of the former after a single “active” click. The user eventually backed out but never realized what he was about to do.

I’m not sure why KDE makes single-clicking its “active” click. Historically Macintosh and Windows desktops adopted the same conventions for icon selection and document opening: single-click is for selecting, double-click is for opening or launching. This is not always the case anymore but I have yet to see a computer user single-clicking on an icon to open a document or launch an application. The notion that double-clicking “does something” is in fact so entrenched that it is not uncommon to see people double-clicking on hyperlinks when surfing the web.

The Big Issue: Embedded Viewers

While all participants were able to go through most of the test with various degrees of speed and confidence, one task, number five, consistently frustrated all of them. This was due to KDE 3.0’s default left click action, be it single or double-click, which is to show a document using an “embedded viewer.”

Opening a text file in Konqueror: embedded viewer

When users were asked to edit a text file located in their home directory, they clicked on the file’s icon opening the document with the embedded viewer in read-only mode. Because the interface gives no indications of what really happened half of the users failed to complete task five, adding a line of text at the end of a file. Even the participants who succeeded did so after much searching and experimenting, by either right-clicking on the document when previewed and finding the “Open With” item of the context menu, or by moving the document to the desktop which does not support embedded viewers.

The problem with embedded viewers did not only reveal itself with text files but also with compressed archives. Clicking on a compressed file’s icon displays the file’s content as if it had never been compressed in the first place, or as if it had been automatically expanded by Konqueror. At least one user started working on one of the compressed files as if it had been available to her (and the file system) in its uncompressed form.

Because from the user’s perspective a document appears to be opening normally when displayed by the embedded viewer, users realizing that they cannot modify that document react as if they had done something wrong. In other words, whereas sometimes users can recognize unexpected behavior from the computer and try to adjust, this is one of these cases when it seems like the only explanation for their difficulties is that they are at fault, not knowing something as simple as editing a text file. Users blaming themselves for the ineptitude of their computer systems is unfortunately quite common.

Opening a text file on the desktop: KWrite

Using embedded viewers by default makes sense for documents which are almost never edited directly such as PDF or Postscript files. It may also make sense in the case of documents which are primarily viewed or heard and not edited such as HTML or MP3 files. For all other file types, the current choice is more questionable and as long as there isn’t an easy way to tell the user that what he/she is looking at is read-only I think that it should be reconsidered. The difficulties presented by embedded viewers is even more apparent when you contrast it to how easily people are able to open files placed on the desktop when using embedded viewers is not an option.3

Well, that’s it for this first test. I hope that you could find something in it to inspire you to make KDE even better than it is now. I know I have learnt a lot from it. As I start writing the next test, I’d really like to read what you have to say about this one. The best comments will be added, with their author’s permission, to this article. You can drop me a line at <seb at viralata dot net>