Anyways. This comes from the middle of a discussion about ubiquitous computing and the idea of phicons.

“The Media Lab is, above all, a place where people build working models of wild ideas like phicons. Ishii led me to a wide, blank table surface. At the edge of the table were several wooden objects the size of large alphabet blocks. One of them was a model of MIT’s landmark dome. I picked it up and put it on the table. The blank table turned into a map of the MIT campus. I moved the phicon, and the map moved. I rotated the phicon, and the map rotated. Ishii handed me a second object, which was recognizable as a model of the I M Pei designed Media Lab building. I put it down on the table and the map shifted to that both the dome and the lab were in their proper places. I shifted one, then the other phicon; the map shifted to adjust, so that both buildings were always in correct juxtaposition to the rest of the landscape.”

I don’t believe we saw this TUI in particular this semester, but it strikes me as another simply perfect use of that kind of tabletop phicon interface. The model isn’t right for a lot of things, but for a map – it’s perfect. A brilliantly clear and concise explanation of what is a clear and concise TUI. That’s how you know something is right: it’s easy to explain.

On a similar but more reflective note, this semester in tangible user interfaces has opened my eyes to a lot of things. I’m increasingly inspired by the image and idea of truly ubiquitous computing. I will admit that my visions aren’t of a typical TUI world where EVERY object has a tag or a URL or embedded chips; they’re more like of a world where every surface is computerized and all your data is stored in the epic data cloud, so every wall and table knows who you are and what you want… anyways! Enough.

Strangely, one of the things I’ve taken away most strongly is something one of the guest speakers talked about. We discussed the difference between treating technology as your really smart friend versus treating technology as an extension of yourself, like a third limb. The conversation was actually about the difference between gestural input and voice input, but I’ve been thinking about it a lot in terms of social networks, a cloud of data, etc. For instance, if you go to a computer and open a browser window to Google something, that’s the computer-as-friend method. If, on the other hand, you were able to put your hand down onto the Surface and be provided with the most recent news relevant to your life (in terms of your work life, your social network, your interests in world news, etc) that’s more like a computer-as-limb method. And wouldn’t that be cool?

Anyways, it’s been an interesting semester and I look forward to continuing my train of thought in HCI next fall, assuming my schedule doesn’t rearrange itself. I’m working up the website to showcase the Dynamic Rehearsal project, and it’ll be up here soon! So for now, TUI section of my blog, farewell.

Dynamic Rehearsal Project from Rebecca Spitzer on Vimeo.

We did get a lot of great feedback, though, so that was really rewarding! I’m pleased with how much we got done, and it seems so was everyone else.

We’re spending this week wrapping everything up, making a display website and commenting up our code files. What a crazy couple of weeks!

The main thing that we’re excited about is the dynamic generation of screen elements, like the timeline at the top of the rehearsal workspace and the list of notes for each rehearsal. Each user has a newly generated color that appears on their timeline, on their notes, and in the key at the bottom of the screen. The video works (you can pause and play again!) and there is a slider that allows to you skip to new portions of the video. You can pull an individual note out of the stack, and it appears to the left of your rehearsal workspace for later review. And it looks good!

Screencaps:

Starting log-in screen (currently doesn’t have distinct user log-in codes, you just tap to start; we hope that in the future you can log in using the Surface as a fingerprint reader to automatically access your information.)

Overall look of all workspaces upon log-in, assuming four users are logged into the collaborative workspace.

Pulling an individual note out of the stack for review (The white highlight of the note marks it as “read” in the stack.)

Workspace for reviewing the most recent unread notes.

Panel to filter displayed notes by type or user; this button panel shows up when you tap on the timeline, allowing you to control what you’re reviewing.

Thumbnail spaces of past rehearsal. These are currently static, but in future implementations would have the ability to expand to the full size of the main rehearsal workspace we see to the right. This way, you’d be able to use all the real estate of the Surface size to review two rehearsal videos side by side for comparison.

Small key at the bottom of the Surface (can move to the opposite side depending on where you’re sitting) to show you who is logged in and what color refers to them.

Futhermore, there are other aspects of the project that are not clearly implemented at this stage; the project has a ubiquitous computing element consisting on a video camera that records rehearsals, RFID tags and readers to record when actors are onstage, and computerized pens that record notes being taken. Since there are no photos of the setup, here’s my small diagram:

We do HAVE each of the elements – a video camera, RFID tag readers, and the computerized pen, but none are linked together. The pen is the coolest part; it does indeed record the notes you’re taking! 

All in all, I’m pleased with the amount of work we’ve accomplished in only a few weeks; we have a working interface that is aesthetically presentable! I was originally very wary of C#, which the Surface is programmed in, and did most of my design work in XAML, which can control the look of the interface. In spending the last week integrating my XAML snippets with the existing C# code, however, I’ve become much more familiar with both languages. Hurrah! Our presentation is in 30 minutes and I hope to have some good feedback and discussion with attendees and other presentation groups; if we do, I’ll certainly discuss it here. There is also a video describing the whole project that has yet to be uploaded to the web; when it is uploaded, I will certainly link to it here.

At any rate, the central rehearsal review panel was redesigned to a vertical panel from a horizontal panel, which has pros and cons. The major problem with this is that the timeline across the top doesn’t have as much space to expand from side to side; a panoramic timeline is why we originally chose a horizontal interface. However, it turns out that resizing the timeline wasn’t as much of a problem as we thought it would be – it is slightly smaller, but still readable. The big plus side of moving to a vertical implementation is that we were able to get rid of the expanded timeline and move the notes under the video, which makes a lot of sense visually and frees up space elsewhere on the surface for other workspaces.

Those other workspaces are the previously discussed “Recent Notes” panel, which is actually smaller in our real implementation than pictured below, and the “Individual Note” panel, which is useful when a single note is pulled out of the stack for review. Additionally, I created a informational key to live at one edge of the Surface, providing data about the show itself and the list of current logged in users (and their colors).

The workspaces, as seen in Blend, not quite as implemented:

These elements are somewhat different in the actual implementation, which we’re working out now. It’s coming along slowly but is starting to actually LOOK good as well as function well. When we’ve finished the coding tonight and tomorrow, there will be more accurate screencaps to share!

Because so much of the interface involves dynamically generating elements of the interface based on who is logged in to the system and what data is most recent, we decided to start with a one of the most basic and difficult elements of the interface: the timeline. For every rehearsal displayed by the interface, there is a timeline generated to display which actors are onstage at particular times, as well as where notes were taken on their performances. To work with this, we started from scratch, building a number of original classes (ie: “Rehearsal”, “Person”, “Timeline”, “Note” objects, etc) and then writing methods to dynamically generate an image.

The second aspect of the interface we chose to implements was the parsing of notes files. We have a computerized pen that will record the notes a director takes, and we need to be able to import that information into our program, creating new Note objects and assigning them to the right people and timestamps. Though the notes do not display at this point, we have been able to parse them in – very exciting.

Personally, I focused most on the design of the interface of the project, which has been one of the more disastrous aspects of this milestone. Without any data to work with, I drew up a basic interface; it was difficult, however, to visualize how it would look with an actual timeline, video, and notes. It wasn’t until last night (well, morning, really) that we finally returned from Thanksgiving break to integrate our work together, and at that point we realized we were working with a terribly ugly and confusing interface. (I think we started calling it “the ugliest interface we’ve ever made,” actually.) Currently, it looks like this in the simulator:

(We’re not sure where that grey border at the bottom is from, why the interface doesn’t fit on the screen, or why some notes are black. Debugging will be fun!)

So! For the next deadline, we’ve decided to overhaul the interface. One of the biggest hurdles to get past is making our interaction workspaces show up in ScatterView panels, which are able to be moved around the Surface with gestural interaction. I will be reorganizing this main interaction space into a smaller ScatterView panel, as well as designing a smaller “Recent Notes” panel and a singular “Note” panel so that users can pull a note from the list to view individually. Hopefully, this will make the application easier to use, more collaborative, and more aesthetically pleasing. Everything needs to be resdesigned before we can continue to program interactions though, so we’ll be jumping in to the design tonight. We’ll also be filming a rehearsal tonight (of the show I’m working on, Gibraltar, which opens Thursday) so that we can have some accurate, useful source material for our final presentation.

Overall, the programming and design of the project have both been difficult; I think every member of our team is struggling at every turn. We have made huge amounts of progress since our paper prototype, though, so I have hope that we will be able to both fix the problems we’re facing and continue to implement more interactivity. I’m excited to see how the next two weeks go, and I’m also excited to see more progress on the projects other groups are working on, for everyone’s presentations on their functional prototypes were great!

Weiser’s paper, published in 1991, describes a world of the future where computers are all around us. He draws a parallel between computing and writing, poignantly noting that “carrying a super-laptop is like owning just one very important book. Customizing this book, even writing millions of other books, does not begin to capture the real power of literacy.” To combat that problem, Weiser envisions a world where we have many types of computers all around us. He describes these computers on three scales: tabs (little thumbnail screens), pads (size of a pad of paper), and board (large whiteboard sized wall screens). These computers respond to gestural interaction, are smart enough to sense our presence and preferences, and aid our everyday communication.

For 1991, this paper is extraordinary. Weiser was able to think critically about the future and where computing was headed, long before the proliferation of laptops, mobile devices, and other portable computing. While reading the paper in 2009, however, I am able to see a few places where the argument could be taken further.

First of all, Weiser has a strange conception of his board computers. He smartly describes shared displays and electronic chalk, but still notes the difficulty of using a keyboard with a smart board (which, if absolutely necessary) shouldn’t be difficult at all. More interestingly, he notes that since not everyone will be able to reach the top of a board, a menu bar will have to run along the bottom instead of the top. Today, we have pulled back from the once-necessary menu bars, finding more intuitive ways to send instructions to a computer. Here, Weiser could have taken his vision farther.

Secondly, Weiser moves into uncertain territory when he begins to describe the life of someone in the future’s computing environment. Asking for coffee is interesting, but seeing electronic trails of where her neighbors have walked throughout the morning, even if there is no video footage, does seem a little privacy-invasive. When he notes that she still prefers the paper form of the newspaper, I laughed to myself – did computer engineers not see the end of the newspaper industry, either? Throughout the rest of the day that Weiser describes, there continues to be some interesting aspects and some confusing aspects (why does she drink so much coffee?), but overall the description is not unlike the Microsoft video I posted a few months ago. I enjoyed his vision for the future; I especially enjoyed the conclusion that ubiquitous computing will help us overcome information overload, which I think is one of the greatest problems we face.

Ishii and Ullmer approached the question of ubiquitous computing in 1997 from a slightly different (and more modern) angle, looking not just to put little computers all around us but to make the objects we already have computerized, joining bits (computers) with atoms (physical worldly objects) to make a smarter world. After describing practically every TUI created before 1997, they introduce the metaDESK, the transBOARD, and the ambientROOM as three proposed parts of a solution.

Across all of the new models they describe, I was drawn most intensely to the ambientROOM, which would use ambient media (light, sound, etc) to alert a user to changes in the computing environment. Much as a darkening sky would alert us that rain was on the way, changing patterns in electronic light could alert a user to a flood of incoming messages or web traffic. Ishii and Ullman note that they “found the metaphor of light, shadow, and optics in general to be particularly compelling for interfaces spanning virtual and physical space,” and I agree.

Ishii and Ullman also use a set of physical objects they call “phicons” throughout their work: small, tangible objects that metaphorically correspond to a computer’s icons. A phicon might represent a source of information to recall, a physical handle with which to interact with the computational space. Personally, I agree with the ides of augmenting our existing objects, but I can’t see why we need to add new objects to our already physically jumbled space. When computational spaces make transition and change so easy, it seems like a step backwards to require them to interact with a static, unchanging object. We have enough objects in our world to interact with; why can’t we work computers into existing interaction modules?

For our project, Dynamic Rehearsal, we are certainly embracing a form of ubiquitous computing akin to what Weiser, Ishii, and Ullman describe. Through the use of multiple tools that can “talk” to each other, we are creating a smarter environment. Our installed video camera, RFID tags, augmented pens, etc. all capture activities that we already perform, and that information is accessible on a gestural interface provided by the Microsoft Surface. It empowers a user to work freely with the knowledge that their actions will accessible for later review, just like Weiser’s futuristic character can review the content and attendees of her past business meetings. Overall, we would be making the theater, a space which already exhibits an interesting juxtaposition of live, spontaneous, untouched emotion with fully wired, computationally rich technical systems, into a smarter space with a communal ubiquitous computing system.

In the end, we decided to simplify the interaction by synchronizing the responses to an interaction of every element on screen. There are three main points of interaction, and each one has the same result. A user can move a highlighted bracket across the main timeline, slide through the expanded timeline, or scroll through the panel of notes. At any of these interactions, the surface comes back with the same response: the highlight on the main bracket moves, the expanded timeline scrolls, the notes scroll, and the video clip moves to a new section. Hopefully, synchronizing these elements should simplify the program and the user experience.

Outside of those interactions, there are only a few other main workspace interactions to deal with: deleting and sorting notes. We’ll be working with these main workspace interactions this week as we move into our horizontal and vertical implementations in C#.

Our system is created for the general use of a theatrical production team: director, cast, and crew. A typical user has extensive experience in theatre, but not with computers. With the assistance of our system, they will be able to enjoy the benefits of computer-assisted analysis, review, and documentation of rehearsals without a basis of knowledge about computers or programming. 

The system will record the rehearsal process with video, audio, location information, and text. Video and audio will be provided by an installed video camera, location information will be provided by RFID tags worn by actors, and text will be automatically sourced from notes being taken by the director, stage manager, and other members of the production staff. With computerized pens, we will be able to digitally record the notes being taken by these users. All interactions take place within the theater space. 

The interaction with our system is primarily with the Microsoft Surface. The video camera will be linked to the Surface, automatically recording when the Surface application is activated in Rehearsal mode. Actors only need to wear their RFID tags when they are onstage (to record at what points they enter and exit the stage space) and production staff members only need to write with computerized pens, which electronically record, convert and transfer their notes to the Surface. All other interaction is with the Surface. 

At any point, any user can log into the Microsoft Surface application tailored to their production, which will store and sort the data recorded during rehearsals. Upon logging in, a user is provided with a tailored selection of recent rehearsals. For the most common user, an actor in the production, the Surface application would provide a number of workspaces: an area within which video and notes from the most recent rehearsal could be reviewed, an area within which the most recent unread notes for the actor could be reviewed, and thumbnails with which older rehearsals and notes could be accessed. With these workspaces, a user is able to both review their performance and quickly see which areas of the production need further work. 

To begin a user’s interaction with the Surface, they only need to place their hand on the surface to bring up their workspaces: recent rehearsal, recent notes, and thumbnails. Once these spaces are open, they can be moved around the surface and resized as necessary. A user can replay video, shuffle through notes, mark notes as read, and compare rehearsals to review their progress. Within the main workspace, there are three main elements. At the top of the space is a global timeline of the entire rehearsal. The times in which the given actor was onstage are noted on the timeline with a color-coded bar, as are all notes for the actor. On the left side space is the video display and an expanded timeline, showing a more specific portion of the rehearsal. The section highlighted in the expanded timeline is noted with brackets; identical brackets also appear on the global timeline. The expanded timeline can be pushed to the left or right to move to a different point in the video. On the right side of the screen is a list of notes relevant to the highlighted portion of the rehearsal and to the current user. Each note is displayed with markers to show what type of note it is and which user it pertains to. Notes can be marked as read or unread, and notes that the user is finished with can be discarded. The full list of notes is also scrollable at the top and bottom. Notes in the panel are also sortable by character or type by selecting the tabs to the right side of the panel. This notes panel can be collapsed into the workspace if the user is solely interested in the video. 

The system can also be user collaboratively, with multiple users. To join the review process, a second users needs only to place their hand on the surface, which will add their profiles to existing workspaces. The most recent shared rehearsal will occupy the recent rehearsal space, and any recent unread notes targeted to both users will appear in the recent notes space. Together, multiple users can watch video, discuss notes, and review their progress. In expanded rehearsal panels, information about all active users are displayed: on the global and expanded timelines, the appearances of all users are denoted with extra bars and in the notes panel, only notes pertinent to all active users are displayed. Information about a certain actor can be deactivated by selecting the appropriate tabs to the right side of the notes panel. Multiple users may choose to open more than one expanded rehearsal workspace to compare videos. 

With this system, a theatrical production team gains the ability to record, analyze, and review their progress without having to alter their usual rehearsal routine. Information in the form of video, audio, location, and text is put together to provide a comprehensive record of the rehearsal process. The system will aid in the review process and facilitate a more collaborative approach.

Our team also worked to put together the following storyboard elements, though recent conversations have led to some basic changes.

a bird’s eye view of our setup
(the large purple areas are audience seating)

our main interaction screen on the surface

our main collaborative screen on the surface

other workspaces

a more detailed interaction storyboard

More about our proposed changes to come!

Our proposed interface will consist of four tangible objects (or sets of objects) and will be used in two phases. In the first phase (the recording phase) three objects will be used to record data about a rehearsal. An installed video camera will record any scenes being practiced. Location tags worn by actors will record who is onstage or offstage during said scenes. A computerized note-taking system (ie: tablet computers, computerized pens, etc) will be used by people watching the rehearsal to document critiques or congratulations. In the second phase (the review phase) a single object will be used to review data about past rehearsals. A Microsoft Surface (or other multi-touch surface) will display video footage of rehearsals tagged with notes at the moment the notes were taken during the rehearsal. Notes can also be reviewed separately or sorted by note-taker or note-receiver. Video footage will also be sorted by actor, so an actor can view a whole rehearsal or only their scenes. All of this data will have been processed and analyzed by the Surface.

Overall, the project should help theatrical teams to get more out of rehearsals and create a more collaborative process of review and critique, both for singular rehearsals and for the entire process over time. I’m specifically drawn to this project as a stage manager; I have been working in theater since I was 14 and have often been frustrated by the process of giving notes to actors and reviewing rehearsals. For example, being able to quickly take lines notes on a tablet that would be automatically tagged to video and sent to actors to review at a later date would be amazing. Though the project is extensive, we will hopefully be able to implement enough aspects to prove its worth as a tangible user interface.

ANDREAS LUTZ – Because clicking is so 90s! from andreaslutz on Vimeo.

I’m skeptical as to how well it works, so I’d like to try the site out; however, I’m in the library right now, and other students tend to frown upon using voice activated software in quiet spaces. I could probably get away with the giant hand gestures, but I don’t think I should jeopardize my relationships with the neighboring thesis carrels just yet.