Multi-touch interface

All unapproved Citizendium articles may contain errors of fact, bias, grammar etc. A version of an article is unapproved unless it is marked as citable with a dedicated green template at the top of the page, as in this version of the 'Biology' article. Citable articles are intended to be of reasonably high quality. The participants in the Citizendium project make no representations about the reliability of Citizendium articles or, generally, their suitability for any purpose. This article is currently being developed as part of an Eduzendium student project. The course homepage can be found at CZ:Special_Topics_2010. To provide students with experience in collaboration, you are warmly invited to join in here, or to leave comments on the discussion page. The anticipated date of course completion is 13 August 2010. One month after that date at the latest, this notice shall be removed. Besides, many other Citizendium articles welcome your collaboration!

Tom Cruise in Minority Report (2002) using gestures similar to those used in Multi-Touch Technology

A multi-touch interface is a human-computer interaction technique which allow users to compute without mechanical input devices (e.g., mouse, keyboard), often by fingers touching and moving across a display screen. After several decades of research by various universities, companies, and research groups, multi-touch exploded onto the commercial scene with the release of both Apple's iPhone and Microsoft's Surface tablets in 2007. Multi-touch is now a standard feature in smart phones and tablet PC's, and its use is growing rapidly.

Even before iPhones made multi-touch ubiquitous, films and television began depicting multi-touch in shows like CSI, NCIS, or Fringe, where large wall touch displays were used to scan through criminal evidence. Multi-touch screens also began to be used by weathermen, ESPN (March Madness Selection Sunday), and news reporters (2008 Presidential Election).

Multi-touch systems use a touch screen, table, wall, or touchpad, along with software that recognizes multiple simultaneous touch points, as opposed to computer mouse pads or ATM machines which recognize only one touch at a time. A variety of hardware means exist for detecting touch points and movement, including heat, finger pressure, high capture rate cameras, infrared light, optic capture, and shadow capture.

How does Multi-Touch Technology work?

Multi-touch signal processing and recognition

Multi-Touch Technology (MTT) is a type of Human Computer Interface that’s grown in popularity since 2007 when Apple released the commercially successful iPhone. It was one of the first implementations of MTT that was targeted for the public at large. In MTT devices, the user interacts with the computer by touching and making gestures on a specialized touch-sensitive display surface that can detect multiple points of contact (touches) as well as gestures.

One of the major features to the technology is that the interface replaces a number of physical peripheral devices with a soft replacement (i.e. non-tactile) in a single self contained device, thereby increasing the portability and functionality of the device. Though the specific input interface varies amongst devices and applications, they usually include:

• A virtual, or soft, keyboard to allow the user to input text and numbers.
• Ability to simulate a mouse, by dragging a finger across the display.
• A number of gestures that have specific application functionality (i.e. the pinch which zooms in and out).

Generally, multi-touch devices share a similar architecture composed of three parts: the touch-sensitive display surface, controller and software drivers. The surface registers the touch event and sends the signal to the controller. The controller does the signal processing (filtering noise, determining pressure signature, and analog-digital conversion) necessary to compute the coordinates of the touch. The controller sends this data to the software which handles the event.

Touch-sensitive display surfaces supporting MTT can be implemented in a number of ways. Among them are capacitive, resistive, infrared, and surface acoustic wave sensing mediums. This article focuses on capacitive and resistive, which are the types of technology used in mobile phones.

Capacitive Touch Screens

How Stuff Works, Mutual Capacitance

Capacitance-sensing multi-touch screens are the most prevalent implementation behind multi-touch cell phones.. There are a number of technologies within the capacitance-sensing family to detect multi-touch but, in smart phones, mutual capacitance sensing is the most dominant. The technology works by having a capacitive material whose local electrical charge gets changed when touched by your finger (or any other conductive material).

According to Apple’s patent (US 7,663,607 B2), in Mutual capacitance there is an electrically conducting medium like Indium Tin Oxide (ITO) which is arranged in the following set up. The medium is in an array formed by two different layers. Driving lines form the first layer, and sensing lines, which usually run perpendicular to the driving lines, form the second layer. The driving lines are connected to a voltage source and the sensing lines to forming a capacitive sensing circuit. During a touch, the local electrical properties of the screen change and are detected by the nodes.

Perhaps the main limitation to this approach is that it can’t be used in situations where the user has to wear gloves. Another limitation is its expense. Limitations aside, it has a high-resolution, high-clarity, and is not affected by environment. It should be noted that on February 16, 2010 Apple was awarded a patent for capacitive multi-touch which covers two implementations: single and mutual capacitance (patent no, US 7,663,607 B2).

Resistive Touch Screens

Resistive sensing multi-touch screens are growing in prominence for several reasons:

• They allow the user to use gloves
• The technology is significantly less expensive
• The devices have lower power consumption than their peers.

Resistive multi-touch screens consist of two glass or acrylic panels that are coated with a conductive material like Indium Tin Oxide and are separated by a narrow gap. When the user touches the display, the conductive layers are connected establishing an electrical current which is then measured and processed by the controller. A limitation of this technology is that it is not as robust as its capacitive counterpart. Currently, attempts to add additional screen protection limit device functionality, significantly limiting the ruggedness of the device.

A brief history of Multi-Touch technology

There have been a number of significant inventions, dating back to the 1970’s, that have been instrumental in the development of multi-touch technology. The following section is adapted from a broad history of multi-touch already written by Bill Buxton, which is available on his website. Buxton is considered one of the leaders in human computer interaction research, and his paper on the history of touch technology is frequently cited.

Bill Buxton using ActiveDesk (1991-1994)

One of the first, and most significant, inventions relative to the creation of multi-touch technology was the development of a touch screen able to recognize a single touch. The first accepted example of this technology is the PLATO IV. PLATO IV was developed in the 1970’s by the University of Illinois Computer-based Education Research Laboratory. This device was a computer assisted educational device which recognized single touch, non-pressure sensitive inputs from the user. The touch panel consisted of a 16 X 16 grid of sensors and infrared-light emitting diodes around the plasma panel, allowing the computer to identify from any of the 256 regions (Sherwood, 1972).

In 1983, Myron Krueger completed a prototype of the VIDEOPLACE which allowed a user to interact with images on a vision based system via a set of predefined gestures. Though the system did not rely on touch, the significance of this work was that it was able to recognize a number of gestures, such as, pointing, pinching, and dragging.

In 1985, the first multi-touch tablet using capacitive touch screens was developed by the Input Research Group at the University of Toronto. This tablet was capable of sensing more than one point of contact at a time, determining their locations and measuring their degree of contact. The additional significance of this tablet, at the time, was its method of scanning user input and its ability to detect contact with a high degree of resolution.

In 1991, a multi-touch desktop display was developed that aimed to give the user’s desk and workspace the properties of an electronic workspace was developed by Rank Xerox EuroPARC. The significance of this project is that rather than using capacitive touch screens, it used a computer controlled camera in combination with a projector above it to sense the user input. In the set up, the camera detected the location of the user’s input and what they were pointing to. The projector displayed feedback and electronic objects onto the surface.

In 1992, IBM and Bell South released the first touch screen smart phone called Simon. The input depended on both stylus and touch replacing physical buttons with a virtual (or soft) keyboard on the screen. Simon’s functionality included that of a phone and a PDA. One of the limitations of this technology was that it could not differentiate pen contact from finger contact.

In 1997, the T3 was invented by Alias|Wavefront which allowed the user to use more gestures including navigating, panning, rotating, and zooming that responded quickly to the user input. Additionally, the user could combine these basic movements to form more complex ones. This company was later bought by Apple who used FingerWorks designed technologies in their multi-touch devices.

In 2004, the first (commercially available) transparent multi-touch capable screen was released. This device, called the Jazz Mutant, was targeted towards musicians.

In 2007, two commercially successful multi-touch devices were introduced. These two devices were the Apple iPhone and Microsoft Surface. The iPhone was a smart phone that used multi-touch technology and was capable of detecting two points of contact simultaneously. The iPhone was also capable of recognizing basic gestures including, but not limited to, the pinch and swipe. This phone used a capacitive-sensing touch screen. The iPhone was geared for the general public and was integral in making multi-touch technology ubiquitous. Microsoft Surface was a table surface able to sense multiple user touches and gestures simultaneously on a single surface. Because of its price of $14,000, it was geared towards corporations and institutions, where it was successful. Microsoft Surface used optical technology to detect input..

Programming in Multi-Touch

Multi-touch interfaces must inherently allow concurrency in the way they detect user input. Concurrency allows for multiple points of contact on the hardware surface to all be simultaneously detected and interpreted at various stages by the software engineer. The handling of concurrency is typically a low-level process performed by the operating system through interrupts, polling, or signals. The operating system can subsequently package this information in an easily accessible fashion through the application programming interface (API). This abstraction is the key to the relative simplicity of programming for multiple points of contact.

The “Observer” design pattern is a software design pattern used by many APIs to handle user interface interaction, and it is applied to multi-touch interfaces in a similar fashion. It allows a programmer to designate an object, in this case the touch interface, and observers that will be informed of designated changes to the object by activating an observer’s specified method. Instead of handling multiple threads or processes for each point of contact, the operating system provides a single stream of fast-paced, but sequential, “events” to the object’s observers. These events each allow access to information about the state of the touch interface. Through the observers, the software engineer can decide what to do with each event by extracting information from them. The type of information that could be extracted includes the particular point of contact and its position, timing, inertia, etc. In this sense, multi-touch programming is very similar to single-touch programming except that the programmers must keep track of which point of contact they are handling at any given time. The programmer can choose to handle as few or as many points of contact as their application needs. However, the maximum number of simultaneous contact points is likely constrained by the device’s firmware due to hardware limitations.

Many APIs provide additional abstraction in a variety of ways. A popular approach to simplifying multi-touch interaction comes in the form of pre-defined shapes or patterns typically referred to as “gestures”. Android 1.6 introduced a gestures package which provides a statistical approach to demonstrating if a pattern drawn on the touch interface by a user is equivalent to any member of a pre-defined set. The feasibility score is due to the inexactness of this form of communication between the user and the application. Windows 7 provides similar gestural functionality and even allows legacy applications to benefit from touch interfacing by handling some events if they were not interpreted by the application. This approach allows Windows to augment the intuitiveness of your application by allowing interactions such as pushing buttons, moving windows, and panning on scrollable views. Letting Windows handle these intuitive events has the additional benefit of presenting uniform behavior across software applications.

Legal Issues in Multi-Touch

For copyright purposes, it would be convenient to say that multi-touch technology was invented by so-and-so, and that the rights belong to that person. Unfortunately, as with many cutting edge technologies, there is no single “inventor.” Mimicking 1976, when two independent groups came up with very similar ideas to solve the key-exchange problem, there were several different researchers and research groups working on multi-touch technology. Each of these researchers, or groups of researchers, approached the problem in a different, yet very similar, way.

In 1983, Myron Krueger was implementing and using the hand gestures we think of as commonplace today (pinching to zoom/scale and resize, etc.) . Did Myron Krueger invent them? Nobody is really sure who came up with the ideas for these gestures; however, many of these research groups began to implement and use them in their own multi-touch implementations. When Apple Inc. released the iPhone in 2007, they concurrently submitted patents for many of the gestures, as well as the multi-touch technology that those researchers had been using for years. Could Apple really be the first company to take these technologies/ideas to the U.S. Patent Office? Did Apple infringe on any of these researcher’s/developer’s patents? The issued has not been settled.

As of mid 2010, there are a number of companies producing cellular phones with touch screens, and multi-touch screens. A number of these touch screen phones have begun to look more and more like the iPhone. In addition to looking like the iPhone, these phones also respond to many of the same gestures as the iPhone. The similarities in all of these different implementations was bound to result in some sort of litigation and, in the years since Apple’s release of the iPhone, there have been a number of legal issues regarding the multi-touch technology used by each of these phones. The following list of lawsuits is in no way a comprehensive list but is used to give the reader an idea of the type of legal issues that are currently being faced:

Apple’s Lawsuit: Apple vs. HTC

On March 2, 2010 Apple Inc. filed suit against HTC Corp. for violating more 20 of Apple’s Patents related to the iPhone and other Apple Products (U.S. Patent Numbers 7,362,331; 7,479,949; 7,657,849; 7,469,381; 5,920,726; 7,633,076; 5,848,105; 7,383,453; 5,455,599; 6,424,354; 5,481,721; 5,519,867; 5,556,337; 5,929,852; 5,946,647; 5,969,705; 6,275,983; 6,343,263; 5,915,131; and RE39,486) . The patents listed in the lawsuit cover a wide variety of technologies including, but not limited to, touch screen command determination using heuristics, unlocking a device with a gesture, display rotation, camera power management, and signal processing. Many news and technology sources have speculated that this is more of a shot at Google/Android than it is at HTC. Though none of these patents are for multi-touch directly, many of them are for gestures and gesture based capabilities provided by Apple Technology. This court case has yet to see a courtroom.

HTC Counters: HTC vs. Apple

On May 12, 2010, HTC filed a countersuit against Apple Inc. citing 5 of HTC’s patents (U.S. Patent Numbers 6,999,800; 7,716,505; 5,541,988; 6,058,183; and 6,320,957). Only one of these patents (6,058,183) relates to multi-touch technology; however, the lawsuit probably would not have come about if the original lawsuit by Apple was not filed. This case, like the other, has yet to see a courtroom.

Elan Microelectronics files a complaint with ITC

On March 29, 2010, Elan Microelectronics (EMG) filed a complaint with the International Trade Commission over Apple’s use of multi-touch technology in just about all of its products. The patent in question is U.S. Patent Number 5,825,352: “Multiple Fingers Contact Sensing Method for Emulating Mouse Buttons and Mouse Operations on a Touch Sensor Pad”. In EMG’s complaint, they have asked for a complete ban on the import of iPad, iPhone, iPod Touch, MacBook, and Magic Mouse .

The Future of Multi-Touch

The possibilities for the uses of multi-touch are endless. Through the years we have seen futuristic versions of multi-touch in television and on the big screen; but when are those things going to be real for the rest of us? In this section I will mention a few sectors where I believe that multi-touch is going to have a big impact in the future.

Education

Most of us fondly recall the horrible sound of chalk squeaking on the chalk board, or the horrible smell of whiteboard markers from our days in school. What if we could eliminate white boards, black boards, etc. and replace all of them with multi-touch wall displays? While the initial outlay for such technology would likely be hefty, costs for consumable supplies such as chalk, markers, projection bulbs, erasers, etc. would be eliminated. In addition, the time to restock supplies or find shared projectors would be saved. In addition to being incredibly economical, these touch screen displays would provide teachers with improved functionality, responsiveness, and comfort. What teacher likes being covered in chalk day in and day out?

Military

In the world of the military, it is common for things like cellular phones, laptops, etc. to get roughed up. Military members may take their cellular phones or laptops out in the field because they are incredibly useful tools. Knowing full well that these “tools” may get damaged; however, military members are hesitant to buy technology such as the iPad/iPhone for use in the field. Anyone who has sat on their iPhone and listened to it crunch will agree that it is probably not a good idea to take such a device out into the field where anything can happen. Recently, companies have begun producing multi-touch technology that is more durable. For example, HP just has released a multi-touch enabled notebook that is “engineered to meet the tough MIL-STD 810G military-standard tests for vibration, dust, humidity, altitude, and high and low temperatures.” As this type of durable technology becomes more available, military members will be more willing to spend their hard-earned money without fear of destruction. In addition to members of the military buying these products, governments will be more willing to issue devices like this for use by troops.

Health Care

Anyone who has been to a doctor’s office or hospital has seen the massive amount of paper and x-ray sheets that are used on a daily basis. These resources are not free, there has to be a cheaper way to view/read diagnostic images and patient medical histories. Tablet PCs and devices such as the iPad would be perfect for a complete overhaul of the system. Doctors could each have an iPad like device which he could download his patient’s medical history/x-rays/diagnostic images to and have all the information he/she needs in one place. In fact, this plan is already in the “testing” phase with a number of hospitals trying it out. According to John Halamka of Harvard Medical School, “the combination of lower hardware acquisition costs and relative lack of learning curve (since many people already have smartphones) could foster widespread adoption of the iPad in health-care settings and pave the way for electronic heath records to become the norm.” If this idea catches on, it could save the healthcare industry millions of dollars and increase hospital efficiency significantly.

Downsides of Multi-Touch

There are some barriers preventing multi-touch technologies from becoming the ubiquitous method of interfacing with computer systems. Some of the more objective barriers such as the high cost of these devices and the lagging hardware support for high performance software will be alleviated with time. These aforementioned barriers are typical of any nascent technology. While the cost may always remain relatively high compared to a traditional input device (e.g., a keyboard or mouse), the prices will be driven down as the technology matures and becomes widely accepted. An additional barrier is the relative complexity of the technology compared to simpler input devices. This becomes a detriment to the device in situations where durability is required (e.g., military applications) or quick, cheap repairs are necessary.

The more subjective barriers must be approached in different ways particular to each problem they present. Many users prefer the tactile feel of traditional keyboards and text input devices. Reasons for this preference are so diverse that a one-size-fits-all approach to solving such a broad problem is almost always inadequate. Some users perceive their typing to be faster if they receive physical feedback after each key press. Developers have attempted to rectify this problem by delivering haptic feedback; a vibration under the point of contact when input is successfully received. Some users enjoy the ability to use their input device while it remains in their peripheral vision. This benefit is lost on devices, particularly cell phones, that integrate their display device and their multi-touch interface into one unit. There is also a large segment of handicap computer users that are reliant on the physical layout of their input device. An increasing amount of attachment devices are becoming available to compliment the multi-touch interface and allow the user to switch between the touch screen and traditional methods of data input. Multi-touch may not become the exclusive interface to computers, but it can become an integral part of an environment of several input devices that work in a similar harmony as the keyboard and mouse.