Simon’s Blog

There’s been some speculation that the forthcoming 3G iPhone may include a new patented Apple technology - a so-called “integrated sensing display”.   This is an LCD screen, with image-sensing elements inserted in between the pixels in the LCD panel.   The images detected by the many thousands of sensors can be “stitched” together in software to create a single image.    The idea for the 3G iPhone would be to enable video chat/conferencing, without the need to use additional space for a separate camera.  Indeed, Apple’s patent suggest this is an application the company thinks is important:

One use and benefit for such a panel is video conferencing: a user can maintain eye contact with someone on screen because the camera is “in” the screen. In addition, portable devices, such as portable digital assistants (PDAs) and cell phones, have very limited space for displays and would benefit if additional real estate were not used for a camera.

For sure, that would be a unique selling point for iPhone (although people are, on the whole, not really interested in video chatting on mobile phones - people want 3G for downloading data fast).  However, this begs the question - is enabling “better” mobile video chat/conferencing really a killer application of this new technology?  I’m not so sure it is. It seems to me that Apple has not described in its patent what are potentially the most interesting applications of this new technology.   To my mind, the most interesting applications aren’t so much that integrated sensing displays have the equivalent of a single video camera built-in; but rather that they could have the equivalent of two or more video cameras built-in.

With two or more video cameras built-in to the display, it would mean, for example, that the screen could “see” in three-dimensions.  This would open up some interesting possibilities, such as  new types of gesture-based user interfaces where the user doesn’t touch the screen, but simply moves their hands in the air; or even just moves their eyes in particular ways.   New user interfaces are just one application of such “3-D video capture” - there are others too; and, of course, there are other “non 3-D” uses for devices with large numbers of image sensors.  It wouldn’t be a surprise if an application that makes use of the fact there are many spatially separated image sensors (a.k.a cameras) turns out to be the killer use for this technology.