Adrian Cheok, from University of Singapore and Keio University of Japan, delivered an inspiring keynote at the Mobile HCI Conference early this month in Stockholm. His vision is that communication will move from informational to experiential. Experiential communications use visual, auditory and tactile imagery to capture attention, stimulate the senses in many ways: physically, rationally and emotionally. Adrian showed a number of prototypes on experiential communication that he is working on with his team:  e.g. giving hugs to your pet friends, hugging distant people (as explored in the Huggy Pajama project), enriching paper communication. He also showed examples of communication involving food printing, as part of his project called Food Media. The project is exploring new digital food and cooking media for virtual and remote communication in order to enhance remote family communication networks and allows for novel intergenerational interactions between family members through food and food activities. One platform is based on a

3D food tele-generating printer that transforms eating into an interactive communication experience by creating edible layered messages. It produces food by printing layers of flavored agar pixels that are liquefied by a silicon heater and then solidify upon contact with the environment. The system prints the layered information according to drawings, photos, or texts submitted by the sender.

Personally I would be even happier if this way I could plan and cook my dinner remotely from my computer in a way that just after being back home from gym or work, all I have to do is setting my table!

3D food printer

Projected social interaction from MIT PoCoMo

Several geeks, keen on developing interfaces for mobile devices, gathered last August at the Mobile HCI conference in Stockholm.

The MIT fluid media group brought one of the most inspiring presentations. They presented their vision in future projected social interfaces. Personal projection devices are becoming more common. Last year 500.000 picoprojectors were sold worldwide.  Their price is about 30 USD per unit. A couple of mobile phones with embedded picoprojector are out in the market: the  Samsung Beam i8520 Galaxy Beam “Halo” and the Indian manufactured Spice M9000 Popkorn.

Leveraging on these devices,  it is thus possible to imagine a scenario where mobile phones can project images on walls and track other ones’  images by the use of the phone cameras.  The phones would be aware of each others’ projections and use them for social interaction, as proposed by the MIT PoCoMo team (that’s the name of the project).

In my opinion this scenario represents an evolution of AR as conceived today, where we use mobile phones to see-through and add information about the surrounding environment. Using projectors we will be able to add more pieces of information on the real things around us, making possibly easier the process of augmenting reality.

I’ve just spent a few days at InfoCommShow 2011, the biggest event for audio/visual systems, solutions and devices in the US and probably worldwide, which took place in Orlando. It attracted more than 900 companies to exhibit and more than 30.000 visitors. The event was also running several educational sessions and demonstrations.

InfoComm is a trade where to find most A/V solutions for professionals and enterprises. To ease navigating the exhibition, the floor was divided in a few thematic areas (which is quite consistent with the Orlando spirit of hosting several theme parks): unified collaborative conferencing, 3D, digital signage, audio, lighting and staging. In fact around the show you could find displays and projectors, telepresence systems, multi-touch technologies, A/V cables, racks, cameras, microphones, mixers, speakers and more. Most of the best of breed companies of the sector were present, including Cisco, HP, NEC, Samsung, Bose, Barco, LG, Sharp…

The A/V market is flourishing, with a positive CAGR of about 9%. Yet the market is changing: convergence with Information Technology is stronger and stronger, most content is today transported on IP and cables have been joined by wireless as well.

A quick glance to show floor would have given the impression of being in Time Square, New York, more than at the Orlando Orange County Convention Center, due to the massive presence of displays all around: LED displays of different sizes and form-factors, bezel multi-monitor displays, and multi-projector displays. The latter were mainly powered by an edge-blending software solution patented by Scalable Display Technologies, an MIT spin-off, capable of running up to six projectors on one GPU. Scalable enables to create highly immersive displays with a relatively inexpensive solution. Some of the LED displays were used to visualize 3D content, but requiring eyewear, which severely limits their scope. Autostereoscopic displays were not present at InfoComm.

HP Halo new telepresence prototype showing touchscreen control and blended communication

Several telepresence systems were demonstrated with real-time conferencing sessions. I personally attended a demo of the new Halo prototype by HP, which integrates a touch-screen control instead of the classic IP phone and can integrate other incoming videocalls, and demos of other telepresence systems by Polycom, Cisco, LifeSize. The telepresence main cues are currently HD video quality and a set-up to provide virtual eye gaze (to provide realism), stereo sound (to identify who is talking). From the technological point of view not much has been done in order to increase immersiveness: when I asked if 3-D displays were going to be adopted, the experts told me that they envision them to support presentation delivery but not to augment the visual impact of the virtual meeting participants. A few companies, like TelepresenceTech, were proposing to add a fake 3-D effect leveraging on the Pepper’s Ghost effect: the solution consists in putting a semireflective glass at 45° in front of the LCD display to reflect a selected background and creating the sensation that the virtual person was standing beyond this background.

TelepresenceTech system to deliver a virtual 3-D effect, based on Pepper's ghost technique

VGO Communications, an American start-up recently getting $4,5 million funding in Virtual Capital, created a “telepresence” robot that moves on wheels and integrates an LCD display on its head together with speakers and micro. The robot is telecontrolled by the person that wants to remotely communicate. What the VGO robots do is in fact to carry our “physical” avatar around. This concept is particularly interesting for disabled people who want to extend their presence in the place where they live, without installing videocams, micros and speakers in each room.

The VGO Communications telepresence-robot talking with a human...

If you want to understand how a commercial on TV or on paper, a brand, a package, or a product are perceived by a consumer, you have to look inside their brain. That’s at least what Nielsen, a giant  providing information and measurement for a comprehensive understanding of consumers and consumer behavior, believes. Nielsen made $5 billion of revenues out of this business last year.

Now Nielsen has decided to strongly invest on neuromarketing, a new methodology based on looking inside the consumers’ brain using brain imaging techniques. That’s why Neurofocus, a neuromarketing leading company, was acquired by Nielsen a few days ago.

The New York Times, while talking about Neurofocus and Nielsen, explains that:

Neuromarketing is the application of neuroscientific research to marketing, advertising, and entertainment content and messages. Neuroscience research has in recent decades revealed important new discoveries about how the human brain is structured and how it functions. These findings enabled NeuroFocus to develop patented technologies and proprietary techniques that provide greater accuracy and insight into consumer research.

The NeuroFocus mind wireless EEG measurement headset

But how is Neurofocus investigating our brain reactions? Let’s take the case of studying a TV commercial. First of all you have to wear an EEG cap on your head, in order to measure the brain activity while being exposed to the media stimuli (watching the commercial…). Afterward, some keywords that have been carefully associated to the commercial under investigation, are shown onto the screen: those words can refer to the brand, to the message, to the emotions that specific experience is evocating.

Let’s say that we want to understand if  people are perceiving “speedness”, “aggressiveness”, “safety”, “design” in a sportscar commercial. While displaying those words, the EEG measures the brain reaction and creates a reference basis for the test. The commercial itself is then projected, followed again by the same keywords.

In the second round of the keyword projection, the brain reacts differently to each word because of the experience just elicited by the commercial. This reaction is measured with a technique called P300, that in neuroscience is a well-known electrical signal change in the neural activity that follows a relevant visual experience.

Thus the words that for that person were relevant while watching the commercial are identified, without having to ask anything or submitting a traditional questionnaire.

My bet is that in the future those systems will be used to support us also for daily tasks, for instance to instantly find images out of large personal database just following our brain inspiration (which in fact Microsoft Research is somehow already investigating), or to navigate through a movies catalog just being guided by our emotions.

I recently visited the Deutsche Telekom Labs in Berlin, known as T-Labs. T-Labs are carring out basic and technology research that focuses on developing future-proof applications in a wide range of areas. Their research projects develop existing technologies based on the continuous scouting of forward-looking trends. One research team there is looking to usability as the mean to greatly improve the quality and, with it, the sales of interactive products. This team is exploring new ways of interacting with mobile devices.

They had the idea of a touchless data entry system that is based on the interaction between the magnetic fields around a device and a properly shaped magnet. The magnetic field that surrounds the device is generated by a magnetic sensor (compass) that is embedded in the new generation of mobile phones such as Apple iPhone 3Gs and 4G, and Google Nexus one. The small magnet could be worn for instance on a finger, if it is put on a ring: as the hand is moved around the mobile phone, the 3D interaction is measured and transformed in a command. This gestural interface extends to the peripheral area of the device without needing any extra element. The interface can be used to perform certain tasks, based on the pattern of the movement. These tasks can vary from entering a text or digit into the phone to turning pages or zooming while reading an e-book.

Here you can download and see a video of MagiWrite in action.

MagiWrite uses the compass to sense gestures.

This year I had not the opportunity to go and check CHI 2011 conference (Computer Human Interaction), which I consider as one of the most instructive conferences on interfaces, with a strong participation of both academy and industry research centres, like Microsoft, IBM, Xerox etc.

Anyway I had a look to the best awarded papers published online and one came to my attention: Microsoft Research and the formidable Desney Tan, one of the gurus of innovative interfaces in that company, have come out with such an easy to understand idea that I wonder why no one else have had it before (at least to my knowledge).

Our houses are cabled with electrical wiring and full of appliances: those generate some electromagnetic noise out in the surrounding environment (that noise that, at least in the past, was for instance annoying our AM radios…).

Now the MS researchers had the idea of using our own body as an antenna and measuring how we interfere with this noise, using machine learning and artificial intelligence to be able to recognize our position inside the house plan and the gestures we do. In this way we could move one hand on the wall to control an appliance in the same room. Check all details of this concept in the awarded paper.

Analyzing the electrical wiring noise, the whole house can become an interaction surface.

You may have noticed that currently stereoscopic displays have only one parallax direction, the horizontal one.This means that you can look explore an object tridimensionally only moving to his right or left, but not up or down, which would require vertical parallax capability of the display. This applies to the 3D Cinema or TV set solutions that are based on eyewear, or to autostereoscopic displays like those used on the Nintendo 3DS.

Even more complex displays, like those developed and commercialized by Holografika, a hungarian company that patented a ray projection holographic display, are offering a mono-parallactic stereoscopic view: using several optical light projecting modules that are hitting each singular voxel (volumetric pixel) onto the “holographic” screen, they are able to recreate a wider angle view then other techniques can today. Holografika’s technique consists in projecting several different angular views of an object on the screen that can be perceived by the viewer just slightly moving on his left or right, as we would do in a real life object exploration.

Now if you try to watch the upper or lower sides of this virtual object, you will get always the same view of it, as in fact the system is not able to perform vertical parallax, giving back a sensation of fakeness.

A new kind of display has been invented by Holovision that can create high-resolution, three-dimensional moving images that can be viewed with full parallax by people in different locations without special eyewear.

Holovision volumetric display based on MEMS.

This display leverages on the future features brought by MEMS (Micro Electrical Mechanical Systems) that in the next years will be used to create a matrix of moving display elements to guide light at the level of individual display elements (such as voxels). MEMS will be able to spin light at individual voxel level in all directions, thus dynamically recreating a volumetric image.

Array of spinning microlenses at voxel (volumetric pixel) level

Such a revolutionary volumetric display system will have also consequences in terms of all the data that has to be elaborated and transmitted, requiring graphics processing units and transmission channel capabilities to perform several times more than today.

I’m fascinated by gestural interfaces. About an year ago I bumped into one that was developed and demonstrated by the Hasso Plattner Institut in Berlin, a research institution created in 1998 by professor Hasso Plattner, co-founder of the well-known software company SAP.

Here the idea was to develop an interface for a device which does not have a screen nor a keyboard. This interface has been called “imaginary interface”. Using video analysis the system detects the user’s hand silhouettes from a small camera hanging from the neck. One hand, that has to show an L using thumb and forefinger, is used as a reference both for the person who is sketching something in the air, and for the system that sets this hand as the origin of a “virtual whiteboard”.

The other hand is used to actually draw on this “whiteboard” and the resulting drawing is transferred via cellphone to a remote PC. When you want to use the virtual ink on the whiteboard, you have to connect your thumb and forefinger in a circle.

Have a look to the video that shows how simple this can be.

This imaginary interface will let us sending some indications to a friend, taking a short note, sketching a memo just using our free hands…

TMS or tDCS are acronyms for technologies that in the future will be associated to “computer-to-brain” interfaces, the new frontier in connecting humans directly to machines.

Today BCI or “brain-to-computer” interfaces are already existing and used in several areas, from assisted living for disabled people to entertainment application as an innovative controller for games or toys [see this post on BCI].

TMS stands for Transcranial Magnetic Stimulation. It consists in creating a controlled magnetic field that penetrates inside the human brain. This non-invasive method (meaning that no surgery is needed…) is currently used to treat some neurological conditions, like dystonia (a disorder in which muscle contractions cause twisting and repetitive movements or abnormal postures) or major depression. The magnetic field induces an electric field inside the brain, which can activate neurons. Some companies, like BrainsWay,  are manufacturing such systems, that have already being used in clinical trials.

tDCS stands for Transcranial Direct Current Stimulation. This method requires electrods to be put on the head, which drive a small amount of current through the skull and influence the area of brain just beneath the electrods. tDCS only affects neurons that are already active. In a recent article published on Nature, tDCS is mentioned for its potential to enhance the minds of healthy people. Some students treated with tDCS showed improvements in working memory, word association and complex problem-solving.

A few months ago on the neuroscience journal Neuron, a paper was published that refers to the investigation of transcranial pulsed ultrasound on neuronal activity. The author claims that this technique has a better spatial resolution compared to the previously mentioned ones, and can be used to elicit motor responses when the ultrasound stimulation is applied to an intact motor cortex. Darpa, the military US research project agency, decided to fund this research aiming to create a helmet with this non-invase direct to-brain interface.

Maybe the future to transfer bit into the brain is yet unclear, but scientists are clearly looking for that.