With the progress of technology and the availability of smart materials we can rest assured of a future, by the end of this decade I would say, where most surfaces will double up as screen and interaction points.

The sketchy representation of a future cell phone patented by Apple

I had the opportunity of discussing with a start up founder just few days ago. He is focussing on new ways to interact with information based on advanced capabilities offered by Surface 2 (MS) and others. Today these technologies are still quite expensive but the day when they will become commodities is on sight.

The interesting thing, I think, is that the interaction will no longer be a property of the device, rather it will be a property of the person interacting! That is to say that if I am interacting with the surface of a desk in a public space, like  a kiosk in a mall, that surface will first recognise who I am and then will react accordingly. This can be achieved in several ways, by an identification service in the “cloud” or by an interaction with an identity tag embedded in my body (just to name to extreme cases).

The availability of plastic screens, like AMOLED, or ones based on Graphene (NED can be considered a first step in this direction) will make it possible to “wrap” any surface with a screen and touch sensitivity. This, I guess, is the technological bases for a patent presented by Apple and described by ZDNet of a cell phone having a wrap around screen, as shown in the drawing above.

The curvature of the surface, according to interpretation by ZDNet, can be used to provide a sense of 3D. The wrap around can also be used to identify (my speculation) the user by “sensing” some characteristics of her hand.

Not sure if and when we are going to see an iPhone like that but for sure the new smart materials invented every day by researches are freeing designer imagination.

3 LCD screens are overlaid, each presenting part of the image. The perceived effect is an image that changes as we change our point of sight

3D imaging today is quite different from our reality perception of the 3D world around us. Today’s screens can provide you (with or without special glasses) a sense of depth that gives you an illusion of viewing a 3D image. However, in the real world we get a 3D perception not just because we sense depth in an image but also because as we change perspective the image changes.

This is not the case in current 3D screens. Every single person watching a 3D movie at a theater will see exactly the same image, independently of where he is seated. That would not be the case in the real world!

To get a changing perspective you should use holographic technology but today, and for several years ahead, this technology is still very expensive and has so many shortcomings that cannot be used (it is almost impossible to provide good colors and moving image with present technology…).

Researchers at the Media Lab have developed a new technology that is able to deliver a closer to life 3D experience. They are using three LCD screens, a technology that is now widely available and reasonably cheap, to display an image that results from the composition of those displayed in the three screens, as shown in the figure on the left.

The system is amazingly complex, not in the hardware, although also the hardware is at the prototype stage since it custom built needing a refresh time of 360 frames per second on each LCD screen, where the most advanced screens you can get on the market today have a refresh rate of 240 frames per second and most to not exceed 120 frames per second, but in the software needed to create the right signal.

The image is continuously recalculated and the high refresh rate is used to send more “images” to each of the screens so that depending on the point of sight the viewer will see a different scene. So two person looking at the same screen will see the image from two different viewpoints.

Actually, a refresh rate of over 1,000 frames per second would be required to provide these different point of sight but on the average quite a bit of a scene does not change significantly (it is not perceived to change) as you change your point of sight. This is exploited through number crunching to reduce the required refresh rate to 360 frames per second.

How much bandwidth is required for this type of display? Difficult to say. It really depends on where the rendering is being done. With processing power still increasing at the Moore’s law pace, it can be expected that the rendering will take place in the television itself, thus limiting the increase of bandwidth to some 3/4 times what would be needed today (a 6 mbps for today HD TV would be pushed to a 20 Mbps for this kind of display, whilst the 25 Mbps needed to feed a 4k screen would be pushed up to 100 Mbps to feed this technology).

If you were to perform the rendering at a service point and then to drive the three LCDs at 360 frames per second, well , that would really put a strain on the transport infrastructure since we would be talking of a bandwidth in the range of 100 Mbps for a “normal” HD TV and over 500 Mbps for a 4k resolution!

Andyvision wandering around in a shop

Robots are now a constant presence in modern society but we usually don’t see them since they operate in backstage. Factories are more and more “inhabited” by robots manufacturing cars, computers and shoes to name but a few. We have robots in our homes, but they are disguised and we don’t perceive them, like some vacuum cleaners or cooking devices. Hence, when we recognize a robot wandering around we are still surprised. That is not the case in South Korea were people are getting used to see robots in the subway stations, post offices, hospitals and also in department stores.

At CMU researchers have developed a robot that can move around in a shop to look at shelves, track merchandize and send alerts if needed, like the shortage of a certain ware.

To do this researchers have embedded many technologies in the robot, including 3D vision, image recognition automatic learning. They are not new but their use in a coordinated way is new. Don’t underestimate the difficulties: can you tell by looking at a shelf what sizes of a certain shirts are there? The robots try to look at bar codes (but quite often they are hidden inside the shirt…) measure the length of the shirt and figures out how it has been folded to derive the size. It also checks the inventory data base to get hints on what it sees and to compare its calculation with what should be available in the store.
All these technologies used to be pretty expensive but their price went down considerably (and it is still going down) and it may reach a point when it might be easier to use vision to understand an ambient than place RFID tags on any object.

The robot, called Andyvision, wanders the shop and creates a “mental map” of the shelves and the various products in its first “rounds”. Once it has understood the lay of the land it starts its rounds to check if there is any shortage of a product on a shelf, if a product has been misplaced and so on. In practice it provides a real time inventory of the shop. So far it has been “at work” in the CMU store since mid May and by September the researchers hope to be able to draw some conclusions on its effectiveness and to evaluate the potential money saving made possible by a real time environment.

From this to helping customers finding a product it is a small step. And that can led to increase a shop revenues.

Interestingly, studies are showing that the trend for mixing robots and humans is gaining steam and some analysts consider this to be the reason why in the USA global productivity has increased since 2007 in spite of a decrease of total jobs. For the first time the shift from one kind of society (agriculture to industrial, industrial to services) has not resulted to a shift of jobs that overall rebalances the job destruction with job creation.

Google Earth keeps amazing me, having the possibility of seeing most parts of the world as flying over it with a small plane at low altitude is really impressive. And with the new features letting you to see some parts of the world in 3D it is even better.

Now C3 Technology, a new Swedish company that was spun out of SAAB, is delivering an incredibly detailed view of the World (part of it) in stunning 3D. You’ve got to see it to believe it.

I cannot wait to see this kind of details becoming available for every part of the world. It really makes you feel like you are watching the real thing.

The result is achieved through photography and a lot of rendering software to reconstruct 3D images that can be manipulated even on your iPad.

We are really fading the boundaries between real and virtual world and we are going to benefit a lot from these fading boundaries since it will be possible to create a lot of services in the virtual world where the cost of manipulating and transmitting bits is so low and then come back to the real world using interfaces and communications. Good business for Telcos if they find a way to be in the control seat to deliver the experience.

Sure you had. And, besides, we have come to take for granted that we need to choose what needs to be in focus and what we accept to be out of focus.

This is part of the rules of the game when we use a camera, both film and digital.

But now a start up, Lytro, has found a way to let us take the whole image as it is in front of us and then decide afterward what we want to see in focus.

You can even change the point of view and create, after the shot, 3D images.

It should be available later this year at a price that according to Lytro, will be more than 1$ and less than 10,000$. That is quite a range…so let’s wait before calling it our next gadget!

I’ve just spent a few days at InfoCommShow 2011, the biggest event for audio/visual systems, solutions and devices in the US and probably worldwide, which took place in Orlando. It attracted more than 900 companies to exhibit and more than 30.000 visitors. The event was also running several educational sessions and demonstrations.

InfoComm is a trade where to find most A/V solutions for professionals and enterprises. To ease navigating the exhibition, the floor was divided in a few thematic areas (which is quite consistent with the Orlando spirit of hosting several theme parks): unified collaborative conferencing, 3D, digital signage, audio, lighting and staging. In fact around the show you could find displays and projectors, telepresence systems, multi-touch technologies, A/V cables, racks, cameras, microphones, mixers, speakers and more. Most of the best of breed companies of the sector were present, including Cisco, HP, NEC, Samsung, Bose, Barco, LG, Sharp…

The A/V market is flourishing, with a positive CAGR of about 9%. Yet the market is changing: convergence with Information Technology is stronger and stronger, most content is today transported on IP and cables have been joined by wireless as well.

A quick glance to show floor would have given the impression of being in Time Square, New York, more than at the Orlando Orange County Convention Center, due to the massive presence of displays all around: LED displays of different sizes and form-factors, bezel multi-monitor displays, and multi-projector displays. The latter were mainly powered by an edge-blending software solution patented by Scalable Display Technologies, an MIT spin-off, capable of running up to six projectors on one GPU. Scalable enables to create highly immersive displays with a relatively inexpensive solution. Some of the LED displays were used to visualize 3D content, but requiring eyewear, which severely limits their scope. Autostereoscopic displays were not present at InfoComm.

HP Halo new telepresence prototype showing touchscreen control and blended communication

Several telepresence systems were demonstrated with real-time conferencing sessions. I personally attended a demo of the new Halo prototype by HP, which integrates a touch-screen control instead of the classic IP phone and can integrate other incoming videocalls, and demos of other telepresence systems by Polycom, Cisco, LifeSize. The telepresence main cues are currently HD video quality and a set-up to provide virtual eye gaze (to provide realism), stereo sound (to identify who is talking). From the technological point of view not much has been done in order to increase immersiveness: when I asked if 3-D displays were going to be adopted, the experts told me that they envision them to support presentation delivery but not to augment the visual impact of the virtual meeting participants. A few companies, like TelepresenceTech, were proposing to add a fake 3-D effect leveraging on the Pepper’s Ghost effect: the solution consists in putting a semireflective glass at 45° in front of the LCD display to reflect a selected background and creating the sensation that the virtual person was standing beyond this background.

TelepresenceTech system to deliver a virtual 3-D effect, based on Pepper's ghost technique

VGO Communications, an American start-up recently getting $4,5 million funding in Virtual Capital, created a “telepresence” robot that moves on wheels and integrates an LCD display on its head together with speakers and micro. The robot is telecontrolled by the person that wants to remotely communicate. What the VGO robots do is in fact to carry our “physical” avatar around. This concept is particularly interesting for disabled people who want to extend their presence in the place where they live, without installing videocams, micros and speakers in each room.

The VGO Communications telepresence-robot talking with a human...

You may have noticed that currently stereoscopic displays have only one parallax direction, the horizontal one.This means that you can look explore an object tridimensionally only moving to his right or left, but not up or down, which would require vertical parallax capability of the display. This applies to the 3D Cinema or TV set solutions that are based on eyewear, or to autostereoscopic displays like those used on the Nintendo 3DS.

Even more complex displays, like those developed and commercialized by Holografika, a hungarian company that patented a ray projection holographic display, are offering a mono-parallactic stereoscopic view: using several optical light projecting modules that are hitting each singular voxel (volumetric pixel) onto the “holographic” screen, they are able to recreate a wider angle view then other techniques can today. Holografika’s technique consists in projecting several different angular views of an object on the screen that can be perceived by the viewer just slightly moving on his left or right, as we would do in a real life object exploration.

Now if you try to watch the upper or lower sides of this virtual object, you will get always the same view of it, as in fact the system is not able to perform vertical parallax, giving back a sensation of fakeness.

A new kind of display has been invented by Holovision that can create high-resolution, three-dimensional moving images that can be viewed with full parallax by people in different locations without special eyewear.

Holovision volumetric display based on MEMS.

This display leverages on the future features brought by MEMS (Micro Electrical Mechanical Systems) that in the next years will be used to create a matrix of moving display elements to guide light at the level of individual display elements (such as voxels). MEMS will be able to spin light at individual voxel level in all directions, thus dynamically recreating a volumetric image.

Array of spinning microlenses at voxel (volumetric pixel) level

Such a revolutionary volumetric display system will have also consequences in terms of all the data that has to be elaborated and transmitted, requiring graphics processing units and transmission channel capabilities to perform several times more than today.

I had the opportunity of visiting Infocomm Asia in Hong Kong, http://www.infocomm-asia.com/ , and the main theme was 3D screens, movies and television, advertisement and games.

3D screens everywhere at Infocomm Asia

I have to confess that those many screens had a magnetic effect on me and I spent several minutes looking in awe at the images, through polarized glasses and also without them. Then, I had to quit.

I started to feel uneasy and then a full blown headache hit me. It was not the first time. When I see 3D screens after a while I get headache. It may be a matter of just few minutes or ten. And the headache lingers for a while.

It is not just me, apparently several people suffer the same. Viewing 3D on a home screen is quite different than viewing it in a cinema. There you are sitting in a darkened environment and all you see is the screen. At home (or at a fair) the ambient is illuminated and therefore you see at the same time the screen and its surrounding. Your eyes scan the whole environment and when they move from the screen to what is at the side have to readjust their focus and this is what affect adversely several people, myself included.

Therefore, yes the images are really fantastic but they are not for me!

At the ICT 2010, organised by the European Commission in Brussels to discuss state of the art and perspectives of Information Communications Technologies there was an extensive display of prototypes. A few of them stimulated my thinking.

As an example, we are talking a lot about 3D, and it has reached the consumer market this year. But it is still “glass” based, that is you need to wear glasses to perceive the 3D effect.

At the ICT I saw some prototypes that get rid of the glasses. True, the result was not comparable, in terms of definition and colours, but at least it is an indication that the future might bring something that would really feels 3D.

Real 3D on a screen designed for a cell phone

One prototype was for a handheld device, and I was told that it is not far from the shelves. Interestingly, the expected cost of this screen is just one dollar more than a normal screen. The trick used is a plastic layer to let our eyes look at two different images, hence the 3D impression.

Of course there is a lot of software to make it work and a dedicated processor is being attached to the screen, as shown in the picture.

The other was a truly 3D image, since I could look around the object being displayed s if it was there. The object presented was an artefact,

I post here two photos taken from a different angle that really present two different views of the artefact, as if it were a real object in front of my eyes.

The image was not good, and this is reflected in the photo. I mention this to avoid you may think the poor quality is due to the photo. The technology still has a long way to go.

The way the image is being displayed is by creating a controlled vibration of the screen and projecting sequentially images created by cameras placed at different angle from the object. Each image becomes visible only when observed from a certain angle and the frequency is such that all of the images are visible at the same time, each from its own angle of view.

I presume that displaying a real object would entail a huge complexity in the lighting, since our eyes are extremely sensitive to tiny difference in lighting and our brain works on these differences to derive a model of what is been seen.

This type of screen is probably at least ten years away from consumer shelves. Surely, if it will ever come to fruition that would demand a tremendous amount of bandwidth since ten or more flows of data are concurrently displayed at once.

I had the opportunity of participating at a workshop in Shanghai at the 2010 Expo on Culture and Arts. I am no artist and my cultural level is …well not good; I was there to discuss how to use technologies (the ones we have today and the ones that are moving their first steps in the labs).
It was interesting to see how the future is already shaping today’s arts and culture. In the restoration process neutron accelerator are used to look inside a statue, special lasers can tell us what is under the paint of a portrait and may be help in understanding the thinking of the artist who changed his mind several times before coming to the final masterpiece.
I found interesting the 3D reconstruction of monuments and entire cities, like Rome and Alexandria, created by Altair, http://www.altair4.com . It made me wonder how it would have been like walking in those narrow streets hand in hand with Julius Cesar. He was not there, but the 3D rendering was so effective and so “credible” that it meade me feel I could have been there.
And then I made the next step: since that reconstruction felt so real, why don’t augment that “reality” to provide me with the feeling of being a citizen there at that time.
We already have Simulation Games, add to them environments feeling really … real, get your avatar to roam the place and experience the feeling. Engage yourself in a discussion with Cato, let Julius tell you about his latest war campaign in the Gauls.
I am pretty sure I would be able to learn so much more than by reading a textbook.
Yes, I feel my nephews will enjoy much better ways of learning.