I wanted to explore the large number of possible forms of enabling interaction between the virual and real world using only cameras and projected images. This seemed like a good venue to get feedback and generate more ideas in other ppl’s minds…
The general setup would be to have a camera facing the projection screen/wall and the computer projecting generated graphics into the screen/wall. Users can interact with the virtual projected image, and the the computer sees it through the camera, and modifies the behavior of the virtual object. All this is imminently doable in Processing with differing levels of difficulty (after the first step of calibrating what the camera is seeing with respect to the projected graphics).
1. The most intuitive interactivity probably consists of simple games…. The computer projects a virtual ping pong ball and walls, and the user (or users) play the virtual ball. The camera tracks the user’s hands (or a bright colored/reflective tag on their hands) to sense the direction of movement, speed, torque etc and uses those as input to the game engine physics.
2. A virtual world could be created which can only be sensed/seen in the projection… As an example, one can imagine the projection consisting of walls and a small creature that sits lonely within the confines pacing around. When it senses people interacting with it (hands petting the creature in the image, pushing it, chasing it around etc), it responds appropriately to engage the user and keep them hooked. Or there could be drama transpiring between two virtual creatures (predator/prey, courting pairs etc) that the external (real) user can affect.
There must be many different incarnations of this setup that can be designed to tug at particular human emotions, value judgments, interests etc such that the user’s interactions become a very personal and meaningful display of their humanness.
This can also easily be combined with the game setup idea (imagine a little castle building genie in the projected graphics who tries to diligently stack up his cube castle that you can punch and kick at to make it crumble, causing the poor guy to constantly keep trying to build its little fortress that you keep destroying)
3. The real world could be mixed with virtual components… For instance, the graphics projected at one end of the wall might be of the same room itself as it appears at the moment, except for the addition of a virtual entity that does not actually exist in the environment. If we were doing this in the class, maybe there is a virtual gremlin that is moving from chair to chair, tugging at ppl or stealing things or creating other kinds of havoc.
The first part of this would be easy.. superposing the virtual entity in the real world setting. The second half of truly convincing interactivity (for instance having the gremlin move around a book from the real world by identifying the book object in all video frames and displacing them in each frame as required by the virtual world) would be really hard. If this could be done, we could get very close to the uncanny valley where the real and the virtual start melding in our minds.
Comments are closed.