Implementing Virtual Reality using OpenCV Libraries

This is a complete Research Oriented concept for the Blog and these techniques are used in Research based Firms and projects.

Normally, we use Mouse and Keyboard to handle Man and Machine Interaction. In few advance systems, specifically for high end gamming, X-Box and PlayStations use Technologies like Kinect and PlayStation Move for Machine Interaction. This Gesture or pattern based command concept can be used in Standard PC’s to perform  its functions. A glove with some color pattern and a 2 MP + Camera is the easiest way to achieve this for an average Computing System. With processing speed  at least  3GHz with min Dual Core CPU and 2GB RAM, this system can work in Real time. This paper is to present that the technology can be developed using Microsoft’s .Net Framework with the help of OpenCV Libraries.

OpenCV (Open Source Computer Vision) Libraries are libraries for Programming function aimed for Real time Image or Computer Vision Processing. This Libraries can be imported on Visual Studio for C++, C#.net Programming and are available on the net without any cost. Eclipse IDE can also be used with this libraries. OpenCV can perform operations for Real time Gesture detection, Pattern Recognition which can be achieved with Image Processing functions. The Image captured by the Camera can be processed to identify a pattern using methods like background removal, eroding and dilating the image and many other methods. The OpenCV Libraries can detect the pattern and then the system required functions can be done with C++, C# or Java Programming Languages.

Installing OpenCV by using pre-built libraries is the easiest way. There are many blog on this installation:

http://opencv-srf.blogspot.in/2013/05/installing-configuring-opencv-with-vs.html 

http://docs.opencv.org/index.html

The simplest way to control a Video or Media Player is by sending Keyboard keys virtually to perform dedicated function. Player like (KMPlayer, VLC, Winamp, Foobar, and many more) supports Keyboard’s Key Press association with media functions.  All functions like Play, Pause, Forward, Rewind, Volume up/down, Exit, Minimize, Full screen and many other can be associated with Key Press events using the Settings option in this Players. Now this tasks can be performed by sending the Key Press Events Virtually (without actually pressing Keys of keyboard) using Virtual-Key Codes provided by Microsoft on MSDN. All Key Press events have Unique Virtual-Key Codes. Specific Color patterns can be used for recognition. Example If Yellow color just beside Red color can be a pattern to perform actions like Play. The camera captures the image and processes it to recognize a pattern  (generally take a few milliseconds or less depending on  CPU,  but  this milliseconds  gap  doesn’t  cause  any Real  time playback delay). This pattern discovery can trigger a specified Virtual-Key Code to control an already  running media player. The camera speed of taking an image needs to be delayed in such a way that,  in 1 or 2  seconds the camera only captures one image so that the  user gets enough time to move his hand away.   The complex part of the system is detecting the pattern in different lighting conditions. This is because light may change the pattern color and camera will not be able to recognize it. This problem can be eliminated by setting specific pattern values for different environment.

The final system should work as (basic Algorithm):

1. Take 1 snap in 1 sec to process.
2. Process the image to discovery patterns if exists.
3. The first technique will be recognizing a hand like structure using the already provided True and False pattern samples.
4. IF-Recognized a hand structure then the system will try to recognize the color pattern. ELSE-Discard the sample and wait 1 sec for another Image (Start Step 1).
5. Trigger the code for Virtual-Key depending on  the type of color pattern.
6. Media Player that is  already running  gets the  Key Input Virtually to perform a specific task  related to that Key Press Event.
7. Start with Step 1 again.

The scope of this Controlling technique is quite wide. Applications like Photoshop, Auto Cad, and many others can be programmed to function on the same technique.  The advantage over a Mouse can be achieved by implementing 3D technique on the program to enable hand patterns as 3D inputs. The 3D controlling Environment for applications used in Modeling and  Designing the architecture will be of great productivity,  easy-to-use and quick. For this 3D scope there will be  requirement of more than 1 HD Cameras  along with much more powerful Computers.

I have successfully made this Project and will continue with explaining it and the actual code in future Blogs.

About the author:

Vidhit D Shah is a consultant in Systems Plus Pvt. Ltd. Within Systems Plus, he actively contributes to the areas of Technology and Information Security. He can be contacted at vidhit.shah@spluspl.com