Monocular vision system

Bob Mottram 38b2840830 list 6 years ago
.metadata 91f0cf02ac 11 years ago
.settings 91f0cf02ac 11 years ago
Debug 5ad6547a5a version 1.043 11 years ago
cppunitlite 9dde49a272 11 years ago
deb 042f748330 11 years ago
docs d4f1b4be1e 11 years ago
images d3a4bc1054 Images 6 years ago
ros 5ad6547a5a version 1.043 11 years ago
segment 03962bd36e segmentation 10 years ago
unittests 2dd2546285 11 years ago
.cproject 5ad6547a5a version 1.043 11 years ago
.project 7318bd72a9 11 years ago
LICENSE ab2b89886d Add license 6 years ago
Makefile 03962bd36e segmentation 10 years ago
README.md 38b2840830 list 6 years ago
anyoption.cpp 7318bd72a9 11 years ago
anyoption.h 7318bd72a9 11 years ago
drawing.cpp 7318bd72a9 11 years ago
drawing.h 7318bd72a9 11 years ago
fast.cpp 7318bd72a9 11 years ago
fast.h 7318bd72a9 11 years ago
harris.cpp 7318bd72a9 11 years ago
harris.h 7318bd72a9 11 years ago
libcam.cpp 1abc776490 11 years ago
libcam.h 91f0cf02ac 11 years ago
linefit.cpp 91f0cf02ac 11 years ago
linefit.h 91f0cf02ac 11 years ago
main.cpp 03962bd36e segmentation 10 years ago
makepackage.sh 01f139f507 11 years ago
motion.cpp 8633e19251 struts 10 years ago
motion.h 1abc776490 11 years ago
omni.cpp 8633e19251 struts 10 years ago
omni.h 8633e19251 struts 10 years ago
polynomial.cpp 91f0cf02ac 11 years ago
polynomial.h 91f0cf02ac 11 years ago

README.md

Omniclops

This is a utility for dealing with images obtained from omnidirectional vision systems. It is intended for use on mobile robots equipped with omnidirectional vision sensors.

How to make an omnidirectional vision system

Introduction

Here's how you can make an inexpensive omnidirectional vision system using a webcam and easily available materials.

Parts

Here's what you'll need:

  • A webcam. This should be USB Video Class (UVC) compliant to ensure that it can be used on Linux. It should also have a narrow field of view (most, but not all, webcams do)
  • A length of 5mm diameter aluminium tubing
  • A silver (reflective) Christmas tree baubel, approximately 60mm diameter (although other sizes could be used)
  • Some cardboard
  • Tie wraps
  • Pliers
  • A bradawl or screwdriver
  • Scissors
  • A drill

Instructions

  1. Pull the top off of the Christmas baubel, then drill a 5mm diameter hole at the opposite end.

  2. Push the aluminium tube through the holes on either side of the baubel, then crimp and bend the end of the tube so that it prevents the baubel from slipping off.

  3. Using pliers, bend the tubing into a frame like this:

The height of the frame should be about 160mm.

  1. Remove the outer casing from the webcam. You may need a precision screwdriver to do this, and how you do it really depends upon the specific model.

  2. Cut a piece of cardboard larger than the triangle shape of the frame.

  3. Use the bradawl or screwdriver to make holes in the cardboard, then fix the webcam onto the cardboard using tie wraps threaded through the holes.

  4. Lay the cardboard over the triangle of the frame and make holes so that you can tie wrap it down. Make sure that the camera is pointing at the centre of the baubel. A good way to do this is to run the camera and adjust the position as needed.

  5. Trim off any excess cardboard using scissors.

The final result should look something like this:

The under side looks like this:

It's also a good idea to tie wrap the camera cable onto the frame, so that it can't get pulled out as it's moved around.

For a more professional system you could use plywood or sheet aluminium rather than cardboard.

Installing the software

To install the software first you will need to ensure that OpenCV version 2 is installed. A package for this can be downloaded from the project home page or downloads tab. At the time of writing this is not the standard version of the OpenCV libraries available from within the package manager, and the main advantage is that the newer version allows images to be encoded for video streaming.

A Debian type Linux operating system is assumed here. To install from the terminal type:

sudo dpkg -i opencv-2.0.deb

Then the main utility can be installed as follows:

sudo dpkg -i omniclops-x.xx.deb

Any additional dependencies should be installed automatically.

Alternatively, once the OpenCV libraries have been installed you can also compile from source in the usual way with:

make clean
make

If you make changes to the code you can also easily create a new deb package by running:

sh makepackage.sh

Configuring for use with Eclipse

To use with the Eclipse C++ IDE:

Create a new C++ project

Import the source from the Import/File System option

Within the project properties select C++ Build/Settings Under GCC C++ Compiler/Directories/Include paths (-l) enter

/usr/include/opencv
/usr/include/gstreamer-0.10

Under GCC C++ Compiler/Miscellaneous enter

-c -fmessage-length=0 -lcam -lcv -lcxcore -lcvaux -lhighgui `pkg-config --cflags --libs gstreamer-0.10` -L/usr/lib -lcv -lcxcore -lcvaux -lhighgui `pkg-config --cflags --libs glib-2.0` `pkg-config --cflags --libs gstreamer-plugins-base-0.10` -lgstapp-0.10

Under GCC C++ Linker/Libraries/Libraries (-l) enter

cv
cxcore
gstapp-0.10
highgui

Under GCC C++ Linker/Libraries/Library search path (-L) enter

/usr/lib
/usr/lib/gstreamer-0.10

Under GCC C++ Linker/Miscellaneous/Linker flags enter

/usr/lib/libcxcore.so /usr/lib/libcvaux.so /usr/lib/libcv.so /usr/lib/libhighgui.so

After applying those settings you should now be able to compile the project within Eclipse.

Running the software

The most basic way of running the software is as follows:

omniclops --dev /dev/video0

where video0 is the camera device to be used. To find out what video devices are available just type:

ls /dev/video*

One of the first things which you'll need to do is specify the inner and outer radii for the mirror. These are expressed as percentages.

In this example the outer radius is set to 90% and the inner to 15%. There are no limits on these figures, so potentially the outer radius could be set to greater than 100% if the camera is close to the mirror.

omniclops --dev /dev/video0 --radius 90 --inner 15

The purpose of the inner radius is to remove the camera itself from the image, since this is typically not of interest. The inner radius can be set to zero if you wish to include the camera.

To exit the program press the Escape key.

Unwarping the image

Unwarping is a particular way in which the omnidirectional image can be visualised, by projecting the raw image to a virtual camera located at the centre of the spherical mirror.

Example usage:

omniclops --dev /dev/video0 --unwarp

Projecting the image onto the ground plane

A useful feature is to be able to project the image from a spherical mirror onto the ground plane. This requires that the system know a few things about its geometry.

Example usage:

omniclops --dev /dev/video0 --diam 60 --dist 50 --focal 3.6 --elevation 400 --ground

where diam is the diameter of the spherical mirror, dist is the distance from the camera lens to the closest point of the mirror, elevation is the typical height of the camera above the ground plane and focal is the camera focal length. All distances are in millimetres.

Planar projection helps to remove the high degree of distortion around the periphery, and this results in the blocky appearance of the below image. Such images, although of low quality, can be useful for visual odometry purposes, since straight lines in the real world appear (mostly) straight within the image.

If you wish to create an image showing the paths of rays of light for your specified geometry, use something like the following:

omniclops --dev /dev/video0 --diam 60 --dist 50 --focal 3.6 --elevation 400 --raypaths rays.jpg

This will save an image called raypaths.jpg. At present only spherical mirrors are supported, although the system could be adapted for other types of mirror geometry.

Detecting lines

Detecting lines within the image can be useful for localisation or visual odometry.

Example usage:

omniclops --dev /dev/video0 --lines

Optionally lines can be saved to file for subsequent analysis.

omniclops --dev /dev/video0 --saveradial lines.dat

The file contains 5 integer values per entry:

1. Orientation of the line in degrees
2. start x coordinate in pixels
3. start y coordinate in pixels
4. end x coordinate in pixels
5. end y coordinate in pixels

Calculating optical flow

Optical flow can be shown by adding the --flow option to the command line.

For example, to show the optical flow in the raw image

omniclops --dev /dev/video0 --flow

or to show optical flow within the unwarped image

omniclops --dev /dev/video0 --unwarp --flow

Running the software in 'headless' mode

When running the software on embedded systems, or if you wish to save a single image and then exit, it can be useful to run without displaying the image on the screen.

Headless mode can be enabled as follows:

omniclops --dev /dev/video0 --save test.jpg --headless

Streaming video over a network

It's possible to stream video over a network using gstreamer. This can be useful for remotely operated vehicles or surveillance systems.

On the server:

omniclops --dev /dev/video0 --stream --headless

then on the client:

gst-launch tcpclientsrc host=[ip] port=5000 ! multipartdemux ! jpegdec ! autovideosink

To do a local test you can use localhost as the IP address.

Saving images, edge features, lines and ray paths

Saving images

One thing which you might typically wish to do is log images to file periodically. To have the program save an image and then exit use the following command:

omniclops --dev /dev/video0 --save test.jpg --headless

Saving edge features

Alternatively you might wish to only save edge features to file, as follows:

omniclops --dev /dev/video0 --saveedges edges.dat --headless

This saves a binary file called edges.dat. This consists of pairs of unsigned short integers (16 bits x 2) giving the x and y image coordinates of each edge feature. To try to increase the accuracy the edge values are sub-pixel interpolated, so to obtain a floating point value for the x and y coordinates divide the values by 32.

Saving 3D rays

You might also wish to save the 3D coordinates for rays of light corresponding to observed edge features. This feature can be useful for applications requiring bundle adjustment or synthesis of the data with that obtained by other sensors.

omniclops --dev /dev/video0 --diam 60 --dist 50 --focal 3.6 --elevation 400 --rays rays.dat --headless

The resulting binary file contains 6 signed short integers per ray, specifying the beginning and end of the ray. The first three values are a point on the mirror surface and the second three values are a point on the ground plane. All values are in millimetres.

Some example videos

Detection of edge features http://www.youtube.com/watch?v=2LEMFjGX1Bk

Planar projection http://www.youtube.com/watch?v=cjd6VbszaA4

Planar projection of edge features http://www.youtube.com/watch?v=AVZ5Osnde-U

Detecting radial lines http://www.youtube.com/watch?v=qH4ExS3XlH4

Image unwarping http://www.youtube.com/watch?v=Q_EmnGQpb2M

Optical flow http://www.youtube.com/watch?v=gWnDruAYCNQ