The goal of the vision part of the Cognitive Colonies project is to reconstruct a representation of the environment from images taken by the robots. The reconstructed models may be used for route planning purposes and for presenting the user with a threedimensional view of the environment. The process of recovering threedimensional information first involves matching features between images from the robots. From feature matching, accurate robot position can be recovered, as well as the 3D location of the features. The multirobot planning system is used to plan and coordinate robot paths in order to ensure field of view overlap and to maximize coverage of the environment.
Feature matching: Since the relative positions of the robots are not precisely known, the relative camera geometry between images is also unknown. As a result, in order to recover the geometry, features must be matched without the benefit of strong constraints. This is the classical correspondence problem. In our case, approximate robot position can be used to guide the search for correspondences and to develop a robust matcher.
Reconstruction: The reconstruction of the 3D geometry of the environment from feature matching is based on standard "structure from motion" algorithms. Challenges include ensuring reconstruction accuracy, outlier elimination, and view planning.
Map construction: The reconstructed model is a partial and sparse model of the environment. For presentation to the user, the partial reconstruction must be converted to a dense 3D map, which involves interpolation between features, or a floor plan. Possible representations for the uncertainty of the reconstruction through uncertainty maps are also being explored.
A typical environment as seen by one of the robots is shown below.
Salient features are extracted from the images taken at different postions by different robots. After matching, the camera geometry and the positions of the robots are recovered ("cooperative stereo"). The displays below show the extracted features and the recovered depth shown as a colorcoded templates.

Robust features: In order to deal with the problem of local similarity, more distinct features were introduced. Because the system is expected to operate in structured environments it would make sense to use a more geometric descriptive feature. Using planes as our feature, gives us this more descriptive feature and offers some additional constraints for the geometric reconstruction.
Geometric more descriptive: A typical reconstruction based on point correspondences also has the drawback that one can only reconstruct a "cloud of points". This type of reconstruction is not appealing because the data is hard to interpret. Our plane based matching method has the additional benefit that the ordering between point quadlets is know and we can therefore reconstruct a plane in 3D. Especially walls, doors, cupboards and teals make very distinct features that are easily recognized in the 3D reconstruction.
Technique: In both images, we generate planes by grouping lines together. Because the number of potential planes found is relatively small, we can exhaustively compute the geometric warp between the planes from both images and compare their similarity. We will then compute a global consistent one to one mapping between the planes in the first to the second image. The computed geometric warp (homography) of the absolute best two matches is used for the initial geometric recovery (epipolar geometry). A robust optimizer is initialized with this initial epipolar geometry and uses then all the remaining matches to recover the final epipolar geometry.
Reconstruction: With the now know correspondences and the know epipolar geometry, we can reconstruct the planes in 3D.
Plane correspondences:
Reconstructed 3D planes: