Efficient assistances can be provided from a 3-D model of a remote environment in order to programm and supervise teleoperation and mobile robots piloting tasks. The scope of the "Pyramide" project dealt with on-line acquisition or update of such 3-D model using embedded sensors, and elaboration of reliable assistances. As environmental conditions are often hard (constrained viewpoints, low quality images, highly specular surfaces), an interactive approach was selected. Acquisition time, obtained accuracy, but also system ergonomy and mission requirements fulfilment, were the main performance criteria.
A new concept of interactive modelling were images from embedded cameras are directly used has been proposed. According to displayed scene and task goals, the operator selects appropriate solid primitives and make them match relevant details in the images. Assistances based on visual features 3-D interpretation algorithms, image processing functions, or integration of knowledge on the remote environment are provided to facilitate this manual work.
A first software mock-up was first realized and validated at IRISA
Laboratory using computer generated images.
Its graphical aspect was rather simple, but it already featured the
most important functions to perform experimental campaigns on
realistic context.
These interactive modelling functions were industrialized within the
ACT-Lib Robotics-CAD library by Aleph Technology Company.
Based on this commercial ACT-Pyramid-Lib library, a professional
release of Pyramide was realized and used at CEA as the integration
platform for the modelling methods that were later designed within
the project.
They provide easier interactive 3-D control of objects using the mouse and analytical solutions for visual features 3-D interpretation.
They are managed through a graphical metaphor:
the manipulation frames.
The manipulation frame is a trihedron which reacts to mouse picking.
It is used to select the active task frame, the active manipulation
constraint, or modelled objects features, such as faces, edges,
vertices, or axes in order to graphically perform paste actions
at once.
The manipulation frame also displays the outputs of visual features
3-D interpretation functions, such as for instance the computed
orientation of three perpendicular edges, or the computed direction
of a set of parallel lines.
Two assistances have been integrated in order to release the operator's workload when performing low level tasks: a solid primitive matcher and a segment attractor. Efficient implementations were found to limit the influence of failures occurence and set up a user-friendly and efficient man-machine collaboration.
The solid primitive matcher relies on a non-linear minimization algorithm (Levenberg-Marquardt) to adjust the dimension and pose parameters of a solid primitive and bring it in close correspondence to the extracted contours, starting from a rough initial solution quickly obtained through interactive means.
The segment attractor relies on a Hough transform to let
user-drawn segments be attracted towards the best extracted contours
in the image neighbourhood.
Spurious contours with better quality than the expected ones may hinder
the assistance. When this occurs, the operator drags the segment away
from the disturbing contours, and the algorithm is run from the new
position.
Most often, the correct solution is thus obtained.
Some well structured environments can be processed using specialized modules based on a set of dedicated CAD primitives associated with optimized modelling procedures and internal constraints handling.
This principle has been validated through the implementation of a piping module. Straight pipe elements are directly inserted from the selection of their contours in a couple of images. They are then connected together using adaptable elbows, flanges or conic pipes. Normalized size values may be specified and used to constrain the piping primitives parameters.
Many evaluation campaigns have been set up all along the project evolution. They mostly focused on the man machine interface ergonomy in order to check the possible use by non specialists and on the achieved accuracy and speed performance.
In the scope of a collaboration with the industrial photogrammetry
ESIC Company, experiments held on images of clustered environments
with high specularity concluded on appreciable time spared compared
to on-the-shelf solutions and on a particular flexibility for
on-line applications.
Piloting assistances provided from Pyramide were evaluated using
different mechanical platforms developed by the CEA or its project
partners:
Centaure double-caterpillar intervention robot,
DATO mobile robot (Eureka/AMR2 project),
Ecureuil ''quadrumane'' inspection robot,
PML long arm carrier,
TLV 2-modules train-like vehicle, and its 4-modules counterpart
developed in the scope of European Teleman project (see pictures).
This interactive 3-D modelling principle achievable accuracy was
evaluated using a full-size mock-up of a 2m x 2m x 4m large
industrial environment mostly composed of tanks, pipes and flanges.
A standard CCD camera with a 8.5mm lens was fixed at the end effector
of a precise industrial arm (Staübli RX90).
The measured modelling accuracy was between 0,5% and 1% of the object
distance to the camera.
Two model-based paradigms for telerobotics application have been designed and experimented: goal designation control and task-oriented modelling.
The goal designation control relies on short cycles of
modelling, programming and execution.
A sufficient model is built from video images in order to programm
robotics tasks and immediately execute them.
This principle was first experimented on a "quadrumane" inspection
robot called "Ecureuil" (squirrel).
The mission consisted in reaching then climbing a ladder.
In most applications, the 3-D model acquisition and exploitation
stages are largely uncoupled.
The goals which are fixed for the modelling work are then
over-dimensioned, and aim at acquiring the most complete, detailed,
and accurate possible model at the expense of a huge realization time.
The task-oriented modelling paradigm suggests to affect some
global requirements to each modelled object in the environment,
according to its function in the robotics system programming and
supervision work.
It was experimented in the scope of a telerobotics mission experiment.
The mission was a visual inspection of a valve interior using the CEA
hydraulic arm "Maestro".
The valve laid within a cluttered environment.
Three different requirement levels were fixed for the modelling:
"navigation beacons" were modelled with low accuracy but featured
some relevant details for recognition tasks, "obstacles" lying on
the robot path were modelled by surrounding volumes with compatible
accuracy with the anticollision module capabilities (up to 10cm),
and "parts" were modelled more accurately to programm some robotics
tasks.
Here the valve was modelled with fine details and accuracy in order
to provide reliable assistances to the semi-automatic control of the
bolts unfastening.
In particular, the valve orientation had to be accurate enough to
constrain the three rotational degrees-of-freedom of the wrench tool.
Pyramide has been selected by UDIN (Nuclear Installation Dismantling
Department of CEA) to build a 3-D model of a radioactive room to be
cleaned and dismantled.
The only access to the room is a 7cm diameter hole in the 70cm thick
concrete ceiling.
Because the first cleaning operation was the pumping of a particular
tank, some intersest area was specified above this tank to ensure a
safe access with robotics devices from the ceiling.
An articulated mast with 3 degrees of freedom (Z position, site and elevation angles) was realized in order to insert a CCD camera in the room. Many difficulties occured all along the modelling mission:
| - available data proved unaccurate (a 3-D model obtained from a video using some standard (and not adapted) modelling software), or even unreliable (obsolete construction drawings), | |
| - the viewpoint was constrained along a vertical line (no possible navigation around the objects to be modelled), | |
| - the light ring that was fixed around the camera highlighted different details according to the viewpoint (few homologous features could be used), | |
| - distant objects images had low quality, mostly because of the cold (and low-powered) light source that was allowed to be entered in the room, | |
| - acquisition campaign were time restricted because of a short camera lifetime in that particular environment. |
Nevertheless Pyramide high flexibility ensured the acquisition of a 3-D model of the room that was accurate enough to prepare the cleaning operations.
Pyramide was designed for robotics missions within structured industrial environments. Its potential applicability to architectural applications was experimented using the CIPA (International Committee for Architectural Photogrammetry) reference data set. The set includes several images of Zurich City Hall building that were taken with two calibrated numerical cameras, and some control points on the building.
Despite of a weakly adapted set of solid primitives for that kind of environment, a detailed and highly hierarchized model of the building was acquired after a 16 hours work using only four views: a global view used to obtain the building proportions, and three closer views used to model details on the facades. This experimental work confirmed the potentialities of Pyramide's interactive modelling principle to tackle new situations with acceptable performances.
Started in 1986, the Pyramide project progressed under different forms, simple actions within other projects, or full R&D project, up to its conclusion in 2001. All along these 15 years, many fellows brought their contributions, engineers from CEA or affiliated companies as well as PhD or Master degree students.
The project scored several software realizations, one industrialization, and many publications. It was used within a french research programm (Eureka AMR), two european projects (Teleman LACWAP and Teleman MESSINA) and six industrial collaborations with partner companies from the nuclear field.
Most of all, it acted as a pioneer for the research in 3-D modelling from images, and is still a reference for further research in that domain.
Details on Pyramide can be found in my HDR report.