March 22, 2005

Progress at NIST Workshop on Evaluating Performance of 3D Laser Scanners

Last week the National Institute of Standards and Technology, Gaithersburg, MD, hosted the second annual LADAR Performance Evaluation Workshop. The meeting brought together many of the world’s leading suppliers of terrestrial 3D laser scanning equipment – iQvolution, Leica Geosystems HDS, Optech, Riegl USA, Trimble, Visi Image and Z+F – with NIST metrology experts, customers ranging from Boeing to the FBI to MAGLEV, and a handful of service providers including Quantapoint and Bohannan Huston, to begin creating a glossary of terminology and to develop protocols for evaluating the performance of 3D laser scanning equipment. 

Hats off to NIST for hosting this workshop. In our opinion, it delivered real value not only to the laser scanning industry but also to engineering firms, construction firms and, most importantly, asset owners.

Glossary of terms

For the newcomer to 3D laser scanning, there’s a new vocabulary to be learned: What exactly is meant by the terms FOV, flash, LIDAR, gridded data, monument, point cloud, registration, survey control, target and on and on? As we learned at last week’s meeting, even the experts don’t all agree on what is meant by accuracy, uncertainty, precision, resolution, etc. We believe that confusion about terminology retards industry growth and contributes to errors and misunderstanding.

The good news is that there are some remedies being developed to address this Babel-like state we find ourselves in. First, attendees at last week’s workshop will be developing and circulating a glossary. If you have an interest in contributing, please contact Gerry Cheok at NIST. Second, some of the hard work has been done already in the related industries of laser tracking and short-range metrology, and much of this is relevant for terrestrial laser scanning. Workshop attendees seemed to welcome the idea of adopting ISO terminology found in a 1993 publication Guide to the Expression of Uncertainty in Measurement, known affectionately as GUM, and NIST Technical Note 1297 Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results. We will report on progress in this area as it unfolds.

Standardized protocols for evaluating performance of 3D laser scanning instruments

It was widely agreed that the performance specifications published in trade magazines allow users to make very limited comparisons of 3D laser scanners. The reason, of course, is not necessarily perfidy or claim exaggeration on the part of vendors, but rather the lack of standardization and agreement on what is meant by such seemingly simple notions as measurement rates, range accuracy, angular resolution, and so on.

Having sat through two days of meetings with key vendors of this equipment, it’s evident to us that developing these protocols is non-trivial. Devising rigorous definitions, protocols, and artifacts for exercising these protocols is difficult. The 3D laser scanning industry can draw on the hard-won experience of some other industries that have already been down this path. We also sensed a willingness among the vendors to cooperate to achieve this. Brent Gelhar, Director, Laser Imaging Division, Optech Incorporated, saw the meeting as a “great step for industry and NIST to understand industry needs.” Ted Knaak, President, Riegl USA, said, “It is a good idea for the industry to get together to look at new ways to evaluate system performance.” Most of the vendors we spoke with expressed the need to have this be an industry-led effort rather than an effort dominated by one or two suppliers. Jim Flint, Vice President of service provider Bohannan Huston, Inc., says, “What was valuable was the consensus to develop a rigorous, consistent way for anyone to compare scanner performance under the guidance of NIST.” “The opportunity to meet with NIST metrology experts who’ve had experience with developing performance evaluation standards in adjacent markets such as laser tracking and CMM is a high value,” said BitWyse Solutions CEO Mark Klusza, who adds, “The subgroup discussions on topics ranging from calibration standards to the need for open data formats were productive.”

In the coming year NIST plans to conduct a survey of protocols used by manufacturers on how they assess the measurement performance of their instruments, develop a draft of test protocols, and send these out for comment. geovisit();geovisit();

Standard artifacts for performance evaluation

Working groups at the NIST meeting made suggestions about what kinds of artifacts would be most useful to evaluate the performance of laser scanning instruments. One big challenge, of course, is that the range of some instruments extends to a kilometer or more – providing controlled measurement conditions for such ranges may require some clever engineering, and some large facilities.

What about academic research in this domain? Some has been carried out, attendees learned. Dr. Wolfgang Böhler, Institute for Spatial Information and Surveying Technology, Mainz, Germany, presented some of his latest findings. However, he pointed out that comparison of scanner performance has many subtleties and that experiments based on a few artifacts do not yield results reliable for ranking scanner performance.

In the coming year NIST plans to conduct a survey of protocols used by manufacturers on how they assess the measurement performance of their instruments, develop a draft of test protocols, and send these out for comment.

Open data formats

The need for open data formats, though not part of the meeting’s original agenda, also came up. Effective filtering of scan data needs to be informed by the order in which the data sets are collected as well as scanner position in addition to the x,y,z and intensity values published in today’s standard ASCII file formats. Several scanner vendors, notably Riegl, Optech and Visi Image, said they would publish open data formats to help facilitate this. NIST executives pointed to a recent report on the $15.8 billion cost of the lack of interoperability to U.S. manufacturing and construction, and appeared to enthusiastically embrace the laser scanning industry’s willingness to be more open with respect to data formats. It will be interesting to see if other industry participants follow suit, either by publishing details of their data formats or by working to produce a protocol for a neutral data format.

Next steps

A follow-up workshop is planned for February 2006. Meanwhile we’ll keep you apprised of progress in this area. Thanks very much to NIST for performing this valuable service. We think it will be a boost to our young industry. 

Want more stories like this? Subscribe today!



Read Next

Related Articles

Comments

Join the Discussion