Source: UNIVERSITY OF NEBRASKA submitted to
A MACHINE VISION SYSTEM FOR PLANT SPECIES IDENTIFICATION, ENUMERATION, AND MAPPING FOR PRECISION CROP MANAGEMENT
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
TERMINATED
Funding Source
Reporting Frequency
Annual
Accession No.
0206826
Grant No.
(N/A)
Project No.
NEB-21-133
Proposal No.
(N/A)
Multistate No.
(N/A)
Program Code
(N/A)
Project Start Date
May 1, 2006
Project End Date
Apr 30, 2012
Grant Year
(N/A)
Project Director
Meyer, G. E.
Recipient Organization
UNIVERSITY OF NEBRASKA
(N/A)
LINCOLN,NE 68583
Performing Department
BIOLOGICAL SYSTEMS ENGINEERING
Non Technical Summary
The ability for a machine to identify species, assess stage of growth, and count individual crop plants or weeds is a complex process, but a potentially large number of applications exist. Weed patchiness has been well documented and herbicide savings of 40 to 70 percent have been projected, if only the patch areas are mapped and treated. The purpose of this project is to extend the use of a prototype machine vision system to applications of plant species identification and real-time monitoring of crop and stress conditions.
Animal Health Component
40%
Research Effort Categories
Basic
50%
Applied
40%
Developmental
10%
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
4041820202010%
4042300202020%
4047210202070%
Goals / Objectives
The overall goal of this project is to develop and test machine vision for identification of plants and monitoring of crop stress conditions. Specific project objectives are to: (1) to improve and the Camargo Neto - Meyer machine vision system for plant species discrimination, mapping, and enumeration of young sparse weed and crop populations in midwestern farm fields, (2) to compare its machine vision classification performance with visual results, and (3) to integrate this machine vision tool with plant growth models to simulate crop-weed competition for yield loss to improve decision support for increased soybean profitability.
Project Methods
A new machine vision with a plant species classification software system has been prototyped, initially tested, and published. Initial results were promising but there is a need to improve, test, and validate this technology. The focus of system development will include improved fitness functions for compound leaf extraction and leaf properties and new features. The project will utilize a large database of images of crops and weeds at two locations at five or six selected times during the growing season. University and commercial cooperator fields will be selected where corn and soybeans are annually rotated. Color images with various resolutions and marked with global positioning coordinates (GPS) will be acquired of soybeans and weeds from randomly selected subplot areas. Images of sufficient resolution should provide the necessary color, shape, and textural information needed for machine and visual classification, training, and validation. Weed scouting and manual photographic analysis will provide baseline results for comparing the performance of new computer classification algorithms. Crop yield (e.g. soybeans) will be determined within each plot and related to the competitive interaction of specific weeds and the crop, determined from image classifications obtained for that plot. Seed yield will be simulated using a crop growth model and spatially applied using geographical information systems (GIS). Traditional (statistical) and soft-computing (fuzzy inference) classiifcation models will be developed, trained, tested, and compared. Using spatial and strategic mapping strategies could best determine how much chemical to apply. Statistical multivariate, resampling, and cross validation techniques will be used to conduct hypothesis testing of comparisons. Improved automated machine vision systems applied across natural vegetation will be very important in remote crop sensing.

Progress 05/01/06 to 04/30/12

Outputs
OUTPUTS: A new plant species identification program has been developed and is currently being tested as potential "ap" for identifying selected plant species found in Nebraska and fields of the Great Plains. A new identification process takes place through an acquired digital leaf image. The identification algorithm performs the classical shape but concentrates on a very detailed leaf venation network feature analysis. Leaf species samples are supplied from plants grown in a growth chamber, greenhouse, and spring/summer/fall field specimens. Plants include various herbaceous species that are native and non-native (e.g., velvet leaf, pigweed, downy brome, phragmites, common reed, leafy spurge, cheat grass). Specimens are imaged using a high-resolution digital camera and special portable lighting system. Three-dimensional laser scanning of canopy plant architecture as an electronic taxonomy tool is also progressing. Testing included training and validation of a fuzzy-logic classification scheme with a high success rate. PARTICIPANTS: George E. Meyer, Professor, Biological Systems Engineering. Stephen L. Young, Assistant Professor, Department of Agronomy and Horticulture, University of Nebraska-Lincoln. Ashok Samal, Professor, Department of Computer Sciences and Engineering, University of Nebraska-Lincoln. Mr. Garret Coffman, MS graduate student, Biological Systems Engineering. Ms. Katy Conroy, UCARE undergraduate research assistant, Biological Systems Engineering. Researchers from the Farm Technology and Plant Research International. Agrosystems research group Wageningen University, The Netherlands have expressed interest in collaborating on new project proposals. TARGET AUDIENCES: A powerful method for real-time plant identification will be made available to growers, crop consultants, and ecologists. However, additional research and development must be accomplished. PROJECT MODIFICATIONS: A follow-on project proposal is in preparation.

Impacts
The focus is to deliver powerful results from scientific breakthroughs in plant identification and precision application technology. Cooperation across disciplines and with industry to form collaborative relationships will have impacts on regional, national, and international groups involved in developing sustainable, low-input cropping systems with smart technology. This work has been an inspirational support to the upcoming UNL-IANR "Big Ideas" seminars in 2013, "Advances in Plant Recognition and Identification Technology". These efforts represent a big step toward an electronic plant taxonomy guide.

Publications

  • Young, S. and G. Meyer, 2012. Precision and Automation Weed Control Technology. Crops and Soils Magazine, November-December, pp 1-9,


Progress 10/01/10 to 09/30/11

Outputs
OUTPUTS: Testing of digital imaging and machine vision techniques of plants and plant leaves continues, including the use of new USB high definition webcams and special portable back lighting techniques. Several large existing photographic plant data bases have been considered. However, the quality of images at the leaf level is not good enough for an automated plant taxonomy system. Our BSE Computer Specialist has demonstrated several machine vision techniques (LabVIEW IMAQ) for extracting leaf shape and textural venation features. The UNL system provides sequential and parallel processing for a potential real time field mapping system. A solution for close-in, digital photography of large scale field areas for precision weed assessment has been achieved with a high definition USB webcam. Color images of weeds in canopies, individual leaves, and inter rows are easy to acquire for FFT leaf or canopy crown shape analysis. However, leaf venation which is distinctive for varieties require better methods. Images were acquired with an agronomist collaborator for developing a special-back lit, weed species database for different stages of growth or plant age. PARTICIPANTS: NEB-21-133 has supported a number of UCARE (undergraduate research) and undergraduate honors thesis projects and has been recently joined by Garret Coffman, BSE Masters Student. Stephen Young, Extension Weed Ecologist (West Central Research and Extension) has been actively engaged in seeking grant funding and providing plant materials for the project. Mr. Coffman is an excellent programmer in LabVIEW and is assisting within development and testing of classification algorithms. TARGET AUDIENCES: Plant taxonomy continues to be a time consuming and challenging activity for the non-botanist, but a necessary skill for correctly assessing plant demographics and impacts by plants on natural resources. In addition, research grade tools with additional features for collecting and transmitting plant and environmental data will provide the most precise measurements for generating reliable information. The successful completion of this project will have an immediate positive impact in the research community to broaden the skills of many botanists and non-botanists needing quick and reliable plant identification. In addition, the general public, including land owners and managers, and many outdoor groups would have access to a tool for identifying new or established populations of invasive plant species and providing data for creating distribution maps at a regional or national scale. PROJECT MODIFICATIONS: A one-year extension is requested.

Impacts
An impact of this effort is that growers will be able to acquire photos of plant leaves with their webcams, smart or iphones phones and transmit them to a server for identification. Color machine vision and global positioning using smart phones have already been demonstrated where growers in California obtain soils data. As this system is developed at the University of Nebraska, it will become distributable as a stand-alone executable or updatable system as a web-based client, or operated as a real-time system for weed identification, using a LabVIEW Professional License. The objective is to put automated plant identification (taxonomy) into the hands of growers, ecologists, and clients. There have been 21 citations during 2011 (many international) noted for the journal articles generated as a result of this project.

Publications

  • Irmak, A., P. K. Ranade, D.B. Marx, K.G. Hubbard, G.E. Meyer, and D.L. Martin, 2010. Spatial Interpolation of Climate Variables in Nebraska. TRANSACTIONS of ASABE, 53(6):1759-1771.
  • Landgraf, D.L., 2011. Plant Species Identification Using LabVIEW Leaf Edge Detection and a MATLAB Fuzzy Logic Inference System. Unpublished Honors Thesis, University of Nebraska, Lincoln, NE.
  • Meyer, G.E., 2011. Machine Vision Identification of Plants, in D. Krezhova (Ed.), Recent Trends for Enhancing the Diversity and Quality of Soybean Products (ISBN: 978-953-307-533-4), InTech, Rijeka, Croatia, pp 401-420.
  • Henry, C.G., G.E. Meyer, D.D. Schulte, R.R. Stowell, and R.E. Sheffield, 2011. Mask Scentometer For Assessing Ambient Odors. TRANSACTIONS of ASABE, 54(2): 609-615.


Progress 10/01/09 to 09/30/10

Outputs
OUTPUTS: Testing of digital photo imaging of plants continues, including the use of new USB high definition webcams. An honors thesis project is in progress. Our BSE Computer Specialist has been also working on the LabVIEW implementation. This system provides sequential and parallel processing for a potential real time field mapping system. A solution for close-in, digital photography of large scale field areas for precision weed assessment has been achieved with a high definition USB webcam for acquiring color images of weeds in canopies and inter rows. This work compliments a continuing study of minimum photographic resolution for concurrent leaf shape and textural venation analysis. Images were aquired by an agronomist collaborator for a weed database for different stages of growth. PARTICIPANTS: George Meyer, Professor. Hanieh Kamelain, Undergraduate research assistant (UCARE), Andrew Landgraf, Undergraduate Honors Thesis, Garret Coffman, BSE Computer Specialist, Jerry Mulliken, Independent Crop consultant in Nebraska. and Stephen Young, Extension Weed Ecologist. TARGET AUDIENCES: Initial tests indicate that our software can discriminate plants from the images provided by Center for Invasive Species and Ecosystem Health. Thus, a powerful method for real-time plant identification could be made available to growers and ecologists. PROJECT MODIFICATIONS: Nothing significant to report during this reporting period.

Impacts
The impact of this effort is that growers will be able to snap photos of plants with their webcams, smart or iphones phones and transmit them to a server for identification. Color machine vision and global positioning using smart phones have already been demonstrated where growers in California obtain soils data. As this system is developed at the University of Nebraska, it will become distributable as a stand-alone executable or updatable system as a web-based client, or operated as a real-time system for weed identification, using a LabVIEW Professional License. The objective is to put automated plant identification into the hands of growers and clients.

Publications

  • Henry, C.G. G.E. Meyer, D.D. Schulte, R.R. Stowell, and R.E. Sheffield, 2010. Mask Scentometer for Assessing Ambient Odors. Paper no. 1010041. American Society of Agricultural and Biological Engineering, St. Joseph, MI.
  • Meyer, G.E. and Gary DeBerg 2010 Flow Measurement. D.R. Heldman (Editor) Encyclopedia of Agricultural, Food, and Biological Engineering, Marcel Dekker, inc.
  • Ranade, P. K, A. Irmak, D.B. Marx, D.L. Martin, K.G. Hubbard, and G.E. Meyer, 2010.Spatial Interpolation of Climate Variablesin Nebraska. Transactions of ASABE,in press.


Progress 10/01/08 to 09/30/09

Outputs
OUTPUTS: A National Instruments LabVIEW adaptation of the original MATLAB weed species discrimination software (Meyer and Camargo Neto, 2008) has been partially tested using a multiple-core personal computer. This system provides sequential and parallel processing for a potential real time field mapping system. A solution for close-in, digital photography of large scale field areas for precision weed assessment has been achieved with a high definition USB webcam for acquiring color images of weeds in canopies and inter rows. This work compliments a continuing study of minimum photographic resolution for concurrent leaf shape and textural venation analysis. The software was demonstrated to producers and crop consultants at the Nebraska Agriculural Technology Associtation (NEATA) annual 2009 meeting. Along with these efforts, images were aquired by an agronomist collaborator for a weed database for different stages of growth. PARTICIPANTS: George Meyer, Professor. Hanieh Kamelain, Undergraduate research assistant (UCARE). Jerry Mulliken, Independent Crop consultant in Nebraska. Mark Bernard, Extension Irrigated Weed Specialist. Viacheslaw Adamchuk, Extension precision agricultural engineer. TARGET AUDIENCES: Growers of grain crops in the Midwest that wish to reduce herbicide inputs through site specific weed management may benefit from this technology in the future. Other potential audiences are ecologists who manually observe and map plant species. PROJECT MODIFICATIONS: Nothing significant to report during this reporting period.

Impacts
The ability for a machine to identify species, assess stages of growth, and count individual crop and weed plants is a complex visual process, but extremely worthwhile in an effort to reduce the impact of weeds on crop production. A potentially large number of applications exist for mapping evasive species and ecological assessments. Weed patchiness has been well documented in the literature and herbicide savings of 40 to 70 percent have been projected, only if the patch areas are mapped and treated, as opposed to spraying the entire field. Applications may include weeds with either GMO or non-GMO crops.

Publications

  • Meyer G.E, 2009. An Improved Instrumentation and Controls Course for Agricultural and Biological Engineering. MidWest ASEE Annual Conference Proceedings, Lincoln, NE (on CD).
  • Govindarajan K.N., G.,P. Chapain, I. Poudel, G.W. Froning, G.R. Bashford, G.E. Meyer, and J. Subbiah. 2009. Predicting Eggshell Strength Characteristics using Ultrasound. Paper No. 09-096753. The American Society of Agricultural and Biological Engineering, St Joseph MI.
  • Govindarajan K.N., L.M. Grimes, J. Subbiah C.R. Calkins, A. Samal, and G.E. Meyer, 2009. Partial least squares analysis of near-infrared hyperspectral images for beef tenderness prediction. Sens. and Instrumen. for Food Quality. (Springer), in press.


Progress 10/01/07 to 09/30/08

Outputs
OUTPUTS: A validation study was completed through a 2007 University of Nebraska UCARE project, which included fuzzy logic classification of weed species, based on shape and textural venation features of computer extracted leaves. A LabVIEW adaptation of the origianl software is currently underway to make use of mult-core personal computers for parallel and faster processing. A solution for close-in, digital photography of large scale field areas for precision weed assessment has not yet been achieved. Satellite and aerial photography to date does not present sufficient resolution or detail for the species identification system. This is also dependent on an unfinished study of minimum photographic resolution for concurrent leaf shape and concurrent textural venation analysis. PARTICIPANTS: No change in participants. TARGET AUDIENCES: Growers of grain crops in the Midwest that wish to reduce herbicide inputs through site specific weed management may benefit from this technology in the future. PROJECT MODIFICATIONS: No project modifications.

Impacts
The technology was presented in a new extension circular NEB EC708 which will reach hundreds of growers in the Midwest. This research was also featured in an article of the Institute of Biological Engineering (IBE) 2007 November e-life Newsletter which reaches over 200 professionals. The ability for a machine to identify species, assess stages of growth, and count individual crop and weed plants is a complex but extremely worthwhile process. A potentially large number of applications exist for mapping evasive species and ecological assessments. Weed patchiness has been well documented in the literature and herbicide savings of 40 to 70 percent have been projected, only if the patch areas are mapped and treated, as opposed to spraying the entire field. Applications may include weeds with either GMO or non-GMO crops.

Publications

  • 1.Sethuramasamyraja, B., V.I. Adamchuk, D.B. Marx, A. Dobermann, G.E. Meyer, and D.D. Jones, 2007. Analysis of an ion-selective electrode based methodology for integrated on-the-go mapping of soil pH, potassium and nitrate contents. TRANSACTIONS of ASABE 50(6):1927-1935.
  • 2.Sethuramasamyraja, B., V.I. Adamchuk, A. Dobermann, D.B. Marx, D.D. Jones, and G.E. Meyer, 2008. Agitated soil measurement method for integrated on-the-go mapping of soil pH, potassium and nitrate contents. Computers and Electronics in Agriculture (Elsevier), 60:212-225.
  • 3.Govindarajan K.N., L. M. Grimes, J. Subbiah, C. R. Calkins, A. Samal, and G.E. Meyer, 2008. Visible/near-infrared hyperspectral imaging for beef tenderness prediction . Computers and Electronics in Agriculture (Elsevier), 64:225-233.
  • 4.Meyer, G.E. and J. Camargo Neto, 2008. Verification of Color Vegetation Indices for Automated Crop Imaging Applications, Computers and Electronics in Agriculture (Elsevier), 63:282-293.
  • 5.Jones, D.D. and George E. Meyer. 2008. Advanced modeling in biological engineering using soft-computing methods. ASABE Paper No. 08-5109, ASABE Meeting Presentation, Providence, RI, June 29-July 2, 2008.
  • 6.Adamchuk, V.I., M.L. Bernards, G.E. Meyer, and J. Mulliken. 2008. Weed targeting herbicide management. Precision Agriculture Extension Circular EC 708. Lincoln, Nebraska: University of Nebraska Extension.
  • 7.Chen, Y.R., G.E. Meyer, and Shu-I Tu, 2007. Optics for Natural Resources, Agriculture, and Foods II (Conference Proceedings Volume) Proc. SPIE, Bellingham WA, Vol: 6761 (ISBN: 9780819469212) 290 pp.
  • 8.Govindarajan K.N., L. M. Grimes, J. Subbiah C. R. Calkins, A. Samal, and G.E. Meyer, 2008. Partial least squares analysis of near-infrared hyperspectral images for beef tenderness prediction. Sens. and Instrumen. Food Qual. (Springer) xx:xxx-xxx.


Progress 10/01/06 to 09/30/07

Outputs
OUTPUTS: A solution for close-in, digital photography of large scale field areas for precision weed assessment is currently being evaluated. Satellite and aerial photography to date has not presented sufficient resolution or detail for the species identification system. A study of minimum photographic resolution for concurrent leaf shape and concurrent textural venation analysis is underway through an undergraduate UNL UCARE project. Efforts are on-going to make the fussy clustering and genetic algorithm leaf extraction algorithm of the weed species identification system more efficient. This is the largest processing time component of the software system. TARGET AUDIENCES: The target audience are weed control machinery companies and Nebraska corn, soybean, and sorghum growers. Other possible audiences are ecological plant survey groups and organizations for tracking plant flora and invasive species.

Impacts
A method for traversing tram lines for visual assessment of weeds from an all-terrain vehicle continues to be successful by a Nebraska crop consultant. (http://www.deere.com/en_US/ag/pdf/furrow/2007/summer_07_F0702816.pdf )

Publications

  • Adamchuk, V.I., R.M. Hoy, G.E. Meyer, and M.F. Kocher. 2007. GPS-based auto-guidance test program development. In: Precision Agriculture: Papers from the Sixth European Conference on Precision Agriculture, Skiathos, Greece, 3-6 June 2007, 425-432. J. Stafford, ed. Wageningen, The Netherlands: Wageningen Academic Publishers.
  • C. Schmid, R. Gaussoin, G.E. Meyer, R.C. Shearman, and G. DeBerg. 2007. A Rapid and Precise Method to Measure Infiltration in situ. Poster presentation 171-1(656), Turf grass Science. ASA-CSSA-SSSA International Annual Meetings at the Ernest N. Morial Convention Center, New Orleans, LA.
  • Meyer, G. E., D. D. Jones, and J. Camargo Neto. 2007. Botanical Species Identification using Image Extracted Individual Leaves and Soft Computing. Institute of Biological Engineering (IBE) Annual Meeting, St. Louis, Mo. (http://www.ibe.org/elife/newsletters/v11i2.pdf)
  • Meyer, G. E. and D. D. Jones.2007. Advanced Modeling in Biological Engineering using Soft-Computing Methods. (AC 2007-2729). Proceedings of Annual Conference. American Society of Engineering Education (ASEE), Honolulu, Hawaii.


Progress 10/01/05 to 09/30/06

Outputs
The machine vision species identification system currently under development has been improved, and tested with plants grown in a greenhouse and field images from the USDA MESA site at Shelton, NE. Some of the critical and novel software algorithms and their tests results were published in two 2006 Elsevier journal articles and a 2005 SPIE conference symposia publication. The system was demonstrated to growers and crop consultants at the Nebraska Agricultural technologies Association (NEATA), Grand, Island, in February 2006. Attendees indicated positive feedback to this new system. A prototype hand-held, electronic, and ergonomic tool was also developed to assist visual stategic recording and mapping of weed species types and their populations (summer annuals and winter perenials) to a hand-held GPS PDA device. The system was successfully used for a number of field observations by a crop consultant from an all-terrain-vehicle (ATV) moving along herbicide spray lines.

Impacts
The ability for a machine to identify species, assess stages of growth, and count individual crop and weed plants is a complex but worthwhile process. A potentially large number of applications exist for mapping evasive species and ecological assessments. Weed patchiness has been well documented in the literature and herbicide savings of 40 to 70 percent have been projected, only if the patch areas are mapped and treated, as opposed to spraying the entire field. Applications may include weeds with either GMO or non-GMO crops.

Publications

  • Camargo Neto, J., G.E. Meyer. 2005. Crop species identification using machine vision of computer extracted individual leaves. In: Chen, Y.R., Meyer, G.E., Tu S. (Eds.), Optical Sensors and Sensing Systems for Natural Resources and Food Safety and Quality, Proc. SPIE, Bellingham WA., Vol. 5996, pp 64-74.
  • Camargo Neto, G.E. Meyer, D. D. Jones, A.K. Samal. 2006. Plant Species Identification using Elliptic Fourier Analysis. Computers and Electronics in Agriculture (Elsevier), 50:121-134.
  • Camargo Neto, G.E. Meyer , D. D. Jones. 2006. Individual Leaf Extractions from Young Canopy Images using Gustafson-Kessel Clustering and a Genetic Algorithm. Computers and Electronics in Agriculture (Elsevier) 51:65-85.