Weed Detection with YOLOv5: 24 Species in Field Photos and Video

Overview

A computer-vision system that finds and names weeds in photos and video taken in the field. It recognises 24 weed species, draws a box around each plant it finds and writes the species and a confidence value on it. The goal is the first step of smart weed control: before anything can be sprayed or pulled only where it is needed, the system has to know which weed is where. I built this project in 2019.

The problem

Weeds do not sit alone on a clean background. In a real photo they grow between crop and other plants, in shadow, in different growth stages, with and without flowers, and several species often share one frame. Many weeds also look alike from above. So the task is not "which plant is this picture of" but "where are the weeds in this picture and which species is each one".

What I did

• Data. Field photos and short videos of 24 species, from tiny single plants on bare soil to dense patches in grass. The photos were taken in the field by other people, and my part was everything that came after: preparing, labelling, training and evaluating. After preparation the dataset has 8,225 images at 416 px, split into 6,580 for training, 822 for validation and 823 for testing. Several variants of every photo (crops and flips) were made to make the data more varied.

• Species list. Together with the botanical notes, species that were really one group were merged (for example, two names of the wild lettuce), and the species with few or weak photos were listed so that more close-ups, especially of their flowers, could be collected.

• First try: classification. I first trained image classifiers, a CNN of my own design and an Inception-type network, which say "this whole picture is species X". The result was not good enough on this data, because a field photo has more than one plant and the label did not say where the weed is.

• Second try: object detection. I switched to labelling every plant with a bounding box (with RectLabel, in YOLO format) and training a YOLOv5 detector. That is the version that worked.

• Training and evaluation. YOLOv5s, 100 epochs, 416 px, batch of 16, on a workstation with a Quadro RTX 5000 GPU (about 42 seconds per epoch, 1 hour 10 minutes in total). I checked the result with the loss curves, the precision and recall curves, and a confusion matrix for every species.

• Use. A script that runs the model on an image folder or on a video and writes the annotated result, so the same model works on photos and on the footage of a moving camera.

Results

• On the validation set the detector reaches an mAP@0.5 of about 0.69, a precision of about 0.65 to 0.70 and a recall of about 0.62 to 0.69.

• Some species are recognised very well: Veronica persica (1.00 in the confusion matrix), Conyza canadensis (0.91), Chenopodium album (0.89), Portulaca oleracea (0.86), Achillea millefolium (0.83).

• The weakest species are Avena fatua (wild oat), Medicago sativa, Cyperus rotundus, Hordeum murinum and Poa annua. They are grasses and low plants that look alike from a distance and need more and better photos.

• The model is fast enough for video; with YOLO it runs at about 140 frames per second on the workstation.

Tools

• Python, PyTorch, YOLOv5 (the open-source Ultralytics implementation), TensorFlow and Keras (for the classification tests)

• OpenCV, NumPy, scikit-learn, Matplotlib, Jupyter

• RectLabel for labelling; Ubuntu with CUDA

Result

A trained weed detector for 24 species that works on photos and on video, with a clear list of where it is strong, where it needs more data, and what a field test must still prove.