From 62a0fb986a6f2ebe654594ea2d30f25e2b713a46 Mon Sep 17 00:00:00 2001 From: Burhan <62214284+Burhan-Q@users.noreply.github.com> Date: Fri, 28 Jul 2023 08:57:24 -0400 Subject: [PATCH] Add new K-Fold cross validation guide in Docs (#3975) Co-authored-by: Glenn Jocher --- docs/guides/index.md | 19 ++ docs/guides/kfold-cross-validation.md | 263 ++++++++++++++++++++++++++ docs/hub/app/index.md | 2 +- docs/integrations/index.md | 28 +-- docs/integrations/openvino.md | 2 +- mkdocs.yml | 3 + ultralytics/engine/model.py | 5 +- 7 files changed, 304 insertions(+), 18 deletions(-) create mode 100644 docs/guides/index.md create mode 100644 docs/guides/kfold-cross-validation.md diff --git a/docs/guides/index.md b/docs/guides/index.md new file mode 100644 index 0000000..2443b57 --- /dev/null +++ b/docs/guides/index.md @@ -0,0 +1,19 @@ +--- +comments: true +description: In-depth exploration of Ultralytics' YOLO. Learn about the YOLO object detection model, how to train it on custom data, multi-GPU training, exporting, predicting, deploying, and more. +keywords: Ultralytics, YOLO, Deep Learning, Object detection, PyTorch, Tutorial, Multi-GPU training, Custom data training +--- + +# Comprehensive Tutorials to Ultralytics YOLO + +Welcome to the Ultralytics' YOLO ๐Ÿš€ Guides! Our comprehensive tutorials cover various aspects of the YOLO object detection model, ranging from training and prediction to deployment. Built on PyTorch, YOLO stands out for its exceptional speed and accuracy in real-time object detection tasks. + +Whether you're a beginner or an expert in deep learning, our tutorials offer valuable insights into the implementation and optimization of YOLO for your computer vision projects. Let's dive in! + +## Guides + +Here's a compilation of in-depth guides to help you master different aspects of Ultralytics YOLO. + +* [K-Fold Cross Validation](kfold-cross-validation.md) ๐Ÿš€ NEW: Learn how to improve model generalization using K-Fold cross-validation technique. + +Note: More guides about training, exporting, predicting, and deploying with Ultralytics YOLO are coming soon. Stay tuned! diff --git a/docs/guides/kfold-cross-validation.md b/docs/guides/kfold-cross-validation.md new file mode 100644 index 0000000..f5c668d --- /dev/null +++ b/docs/guides/kfold-cross-validation.md @@ -0,0 +1,263 @@ +--- +comments: true +description: An in-depth guide demonstrating the implementation of K-Fold Cross Validation with the Ultralytics ecosystem for object detection datasets, leveraging Python, YOLO, and sklearn. +keywords: K-Fold cross validation, Ultralytics, YOLO detection format, Python, sklearn, object detection +--- + +# K-Fold Cross Validation in the Ultralytics Ecosystem + +## Introduction + +This comprehensive guide illustrates the implementation of K-Fold Cross Validation for object detection datasets within the Ultralytics ecosystem. We'll leverage the YOLO detection format and key Python libraries such as sklearn, pandas, and PyYaml to guide you through the necessary setup, the process of generating feature vectors, and the execution of a K-Fold dataset split. + +Whether your project involves the Fruit Detection dataset or a custom data source, this tutorial aims to help you comprehend and apply K-Fold Cross Validation to bolster the reliability and robustness of your machine learning models. While we're applying `k=5` folds for this tutorial, keep in mind that the optimal number of folds can vary depending on your dataset and the specifics of your project. + +Without further ado, let's dive in! + +## Setup + +- Your annotations should be in the [YOLO detection format](https://docs.ultralytics.com/datasets/detect/). + +- This guide assumes that annotation files are locally available. + + - For our demonstration, we use the [Fruit Detection](https://www.kaggle.com/datasets/lakshaytyagi01/fruit-detection/code) dataset. + + - This dataset contains a total of 8479 images. + - It includes 6 class labels, each with its total instance counts listed below. + + | Class Label | Instance Count | + |:------------|:--------------:| + | Apple | 7049 | + | Grapes | 7202 | + | Pineapple | 1613 | + | Orange | 15549 | + | Banana | 3536 | + | Watermelon | 1976 | + +- Necessary Python packages include: + + - `ultralytics` + - `sklearn` + - `pandas` + - `pyyaml` + +- This tutorial operates with `k=5` folds. However, you should determine the best number of folds for your specific dataset. + +1. Initiate a new Python virtual environment (`venv`) for your project and activate it. Use `pip` (or your preferred package manager) to install: + + - The Ultralytics library: `pip install -U ultralytics`. Alternatively, you can clone the official [repo](https://github.com/ultralytics/ultralytics). + - Scikit-learn, pandas, and PyYAML: `pip install -U scikit-learn pandas pyyaml`. + +2. Verify that your annotations are in the [YOLO detection format](https://docs.ultralytics.com/datasets/detect/). + + - For this tutorial, all annotation files are found in the `Fruit-Detection/labels` directory. + +## Generating Feature Vectors for Object Detection Dataset + +1. Start by creating a new Python file and import the required libraries. + + ```python + import datetime + import shutil + from pathlib import Path + from collections import Counter + + import yaml + import numpy as np + import pandas as pd + from ultralytics import YOLO + from sklearn.model_selection import KFold + ``` + +2. Proceed to retrieve all label files for your dataset. + + ```python + dataset_path = Path('./Fruit-detection') # replace with 'path/to/dataset' for your custom data + labels = sorted(dataset_path.rglob("*labels/*.txt")) # all data in 'labels' + ``` + +3. Now, read the contents of the dataset YAML file and extract the indices of the class labels. + + ```python + with open(yaml_file, 'r', encoding="utf8") as y: + classes = yaml.safe_load(y)['names'] + cls_idx = sorted(classes.keys()) + ``` + +4. Initialize an empty `pandas` DataFrame. + + ```python + indx = [l.stem for l in labels] # uses base filename as ID (no extension) + labels_df = pd.DataFrame([], columns=cls_idx, index=indx) + ``` + +5. Count the instances of each class-label present in the annotation files. + + ```python + for label in labels: + lbl_counter = Counter() + + with open(label,'r') as lf: + lines = lf.readlines() + + for l in lines: + # classes for YOLO label uses integer at first position of each line + lbl_counter[int(l.split(' ')[0])] += 1 + + labels_df.loc[label.stem] = lbl_counter + + labels_df = labels_df.fillna(0.0) # replace `nan` values with `0.0` + ``` + +6. The following is a sample view of the populated DataFrame: + + ```pandas + 0 1 2 3 4 5 + '0000a16e4b057580_jpg.rf.00ab48988370f64f5ca8ea4...' 0.0 0.0 0.0 0.0 0.0 7.0 + '0000a16e4b057580_jpg.rf.7e6dce029fb67f01eb19aa7...' 0.0 0.0 0.0 0.0 0.0 7.0 + '0000a16e4b057580_jpg.rf.bc4d31cdcbe229dd022957a...' 0.0 0.0 0.0 0.0 0.0 7.0 + '00020ebf74c4881c_jpg.rf.508192a0a97aa6c4a3b6882...' 0.0 0.0 0.0 1.0 0.0 0.0 + '00020ebf74c4881c_jpg.rf.5af192a2254c8ecc4188a25...' 0.0 0.0 0.0 1.0 0.0 0.0 + ... ... ... ... ... ... ... + 'ff4cd45896de38be_jpg.rf.c4b5e967ca10c7ced3b9e97...' 0.0 0.0 0.0 0.0 0.0 2.0 + 'ff4cd45896de38be_jpg.rf.ea4c1d37d2884b3e3cbce08...' 0.0 0.0 0.0 0.0 0.0 2.0 + 'ff5fd9c3c624b7dc_jpg.rf.bb519feaa36fc4bf630a033...' 1.0 0.0 0.0 0.0 0.0 0.0 + 'ff5fd9c3c624b7dc_jpg.rf.f0751c9c3aa4519ea3c9d6a...' 1.0 0.0 0.0 0.0 0.0 0.0 + 'fffe28b31f2a70d4_jpg.rf.7ea16bd637ba0711c53b540...' 0.0 6.0 0.0 0.0 0.0 0.0 + ``` + +The rows index the label files, each corresponding to an image in your dataset, and the columns correspond to your class-label indices. Each row represents a pseudo feature-vector, with the count of each class-label present in your dataset. This data structure enables the application of K-Fold Cross Validation to an object detection dataset. + +## K-Fold Dataset Split + +1. Now we will use the `KFold` class from `sklearn.model_selection` to generate `k` splits of the dataset. + + - Important: + - Setting `shuffle=True` ensures a randomized distribution of classes in your splits. + - By setting `random_state=M` where `M` is a chosen integer, you can obtain repeatable results. + + ```python + ksplit = 5 + kf = KFold(n_splits=ksplit, shuffle=True, random_state=20) # setting random_state for repeatable results + + kfolds = list(kf.split(labels_df)) + ``` + +2. The dataset has now been split into `k` folds, each having a list of `train` and `val` indices. We will construct a DataFrame to display these results more clearly. + +```python +folds = [f'split_{n}' for n in range(1, ksplit + 1)] +folds_df = pd.DataFrame(index=indx, columns=folds) + +for idx, (train, val) in enumerate(kfolds, start=1): + folds_df[f'split_{idx}'].loc[labels_df.iloc[train].index] = 'train' + folds_df[f'split_{idx}'].loc[labels_df.iloc[val].index] = 'val' +``` + +3. Now we will calculate the distribution of class labels for each fold as a ratio of the classes present in `val` to those present in `train`. + +```python +fold_lbl_distrb = pd.DataFrame(index=folds, columns=cls_idx) + +for n, (train_indices, val_indices) in enumerate(kfolds, start=1): + train_totals = labels_df.iloc[train_indices].sum() + val_totals = labels_df.iloc[val_indices].sum() + + # To avoid division by zero, we add a small value (1E-7) to the denominator + ratio = val_totals / (train_totals + 1E-7) + fold_lbl_distrb.loc[f'split_{n}'] = ratio +``` + +The ideal scenario is for all class ratios to be reasonably similar for each split and across classes. This, however, will be subject to the specifics of your dataset. + +4. Next, we create the directories and dataset YAML files for each split. + +```python +save_path = Path(dataset_path / f'{datetime.date.today().isoformat()}_{ksplit}-Fold_Cross-val') +save_path.mkdir(parents=True, exist_ok=True) + +images = sorted((dataset_path / 'images').rglob("*.jpg")) # change file extension as needed +ds_yamls = [] + +for split in folds_df.columns: + # Create directories + split_dir = save_path / split + split_dir.mkdir(parents=True, exist_ok=True) + (split_dir / 'train' / 'images').mkdir(parents=True, exist_ok=True) + (split_dir / 'train' / 'labels').mkdir(parents=True, exist_ok=True) + (split_dir / 'val' / 'images').mkdir(parents=True, exist_ok=True) + (split_dir / 'val' / 'labels').mkdir(parents=True, exist_ok=True) + + # Create dataset YAML files + dataset_yaml = split_dir / f'{split}_dataset.yaml' + ds_yamls.append(dataset_yaml) + + with open(dataset_yaml, 'w') as ds_y: + yaml.safe_dump({ + 'path': save_path.as_posix(), + 'train': 'train', + 'val': 'val', + 'names': classes + }, ds_y) +``` + +5. Lastly, copy images and labels into the respective directory ('train' or 'val') for each split. + + - __NOTE:__ The time required for this portion of the code will vary based on the size of your dataset and your system hardware. + + ```python + for image, label in zip(images, labels): + for split, k_split in folds_df.loc[image.stem].items(): + # Destination directory + img_to_path = save_path / split / k_split / 'images' + lbl_to_path = save_path / split / k_split / 'labels' + + # Copy image and label files to new directory + # Might throw a SamefileError if file already exists + shutil.copy(image, img_to_path / image.name) + shutil.copy(label, lbl_to_path / label.name) + ``` + +## Save Records (Optional) + +Optionally, you can save the records of the K-Fold split and label distribution DataFrames as CSV files for future reference. + +```python +folds_df.to_csv(save_path / "kfold_datasplit.csv") +fold_lbl_distrb.to_csv(save_path / "kfold_label_distribution.csv") +``` + +## Train YOLO using K-Fold Data Splits + +1. First, load the YOLO model. + + ```python + weights_path = 'path/to/weights.pt' + model = YOLO(weights_path, task='detect') + ``` + +2. Next, iterate over the dataset YAML files to run training. The results will be saved to a directory specified by the `project` and `name` arguments. By default, this directory is 'exp/runs#' where # is an integer index. + + ```python + results = {} + for k in range(ksplit): + dataset_yaml = ds_yamls[k] + model.train(data=dataset_yaml, *args, **kwargs) # Include any training arguments + results[k] = model.metrics # save output metrics for further analysis + ``` + +In this updated section, I have replaced manual string joining with the built-in `Path` method for constructing directories, which makes the code more Pythonic. I have also improved the explanation and clarity of the instructions. + +## Conclusion + +In this guide, we have explored the process of using K-Fold cross-validation for training the YOLO object detection model. We learned how to split our dataset into K partitions, ensuring a balanced class distribution across the different folds. + +We also explored the procedure for creating report DataFrames to visualize the data splits and label distributions across these splits, providing us a clear insight into the structure of our training and validation sets. + +Optionally, we saved our records for future reference, which could be particularly useful in large-scale projects or when troubleshooting model performance. + +Finally, we implemented the actual model training using each split in a loop, saving our training results for further analysis and comparison. + +This technique of K-Fold cross-validation is a robust way of making the most out of your available data, and it helps to ensure that your model performance is reliable and consistent across different data subsets. This results in a more generalizable and reliable model that is less likely to overfit to specific data patterns. + +Remember that although we used YOLO in this guide, these steps are mostly transferable to other machine learning models. Understanding these steps allows you to apply cross-validation effectively in your own machine learning projects. Happy coding! \ No newline at end of file diff --git a/docs/hub/app/index.md b/docs/hub/app/index.md index af20578..dd44d95 100644 --- a/docs/hub/app/index.md +++ b/docs/hub/app/index.md @@ -49,4 +49,4 @@ Welcome to the Ultralytics HUB App! We are excited to introduce this powerful mo - [**iOS**](./ios.md): Learn about YOLO CoreML models accelerated on Apple's Neural Engine for iPhones and iPads. - [**Android**](./android.md): Explore TFLite acceleration on Android mobile devices. -Get started today by downloading the Ultralytics HUB App on your mobile device and unlock the potential of YOLOv5 and YOLOv8 models on-the-go. Don't forget to check out our comprehensive [HUB Docs](../) for more information on training, deploying, and using your custom models with the Ultralytics HUB platform. +Get started today by downloading the Ultralytics HUB App on your mobile device and unlock the potential of YOLOv5 and YOLOv8 models on-the-go. Don't forget to check out our comprehensive [HUB Docs](../index.md) for more information on training, deploying, and using your custom models with the Ultralytics HUB platform. diff --git a/docs/integrations/index.md b/docs/integrations/index.md index 89e5964..d4d3ad2 100644 --- a/docs/integrations/index.md +++ b/docs/integrations/index.md @@ -42,20 +42,20 @@ Welcome to the Ultralytics Integrations page! This page provides an overview of We also support a variety of model export formats for deployment in different environments. Here are the available formats: -| Format | `format` Argument | Model | Metadata | Arguments | -|--------------------------------------------------------------------|-------------------|---------------------------|----------|-----------------------------------------------------| -| [PyTorch](https://pytorch.org/) | - | `yolov8n.pt` | โœ… | - | -| [TorchScript](https://pytorch.org/docs/stable/jit.html) | `torchscript` | `yolov8n.torchscript` | โœ… | `imgsz`, `optimize` | -| [ONNX](https://onnx.ai/) | `onnx` | `yolov8n.onnx` | โœ… | `imgsz`, `half`, `dynamic`, `simplify`, `opset` | -| [OpenVINO](/integrations/openvino.md) | `openvino` | `yolov8n_openvino_model/` | โœ… | `imgsz`, `half` | -| [TensorRT](https://developer.nvidia.com/tensorrt) | `engine` | `yolov8n.engine` | โœ… | `imgsz`, `half`, `dynamic`, `simplify`, `workspace` | -| [CoreML](https://github.com/apple/coremltools) | `coreml` | `yolov8n.mlmodel` | โœ… | `imgsz`, `half`, `int8`, `nms` | -| [TF SavedModel](https://www.tensorflow.org/guide/saved_model) | `saved_model` | `yolov8n_saved_model/` | โœ… | `imgsz`, `keras` | +| Format | `format` Argument | Model | Metadata | Arguments | +|-------------------------------------------------------------------|-------------------|---------------------------|----------|-----------------------------------------------------| +| [PyTorch](https://pytorch.org/) | - | `yolov8n.pt` | โœ… | - | +| [TorchScript](https://pytorch.org/docs/stable/jit.html) | `torchscript` | `yolov8n.torchscript` | โœ… | `imgsz`, `optimize` | +| [ONNX](https://onnx.ai/) | `onnx` | `yolov8n.onnx` | โœ… | `imgsz`, `half`, `dynamic`, `simplify`, `opset` | +| [OpenVINO](openvino.md) | `openvino` | `yolov8n_openvino_model/` | โœ… | `imgsz`, `half` | +| [TensorRT](https://developer.nvidia.com/tensorrt) | `engine` | `yolov8n.engine` | โœ… | `imgsz`, `half`, `dynamic`, `simplify`, `workspace` | +| [CoreML](https://github.com/apple/coremltools) | `coreml` | `yolov8n.mlmodel` | โœ… | `imgsz`, `half`, `int8`, `nms` | +| [TF SavedModel](https://www.tensorflow.org/guide/saved_model) | `saved_model` | `yolov8n_saved_model/` | โœ… | `imgsz`, `keras` | | [TF GraphDef](https://www.tensorflow.org/api_docs/python/tf/Graph) | `pb` | `yolov8n.pb` | โŒ | `imgsz` | -| [TF Lite](https://www.tensorflow.org/lite) | `tflite` | `yolov8n.tflite` | โœ… | `imgsz`, `half`, `int8` | -| [TF Edge TPU](https://coral.ai/docs/edgetpu/models-intro/) | `edgetpu` | `yolov8n_edgetpu.tflite` | โœ… | `imgsz` | -| [TF.js](https://www.tensorflow.org/js) | `tfjs` | `yolov8n_web_model/` | โœ… | `imgsz` | -| [PaddlePaddle](https://github.com/PaddlePaddle) | `paddle` | `yolov8n_paddle_model/` | โœ… | `imgsz` | -| [NCNN](https://github.com/Tencent/ncnn) | `ncnn` | `yolov8n_ncnn_model/` | โœ… | `imgsz`, `half` | +| [TF Lite](https://www.tensorflow.org/lite) | `tflite` | `yolov8n.tflite` | โœ… | `imgsz`, `half`, `int8` | +| [TF Edge TPU](https://coral.ai/docs/edgetpu/models-intro/) | `edgetpu` | `yolov8n_edgetpu.tflite` | โœ… | `imgsz` | +| [TF.js](https://www.tensorflow.org/js) | `tfjs` | `yolov8n_web_model/` | โœ… | `imgsz` | +| [PaddlePaddle](https://github.com/PaddlePaddle) | `paddle` | `yolov8n_paddle_model/` | โœ… | `imgsz` | +| [NCNN](https://github.com/Tencent/ncnn) | `ncnn` | `yolov8n_ncnn_model/` | โœ… | `imgsz`, `half` | Explore the links to learn more about each integration and how to get the most out of them with Ultralytics. diff --git a/docs/integrations/openvino.md b/docs/integrations/openvino.md index 0cc1468..9460449 100644 --- a/docs/integrations/openvino.md +++ b/docs/integrations/openvino.md @@ -236,7 +236,7 @@ Benchmarks below run on 13th Gen Intelยฎ Coreยฎ i7-13700H CPU at FP32 precision. ## Reproduce Our Results -To reproduce the Ultralytics benchmarks above on all export [formats](/modes/export.md) run this code: +To reproduce the Ultralytics benchmarks above on all export [formats](../modes/export.md) run this code: !!! example "" diff --git a/mkdocs.yml b/mkdocs.yml index 7faccef..9687f2c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -208,6 +208,9 @@ nav: - MNIST: datasets/classify/mnist.md - Multi-Object Tracking: - datasets/track/index.md + - Guides: + - guides/index.md + - K-Fold Cross Validation: guides/kfold-cross-validation.md - Integrations: - integrations/index.md - OpenVINO: integrations/openvino.md diff --git a/ultralytics/engine/model.py b/ultralytics/engine/model.py index b5aeff9..68891bf 100644 --- a/ultralytics/engine/model.py +++ b/ultralytics/engine/model.py @@ -456,9 +456,10 @@ class Model: @property def task_map(self): - """Map head to model, trainer, validator, and predictor classes + """ + Map head to model, trainer, validator, and predictor classes. Returns: - task_map (dict) + task_map (dict): The map of model task to mode classes. """ raise NotImplementedError('Please provide task map for your model!')