Efficient Hyperparameter Tuning with Ray Tune and YOLOv8 - Ultralytics YOLOv8 Docs

[giscus[bot]]

Efficient Hyperparameter Tuning with Ray Tune and YOLOv8 - Ultralytics YOLOv8 Docs

Learn to integrate hyperparameter tuning using Ray Tune with Ultralytics YOLOv8, and optimize your model's performance efficiently.

https://docs.ultralytics.com/usage/hyperparameter_tuning/?h=hyperparameter

[Won-JoonChoi]

Any resolutions when using yolov8m, yolov8l, yolov8x models?
ValueError: The actor ImplicitFunc is too large (198 MiB > FUNCTION_SIZE_ERROR_THRESHOLD=95 MiB). Check that its definition is not implicitly capturing a large array or other object in scope. Tip: use ray.put() to put large objects in the Ray object store.
I can't figure out ways to use 'ray.put()'.

[esgitty]

Also having a similar issue, would like to know when/how this can be addressed.

[glenn-jocher]

You might try the Ultralytics Tuner instead of the Ray Tuner. Here's an example:

from ultralytics import YOLO

# Initialize the YOLO model
model = YOLO('yolov8n.pt')

# Tune hyperparameters on COCO8 for 30 epochs
model.tune(data='coco8.yaml', epochs=30, iterations=300, optimizer='AdamW', plots=False, save=False, val=False)

[diwashrestha]

I am getting an error while using this technique for the classification model. It is working fine for model.train.

• Code:
  model.tune(data="/kaggle/working/dataset/",epochs=30, iterations=300, optimizer='AdamW', plots=False, save=False, val=False)

• Error:
  `---------------------------------------------------------------------------
  UnboundLocalError Traceback (most recent call last)
  Cell In[15], line 1
  ----> 1 model.tune(data="/kaggle/working/dataset/",epochs=30, iterations=300, optimizer='AdamW', plots=False, save=False, val=False)

File /opt/conda/lib/python3.10/site-packages/ultralytics/engine/model.py:363, in Model.tune(self, use_ray, iterations, *args, **kwargs)
361 custom = {} # method defaults
362 args = {**self.overrides, **custom, **kwargs, 'mode': 'train'} # highest priority args on the right
--> 363 return Tuner(args=args, _callbacks=self.callbacks)(model=self, iterations=iterations)

File /opt/conda/lib/python3.10/site-packages/ultralytics/engine/tuner.py:207, in Tuner.call(self, model, iterations, cleanup)
205 shutil.copy2(ckpt, self.tune_dir / 'weights')
206 elif cleanup:
--> 207 shutil.rmtree(ckpt_file.parent) # remove iteration weights/ dir to reduce storage space
209 # Plot tune results
210 plot_tune_results(self.tune_csv)

UnboundLocalError: local variable 'ckpt_file' referenced before assignment`

[prathikbafna]

@diwashrestha , The first parameter inside the model.tune function should be path to the yaml file and not the dataset.

[abdulhai1234]

I am writing a research paper on the yolov8n for segmentation, where i can find the details about the Architecture of the model, or already available Manuscript on it

[pderrenger]

@abdulhai1234 for detailed information on the architecture of YOLOv8n for segmentation, you can refer to the official Ultralytics documentation. The documentation provides insights into the model's structure, layers, and overall design philosophy. While there may not be a dedicated manuscript specifically for YOLOv8n segmentation, the documentation is a comprehensive source that can guide you through the technical aspects of the model.

If you're looking for more academic or formal references, you might consider citing the repository itself or any related technical reports or blog posts provided by Ultralytics. Additionally, you can explore the repository's commit history and release notes for specific details on the architecture changes and updates.

For segmentation tasks, you might also find valuable information in the documentation sections related to the Segment Anything Model (SAM) and how YOLOv8 handles segmentation in general. Keep an eye on the official Ultralytics channels for any new publications or updates that might be relevant to your research. 📚🔍

[bagherig]

when loading the results of ray tune, how do I restore the original trainable object?

restored_tuner = tune.Tuner.restore(experiment_path, trainable=train_mnist)

[jokober]

Check out this issue: #3630

You only have to use change trainable=train_mnist to trainable="_tune"

[pderrenger]

@jokober to restore the original trainable object when loading the results of Ray Tune, you would typically use the restore method provided by Ray Tune. However, in the context of YOLOv8, you should replace train_mnist with the specific training function or class you used for the YOLOv8 model. If you're following our documentation or examples, it might be a function or class that was defined to handle the training loop for YOLOv8.

For example, if you have a training function named train_yolov8, you would use:

restored_tuner = tune.Tuner.restore(experiment_path, trainable=train_yolov8)

If you're using a predefined trainable from our repository, ensure you're referencing it correctly. If you encounter any issues or need further assistance, feel free to reach out on our GitHub issues page with specific details, and we'll be happy to help! 🚀

[Mallikarjunakirapalli2110]

I have a trained yolov8l model and when i try to furthur train it with newer data,it is failing to work as it used to

[voiresa]

have you know how to do it on new datasets now？I am also struggling with the normalization of my own datasets

[pderrenger]

@voiresa when further training a YOLOv8 model with new data, it's important to ensure that the new dataset is prepared and normalized correctly. Here are some steps you can follow to address issues with training on new datasets:

1. Dataset Preparation: Verify that your new dataset is correctly annotated and structured in a way that YOLOv8 expects. The dataset should have the appropriate directory structure and annotation files (e.g., in COCO format).

2. Normalization: Ensure that your images and bounding boxes are normalized correctly. YOLOv8 expects images to be in the RGB color space and annotations to be in relative coordinates (i.e., as a fraction of the image width and height).

3. Configuration File: Update your dataset configuration file (usually a .yaml file) to reflect the paths to your new dataset's images and annotations. Also, specify the number of classes if it differs from the pre-trained model.

4. Hyperparameters: If you're using custom hyperparameters, ensure they are suitable for your new dataset. Sometimes, the hyperparameters tuned for one dataset may not work well for another.

5. Transfer Learning: When further training (fine-tuning) a model, you might want to freeze the early layers of the network and only train the later layers. This can be done by setting the freeze parameter in your training script.

6. Resume Training: If you're resuming training, ensure that you're using the resume flag correctly and that the checkpoint file is accessible.

7. Validation: Use the val mode to validate your model on the new dataset to check its performance before continuing training.

8. Error Messages: Pay close attention to any error messages or warnings during training, as they can provide clues about what might be going wrong.

If you're still facing issues, consider revisiting the Ultralytics Docs for more detailed guidance on training with custom datasets, and ensure that you're following the recommended practices for YOLOv8. If the problem persists, you can reach out with specific error messages or issues you're encountering for more targeted assistance. 🛠️

[MrABC11]

Hello, when Python 3.11 IDLE tries to download yolov8n.pt while running the line "model = YOLO('yolov8n.pt')" it gets crahed and says Not responding every time. Can anyone help me?

[glenn-jocher]

If you want you can also download models manually from https://github.com/ultralytics/assets/releases/tag/v0.0.0

[nadinator]

What's train_mnist in the restoration part of the guide?

[pderrenger]

@nadinator the train_mnist reference seems to be a placeholder or an error in the documentation. It should likely refer to a function or method related to training a model on the MNIST dataset, which is a common example for demonstrating machine learning tasks. However, in the context of YOLOv8 and Ultralytics, you would be training on datasets more complex than MNIST, typically involving object detection tasks.

If you're looking to restore or resume training with YOLOv8, you would use the resume parameter in the training configuration. For example:

from ultralytics import YOLO

# Load the model
model = YOLO('path/to/your_last_checkpoint.pt')

# Resume training
results = model.train(resume=True)

This will continue training from the last saved checkpoint. If you have further questions or need clarification, please refer to the Train documentation for more details on how to train your YOLOv8 models. 😊🚀

[Shivangi1807]

Hi,
How can we do ensemble modelling of Yolov8 different models using CLI for custom dataset? Please help.

[pderrenger]

@Shivangi1807 ensemble modeling with YOLOv8 models can be achieved by running inference with multiple models and combining their predictions. Currently, the CLI does not directly support ensemble modeling as a single command, but you can perform ensemble modeling by following these steps:

1. Run predictions with each of your YOLOv8 models separately on your custom dataset.
2. Save the prediction results (e.g., bounding boxes, class labels, and confidence scores) for each model.
3. Write a custom script to combine the predictions from all models. This could involve techniques like Non-Maximum Suppression (NMS) across all model outputs, weighted box fusion, or other methods to merge the results.
4. Use the combined predictions as the final ensemble output.

For the first step, you can use the predict mode with each of your models:

from ultralytics import YOLO

# Load and run predictions with the first model
model1 = YOLO('path/to/your/first_model.pt')
results1 = model1.predict(source='path/to/your/dataset')

# Load and run predictions with the second model
model2 = YOLO('path/to/your/second_model.pt')
results2 = model2.predict(source='path/to/your/dataset')

# ...repeat for additional models

After you have the predictions, you'll need to implement the logic for combining them. This is not provided out-of-the-box with YOLOv8, so custom code is required.

Remember to consider the trade-offs when using ensemble methods, such as increased computational cost and complexity. Happy coding! 😊🚀

[abdallah1989203]

what is that meining optimizer:auto?

[nevercast]

That's the default optimizer and it means that YOLO will try to find the best learning rate and momentum at the beginning of each training run. If you are wanting to tune the learning rate and momentum (probably yes) you should add optimizer='AdamW' to your tune() arguments, like in the example. Otherwise, the Tuner values for learning rate and momentum will be ignored.

[pderrenger]

@nevercast the optimizer: auto setting in the YOLOv8 configuration allows the model to automatically select the best optimizer based on the training dataset and model architecture. This feature is designed to simplify the training process for users by handling the optimizer selection internally. However, if you're conducting hyperparameter tuning and wish to specify a particular optimizer, such as AdamW, you can override this setting by explicitly setting the optimizer parameter in your training script. This ensures that the optimizer you want to tune is used during the training process. 🛠️

[MariamMoawed]

How to save my yolov8 model, when I am currently working on gpu runtime And I need my model to attach it in a larger system?

[YawarGuguma]

the trained Yolo v8 models are automatically saved in the following folders depending upon the task
For detection
./runs/detect/train/weights
For classification
./runs/classify/train/weights
etc

You may also save the model weights in each epoch using the "save_period=1" in following statement in python
results = model.train(data='path/to/data/', epochs=100, save_period = 1)

[pderrenger]

@YawarGuguma that's correct! When training YOLOv8 models, the weights are automatically saved in the respective runs subdirectories based on the task. If you're working on a GPU runtime and need to save your model to integrate it into a larger system, you can specify the save_period parameter during training to ensure that your model weights are saved at the frequency you desire. For instance, setting save_period=1 will save weights after every epoch, which can be useful for long training sessions or when using cloud-based GPU resources where you might need to persist your progress frequently.

Here's a concise example of how to set this up:

from ultralytics import YOLO

# Initialize the model
model = YOLO('yolov8n.pt')

# Train the model with a custom save period
results = model.train(data='path/to/data.yaml', epochs=100, save_period=1)

After training, you can find your saved weights in the runs/train/exp*/weights directory, where exp* is the experiment name that may include a timestamp. You can then transfer these weights to your larger system as needed. 🚀

[scott-kim94]

I tried to learn the yolov8 object detection model using the ultralytics package.

I want to get predictions of each epoch checkpoint.
So, I input the train settings like that.

save=True,
save_period=1,
val=True,

But, the prediction images for validation dataset is not exposed to DEBUG SAMPLES in clearml.
How can get predictions of each epoch checkpoint ?

[YawarGuguma]

You manually need to give the path for each model to load it as follows
model = YOLO('./runs/detect/train/weights/epoch12.pt')

and then test it using the following
results = model(testing_frame, verbose=False, show=True)

[pderrenger]

@YawarGuguma to obtain predictions for each epoch checkpoint during training, you can use the predict mode after each epoch. However, this process is not automated within the training loop. You would need to manually load each checkpoint and run predictions as follows:

from ultralytics import YOLO

# After training, for each epoch's checkpoint
for epoch in range(1, total_epochs + 1):
    checkpoint_path = f'runs/train/exp/weights/epoch{epoch}.pt'  # Adjust the path as needed
    model = YOLO(checkpoint_path)  # Load the checkpoint
    results = model.predict(source='path/to/validation/data', save=True)  # Run prediction
    # Process results or integrate with your logging system here

Please note that this is a manual approach and you would need to integrate it with your training loop or run it after the training is complete. If you're using ClearML for experiment tracking, you would need to log the prediction results manually to ClearML. The save=True flag will save the prediction images, which you can then upload to ClearML as artifacts.

For a more automated approach, you might consider customizing the training loop or using a callback mechanism if available in your training environment. 🤖🛠️

[nlm-yuh5]

I want to set static learning rate for SGD, so I tried to set lrf=1.0, however, when tuning started, lrf still started with 0.01. What should I do instead? @glenn-jocher

model.tune(data=data_yaml, epochs=epoch, iterations=iterations, optimizer=optimizer_str, plots=False, save=False,
               val=False, batch=-1, imgsz=1024, lrf=1.0, dropout=0.5, rect=True, project=out_dst+str(folder_num)+'/')

[glenn-jocher]

@nlm-yuh5 the arguments passed are only used for the very first reference model, after that all arguments are changed or mutated based on the ranges allowed. See tune.py for more details.

[cheukhongip]

How do I resume training with model.tune() method after the training process crash?

[pderrenger]

@cheukhongip to resume hyperparameter tuning with the model.tune() method after a crash, you can use the resume argument. Set resume=True to automatically resume from the last checkpoint. Here's a concise example:

from ultralytics import YOLO

# Load your model
model = YOLO('yolov8n.pt')

# Resume hyperparameter tuning
model.tune(data='coco128.yaml', resume=True)

This will pick up the tuning process where it left off, using the saved checkpoints. Make sure that the checkpoint files from the previous tuning session are still available in the directory where they were saved. 🔄

[zfli-xdu]

Hello, is there any approaches to tune hyperparameters in mult-gpu mode, thereby speeding up the tuning process?

[pderrenger]

@zfli-xdu absolutely! YOLOv8 supports multi-GPU training out of the box, which can be utilized to speed up the hyperparameter tuning process. When using Ray Tune or any other hyperparameter optimization library, you can specify the number of GPUs to use for each trial. Here's a concise example of how you might set up Ray Tune to use multiple GPUs:

from ray import tune
from ultralytics import YOLO

def train_yolov8(config):
    # Initialize YOLOv8 with hyperparameters from the config
    model = YOLO('yolov8n.yaml', device='0,1')  # Example for 2 GPUs
    # Set hyperparameters
    model.hyp['lr0'] = config['lr0']
    model.hyp['momentum'] = config['momentum']
    # ... set other hyperparameters
    # Train the model
    model.train()

# Define the search space for hyperparameters
search_space = {
    'lr0': tune.loguniform(1e-4, 1e-1),
    'momentum': tune.uniform(0.8, 0.99),
    # ... define other hyperparameters
}

# Run the hyperparameter search
analysis = tune.run(
    train_yolov8,
    resources_per_trial={'gpu': 2},  # Specify the number of GPUs here
    config=search_space,
    num_samples=10,
    # ... other Ray Tune options
)

Make sure to adjust the device argument in the YOLO initialization to match the number of GPUs you want to use for each trial. Also, the resources_per_trial parameter in tune.run should be set to the appropriate number of GPUs.

Keep in mind that the actual implementation may vary based on your specific setup and the version of the libraries you are using. Always refer to the official documentation for the most accurate and up-to-date information. 🚀

[Githubprojectsacc]

Hello! I want to get the metric values like map score for object detection for hyper parameter tuning using yolov8 model. I used the same code the above for finetuning with ASHA and hyperopt search algorithm. In the training function I returned the evaluation score .But in the result I got the same metric values in every trial .Can u pls help me with this.

[pderrenger]

@Githubprojectsacc hello! It seems like you're on the right track with integrating hyperparameter tuning for your YOLOv8 model. If you're observing the same metric values across all trials, it could indicate that the training function might not be properly accessing or updating the model's state for each set of hyperparameters. Without seeing your exact implementation, here's a common approach to ensure your training function correctly returns varied evaluation scores for different hyperparameter sets:

from ultralytics import YOLO

def train_evaluate(config):
    # Initialize your model with config hyperparameters
    model = YOLO('path/to/model.yaml', device='cuda:0')  # Example device setup
    model.hyp.update(config)  # Update hyperparameters
    
    # Perform training here
    # ...
    
    # Perform validation and return the mAP score
    results = model.val()  # Make sure to use your validation data here
    return results.box.map  # Example of returning the mAP score
    
# Example hyperparameter space and training execution (Assuming you're using Ray Tune)
from ray import tune

config = {
    'lr0': tune.grid_search([0.01, 0.001, 0.0001]),
    'momentum': tune.grid_search([0.9, 0.95, 0.99]),
    # Add other hyperparameters here
}

analysis = tune.run(
    train_evaluate,
    # Add other Ray Tune settings here
    config=config,
)

Make sure your training and evaluation steps within train_evaluate function are properly defined and utilize the config hyperparameters. Your issue might also arise from incorrectly setting up the model for each trial, so double-check that model.hyp.update(config) effectively updates your model's hyperparameters.

If you're still facing issues, consider reviewing the hyperparameter update and evaluation part of your code for any mistakes. Happy tuning! 🚀

[Githubprojectsacc]

Thank you!@pderrenger
This is the code that I had implemented before.I got same metric value in each trial.have some issue in the triaining function i guess.

from ultralytics import YOLO

import os
import ray
from ray import tune
from ray import train
from ray.tune import ResultGrid
from ray.train import Result
from ray.tune.schedulers import ASHAScheduler
from ray.tune.search.hyperopt import HyperOptSearch

def model_train(config):
    model = YOLO("yolov8n.pt")
    data = '/content/drive/MyDrive/data.yaml'
    momentum=config["momentum"]
    batch=config["batch_size"]
    for epoch in range(1):
     trained_model = model.train(data=data,
                            epochs=1,
                            batch=batch,
                            workers=8,
                            imgsz=640,
                            patience=1,
                            momentum=momentum,
                          )
     eval=model.val()
     result=eval.results_dict
     metrics=eval.box.mp
     newmetric=momentum*2
     metricsnew=eval.box.map75
     train.report({"_metric":trained_model.box.mp})


ray.init(ignore_reinit_error=True)

search_space = {
    
    "batch_size": tune.choice([8, 16, 32]),
    "momentum": tune.loguniform(0.7, 0.98, 0.3)
    "batch_size": tune.choice([8, 16, 32]),
    "num_layers": tune.choice([3, 4, 5]),
    "activation": tune.choice(["relu", "tanh"]),
    "optimizer":tune.choice(['SGD', 'Adam', 'AdamW', 'NAdam', 'RAdam', 'RMSProp']),
}


search_alg = HyperOptSearch(metric="_metric")
num_samples=4

tuner = tune.Tuner(tune.with_resources(model_train,resources={"cpu":1,"gpu":1}),
    tune_config=tune.TuneConfig(
        metric="_metric",
        mode="min",
        search_alg=search_alg,
        num_samples=num_samples,
    ),
    param_space=search_space,
)
results = tuner.fit()

# Clean up Ray resources
ray.shutdown()

In this code I got the metrics value same for each trial the same precision score.I checked with other way of passing newmetric value which is the customized value that i given,whereas it shows different results.But there is some problem with the training method that I had implemented.The evaluation metrics are returning same for each trial.Can u pls guide me regarding this

[pderrenger]

Hello @Githubprojectsacc! Thanks for sharing your code. It looks like you're on the right track, but there might be a few tweaks needed to ensure that the hyperparameters are correctly influencing the training process in each trial.

One potential issue could be the way hyperparameters are being passed to the model.train() function. It's crucial to ensure that all hyperparameters defined in the config are actually being used during the training. Here's a slightly modified version of your model_train function to ensure hyperparameters are updated correctly:

from ultralytics import YOLO
import ray
from ray import tune

def model_train(config):
    # Initialize the model
    model = YOLO("yolov8n.pt")
    data = '/content/drive/MyDrive/data.yaml'

    # Update model hyperparameters from config
    model.hyp['momentum'] = config["momentum"]
    model.hyp['batch_size'] = config["batch_size"]
    model.hyp['optimizer'] = config["optimizer"]

    # Train the model
    trained_model = model.train(data=data, epochs=1, batch=config["batch_size"],
                                workers=8, imgsz=640, patience=1, momentum=config["momentum"])

    # Evaluate the model
    eval = model.val()
    metrics = eval.box.map  # Make sure to use the correct attribute for mAP

    # Report the metrics
    tune.report(mAP=metrics)

Make sure that model.hyp is the correct way to update hyperparameters in your version of the Ultralytics YOLO library. Sometimes, direct assignment (as shown above) might be necessary instead of passing them as arguments to model.train().

Also, ensure that your search space does not have duplicated keys as in your original snippet, and that all hyperparameters are correctly defined and used.

Give this a try and see if the results vary across different trials as expected! If the issue persists, it might be helpful to add more logging inside your training function to debug how hyperparameters are being applied in each trial. 🚀

[JeanN00B]

I am trying to set iterations = 50, but it doesn't change.. Idk if it is a problem with my instalation, or what happen so I'd like to change that value in the .py file but I haven´t find where to modify this.

[pderrenger]

@JeanN00B to adjust the number of iterations for training in YOLOv8, you typically wouldn't modify a .py file directly. Instead, you should set the number of epochs in your training script or command. The number of epochs determines how many times the model will iterate over the entire dataset.

Here's an example of how to set the number of epochs to 50 when initializing and training your model:

from ultralytics import YOLO

# Initialize YOLOv8 model with a pre-trained weight
model = YOLO('yolov8n.pt')

# Train the model with your custom dataset for 50 epochs
results = model.train(data='your_dataset.yaml', epochs=50)

If you're referring to the number of iterations within an epoch, that's determined by the size of your dataset and the batch size. Each epoch consists of number of images / batch size iterations. To change the number of iterations per epoch, you would adjust the batch size accordingly.

If you're using hyperparameter tuning with Ray Tune and want to set a specific number of iterations for the tuning process, you would configure that within the Ray Tune setup, not within the YOLOv8 model training code.

If you're still facing issues, please ensure that your installation is correct and that you're using the latest version of the YOLOv8 repository. If the problem persists, you can refer to the Ultralytics Docs for more detailed guidance on training and hyperparameter tuning.

[krialm]

I want to use yolov8 obb but in this code:
from ultralytics import YOLO

Initialize a model

model = YOLO('yolov8n-obb.pt')
appearse a error: [Errno 2] No such file or directory: 'yolov8n-obb.pt'

[yuanyuanpei7]

I wonder how can I calculate Dice and mIoU instead of mask mAP for segmentation? Who can help me realize that? Thanks.

[pderrenger]

@yuanyuanpei7 to calculate Dice and mIoU metrics for segmentation instead of mask mAP, you would need to modify the validation code to include these metrics. While YOLOv8's default validation metrics for segmentation tasks include mAP, integrating Dice and mIoU would involve custom coding.

You can start by implementing functions to calculate Dice and mIoU based on the ground truth masks and the predicted masks from the model. Then, during the validation phase, you would call these functions and pass the appropriate data.

Here's a high-level overview of how you might approach this:

1. Dice Coefficient (F1 Score): The Dice coefficient can be calculated as 2 * (intersection of predicted and ground truth masks) / (sum of predicted mask + sum of ground truth mask). You'll need to iterate over the batch of predictions and ground truth masks to calculate this metric for each image.

2. Mean Intersection over Union (mIoU): mIoU is the average IoU across all classes. IoU is calculated as (intersection of predicted and ground truth masks) / (union of predicted and ground truth masks). You'll need to calculate IoU for each class and then average them.

After implementing these calculations, integrate them into the validation loop where the model's predictions are compared to the ground truth. You can then log these metrics or print them out as needed.

Please note that this is a non-trivial task that requires a good understanding of the model's output format and the dataset's annotation format. If you're not familiar with implementing these metrics, it might be helpful to look for existing implementations or seek assistance from someone with experience in this area. 🧑‍💻📊

[phyosan]

How to solve this error if the model = YOLO("yolov8s-seg.pt") runs offline?

Transferred 411/417 items from pretrained weights
TensorBoard: Start with 'tensorboard --logdir runs/segment/train33', view at http://localhost:6006/
Freezing layer 'model.22.dfl.conv.weight'
AMP: running Automatic Mixed Precision (AMP) checks with YOLOv8n...

RuntimeError Traceback (most recent call last)
Cell In[10], line 1
----> 1 results = model.train(data="cell_data.yaml",
2 epochs=2,
3 workers=1,
4 batch=8,
5 imgsz=640)

File ~/venv_example/lib/python3.8/site-packages/ultralytics/engine/model.py:338, in Model.train(self, trainer, **kwargs)
336 self.model = self.trainer.model
337 self.trainer.hub_session = self.session # attach optional HUB session
--> 338 self.trainer.train()
339 # Update model and cfg after training
340 if RANK in (-1, 0):

File ~/venv_example/lib/python3.8/site-packages/ultralytics/engine/trainer.py:190, in BaseTrainer.train(self)
187 ddp_cleanup(self, str(file))
189 else:
--> 190 self._do_train(world_size)

File ~/venv_example/lib/python3.8/site-packages/ultralytics/engine/trainer.py:286, in BaseTrainer._do_train(self, world_size)
284 if world_size > 1:
285 self._setup_ddp(world_size)
--> 286 self._setup_train(world_size)
288 self.epoch_time = None
289 self.epoch_time_start = time.time()

File ~/venv_example/lib/python3.8/site-packages/ultralytics/engine/trainer.py:232, in BaseTrainer._setup_train(self, world_size)
230 if self.amp and RANK in (-1, 0): # Single-GPU and DDP
231 callbacks_backup = callbacks.default_callbacks.copy() # backup callbacks as check_amp() resets them
--> 232 self.amp = torch.tensor(check_amp(self.model), device=self.device)
233 callbacks.default_callbacks = callbacks_backup # restore callbacks
234 if RANK > -1 and world_size > 1: # DDP

File ~/venv_example/lib/python3.8/site-packages/ultralytics/utils/checks.py:606, in check_amp(model)
604 try:
605 from ultralytics import YOLO
--> 606 assert amp_allclose(YOLO('yolov8n.pt'), im)
607 LOGGER.info(f'{prefix}checks passed ✅')

[pderrenger]

@phyosan it appears that you're encountering a RuntimeError during the training of a YOLOv8 segmentation model with AMP (Automatic Mixed Precision). This error might be related to the AMP checks that are performed before training starts.

Here are a few steps you can take to troubleshoot and potentially solve this issue:

1. Check PyTorch and CUDA Versions: Ensure that your PyTorch installation is compatible with CUDA and that both are up to date.

2. Verify AMP Installation: Make sure that you have the necessary AMP support installed. If you're using PyTorch 1.6 or later, AMP is included with PyTorch.

3. Model Weights: Confirm that the yolov8s-seg.pt weights are correctly located in your working directory and that they are not corrupted.

4. Data File: Double-check the cell_data.yaml file for any issues with the dataset paths or format.

5. Environment: If you're running offline, ensure that all required dependencies are installed and that there are no missing packages that might be causing the error.

6. Freezing Layers: The error log shows that a layer is being frozen. If this is not intentional, review your configuration to ensure that layers are being frozen correctly.

7. AMP Checks: The error occurs during AMP checks. You might want to temporarily disable AMP to see if the training starts without it. You can do this by setting amp=False in your training configuration.

8. Error Log: The error log provided is incomplete. Look for the full error message and stack trace to get more context on what might be causing the issue.

9. Update YOLOv8: Ensure that you are using the latest version of YOLOv8, as updates may contain bug fixes that could resolve your issue.

10. Resource Availability: Check that your system has enough memory and that the GPU is available for training.

If after trying these steps the issue persists, consider reaching out on the GitHub issues page with detailed information about your setup, including the full error log, system specifications, and the steps you've taken to resolve the issue. This will help the community to assist you more effectively. Remember to keep your communication concise and focused on the issue at hand. 😊🛠️

[minwim]

how to train own daset on windows using multi-gpu, I can train with multi-gpu on linux ,but windows can be error, how to do? it is necessary to me ?

[pderrenger]

@minwim training on Windows using multiple GPUs can sometimes be more challenging due to differences in how Windows handles multiprocessing and CUDA compared to Linux. However, YOLOv8 supports multi-GPU training on Windows as well.

Here are some steps you can follow to train on Windows with multiple GPUs:

1. Environment Setup: Ensure you have the latest NVIDIA drivers, CUDA toolkit, and PyTorch installed that are compatible with your GPUs.

2. Data Preparation: Prepare your custom dataset in the required format. You should have a YAML file that points to your train and validation data, as well as the class names.

3. Training Command: Use the train mode with the --device flag to specify the GPUs you want to use. For example, if you want to use GPU 0 and GPU 1, you would use --device 0,1.

4. Troubleshooting: If you encounter errors, make sure to check the error messages. Common issues on Windows include problems with file paths (use double backslashes \\ or raw strings r'path'), and sometimes issues with CUDA and PyTorch compatibility.

5. Distributed Data Parallel (DDP): YOLOv8 supports DDP for efficient multi-GPU training. Ensure that you're using a version of PyTorch that supports DDP on Windows.

If you continue to face issues, it might be helpful to provide specific error messages or descriptions of the problems you're encountering. This way, the community can offer more targeted advice. Remember that while multi-GPU training is supported, the exact steps may vary depending on your system configuration and the versions of the software you're using.

[unrue]

Hi,

I read such very interesting discussion. Someone could explain how to resume a Yolov8 tuning using Ultralytics Tuner and also RaY Tuner? Thanks.

[pderrenger]

@unrue hello! To resume hyperparameter tuning in YOLOv8 using Ultralytics Tuner and Ray Tune, you can use the resume argument. When you start a tuning session, Ray Tune automatically saves the progress in a directory. To resume, you simply need to set resume=True or resume='AUTO' when initializing the tuning process. This will prompt Ray Tune to look for the last checkpoint and continue from there.

Here's a concise example of how you might do this:

from ultralytics import YOLO, Tuner

# Load your YOLOv8 model
model = YOLO('yolov8n.pt')

# Define your hyperparameter search space
search_space = {
    'lr0': (0.001, 0.01),
    'weight_decay': (0.0001, 0.001),
    # ... other hyperparameters
}

# Initialize the tuner with Ray Tune
tuner = Tuner(ray_backend='ray', search_space=search_space)

# Start the tuning process, resuming if possible
tuner.tune(data='coco128.yaml', epochs=5, resume='AUTO')

Make sure your environment is set up with the necessary dependencies and that you have the latest version of YOLOv8 and Ray Tune installed. For more detailed guidance, please refer to the Ultralytics Docs on hyperparameter tuning. 😊🚀

[unrue]

How to pass best_hyperparameters.yaml to yolov8 once tuning process is finished?

[pderrenger]

@unrue once you've completed the hyperparameter tuning process and have your best_hyperparameters.yaml file, you can pass it to YOLOv8 by using the --hyp argument when initiating training. Here's a concise example of how to do this:

from ultralytics import YOLO

# Load your model
model = YOLO('yolov8n.pt')

# Train your model with the best hyperparameters
results = model.train(data='coco128.yaml', epochs=3, hyp='best_hyperparameters.yaml')

This will start the training process using the optimized hyperparameters from your best_hyperparameters.yaml file. Happy training! 😊🚀

[billpsomas]

Hello, this does not seem to work for me.

SyntaxError: 'hyp' is not a valid YOLO argument.

Any ideas?

[glenn-jocher]

Hello! It looks like there might be a mix-up with the argument syntax. For YOLOv8, hyperparameters are typically set during training by specifying the path to the .yaml file directly in the train() function without using a hyp argument. If you're using the CLI, ensure you're using the correct argument format. If the issue persists, could you please share the exact command or code snippet you're using? This will help us troubleshoot the problem more effectively. 🛠️

[billpsomas]

Hello again! I found a workaround solution:

with open(args.hyps, 'r') as file:
    hyps = yaml.safe_load(file)        
model.train(data=args.data, batch=args.batch, epochs=args.epochs, imgsz=args.imgsz, device=args.device, name=args.name, optimizer='AdamW', **hyps)

This works fine :)

[glenn-jocher]

@billpsomas hello! I'm glad to hear you found a workaround that works for you. Your approach to load the hyperparameters from the YAML file and unpack them into the train() function is indeed a good solution. If you have any more questions or need further assistance, feel free to reach out. Happy training with YOLOv8! 😊👍

[sorooshsorkhani]

Inconsistent Tracking with YOLOv8: Seeking Solutions

I'm using YOLOv8 for object counting and encountering inconsistent tracking results. Here's my setup:

model.track(source_video_path, imgsz=1280, classes=CLASS_ID, stream=True, vid_stride=2, persist=True)

Issue: When running the same code on the same video, the tracker_id assignment varies. Sometimes it's perfect, with each object getting a unique tracker_id. Other times, the same object receives multiple IDs, or an ID is assigned to multiple objects. This inconsistency persists across both botsort.yaml and bytetrack.yaml trackers.

Key Questions:

1. How can I ensure each object receives a unique tracker_id?
2. What steps can I take to achieve consistent tracking results?

Any insights or suggestions to address these issues would be greatly appreciated!

[pderrenger]

@sorooshsorkhani it sounds like you're experiencing variability in tracking performance, which can be influenced by several factors, including the choice of tracker, object appearance, occlusions, and lighting conditions. Here are some steps you can take to improve consistency:

1. Tune Tracker Hyperparameters: Adjust the hyperparameters in your tracker configuration file (botsort.yaml or bytetrack.yaml). Parameters such as iou_threshold, time_window, and others can significantly impact tracking performance.

2. Data Quality: Ensure your input video is of high quality, with stable lighting and minimal occlusions. Poor video quality can lead to inconsistent tracking.

3. Model Confidence: Experiment with the conf parameter to filter detections by confidence score. A higher threshold may reduce false positives but could also miss some detections.

4. Frame Rate: The vid_stride parameter skips frames during tracking. A lower stride (closer to 1) may provide more data for the tracker and improve consistency.

5. Persistence: The persist parameter, when set to True, will continue tracking objects even when they are not detected in some frames. Adjusting this may help with objects that temporarily leave the frame.

6. Post-Processing: Implement post-processing steps to handle tracker ID switches or merges. This could involve analyzing the trajectory and appearance of objects to correct inconsistencies.

7. Custom Tracker: If the provided trackers don't meet your needs, consider implementing a custom tracker or integrating a different one that might be more suitable for your specific use case.

8. Debugging: Use visualization tools to understand when and why tracker IDs are changing. This can help you identify patterns and adjust your approach accordingly.

Remember that tracking is a complex task, and perfect consistency is challenging to achieve, especially in dynamic environments. However, by fine-tuning your setup and considering the above points, you can work towards more stable results. If you continue to face issues, feel free to share more details on the Ultralytics YOLOv8 Docs or raise an issue on the repository for further assistance. 🛠️🔍

[aayush105]

I have got the multiple loss like train/val/box_loss, train/val/cls_loss, train/val/dlf_loss and among how to fine the actual train/val loss to plot the graph and also got precision, recall, mAP50(B) and metrics/mAP50-95(B) so how can i know the accuracy of my model i.e train/val accuracy?

[pderrenger]

@aayush105 in YOLOv8, the overall training and validation loss is typically not represented by a single value, as the loss is composed of multiple components such as box loss (box_loss), class loss (cls_loss), and DFL loss (dfl_loss). To visualize the training process, you can plot these individual loss components over epochs to understand how each is contributing to the total loss and how they are decreasing over time.

For accuracy, YOLOv8 uses mean Average Precision (mAP) as a metric for object detection tasks. The mAP50 (B) metric represents the mAP at an Intersection Over Union (IoU) threshold of 0.5, which is a common threshold for evaluating object detectors. The metrics/mAP50-95(B) represents the average mAP calculated over multiple IoU thresholds from 0.5 to 0.95 (inclusive) in steps of 0.05. This gives a more comprehensive view of the model's performance across different levels of IoU.

To assess the accuracy of your model, you should look at the mAP metrics. Higher mAP values indicate better accuracy in terms of the model's ability to correctly detect objects with precise bounding boxes. Precision and recall are also valuable metrics, with precision indicating the proportion of positive identifications that were actually correct, and recall indicating the proportion of actual positives that were identified correctly.

To plot the graph, you can use the precision, recall, and mAP values logged during training and validation. These metrics are more informative for object detection tasks than a single "accuracy" value, as they provide a nuanced view of the model's performance.

[aayush105]

@pderrenger Thank you for the clarification.

[pderrenger]

You're welcome! If you need further assistance or have more questions, feel free to reach out. Happy to help! 😊🚀

[war1u5]

is it possible to run this model on the jetson nano? i have custom trained the yolov8n according to my scenario and i would like to run the resulting .pt file on the jetson. any idea on where to start, requirements etc? thank you

[pderrenger]

@war1u5 absolutely, you can run YOLOv8 models on the Jetson Nano. To get started, you'll need to ensure that your Jetson Nano is set up with the necessary dependencies, such as PyTorch and torchvision compatible with the Jetson's ARM architecture. Here's a high-level overview of the steps:

1. Setup Jetson Nano: Make sure you have the latest JetPack SDK installed, which includes CUDA, cuDNN, and TensorRT optimized for the Jetson Nano.

2. Install PyTorch: Install a version of PyTorch that is compatible with the Jetson Nano's ARM architecture. NVIDIA provides wheels for this.

3. Install Ultralytics YOLOv8: Clone the YOLOv8 repository and install the required Python packages.

4. Optimize Model: Convert your .pt file to an optimized format like ONNX, and then further to TensorRT for better performance on the Jetson Nano.

5. Inference: Run inference using the optimized model. You may need to adjust the input size and batch size to fit the Nano's memory constraints.

6. Monitor Performance: Keep an eye on the memory and processing load. You might need to fine-tune the model size or input resolution for optimal performance.

Remember to check the documentation for detailed instructions on exporting models and running inference on edge devices. If you encounter any issues or have further questions, feel free to reach out to the community for support. Good luck with your deployment on the Jetson Nano! 🚀🤖

[belafdil-chakib]

Is there a reason for no dropout tuning ?

[pderrenger]

@belafdil-chakib dropout is a regularization technique used to prevent overfitting in neural networks by randomly setting a fraction of input units to 0 at each update during training time. In the context of YOLOv8, dropout is not a hyperparameter that is commonly tuned. This is because YOLO architectures, including YOLOv8, are generally robust to overfitting due to their design and the large datasets they are typically trained on.

However, if you find that your model is overfitting, you can manually introduce dropout layers into your model's architecture and then proceed to tune the dropout rate. This would involve modifying the model configuration file (*.yaml) to include dropout layers at appropriate points in the network, and then experimenting with different dropout values to see their effect on model performance.

Keep in mind that adding dropout to a model like YOLOv8, which is already quite complex and deep, may not always lead to better performance and could potentially degrade the detection quality if not used judiciously. 🧐

[haqueshenas]

Thank you for developing such a valuable tool. As a beginner, I have three questions:

1. How can I export or use the model's output, specifically the pixel data belonging to the segmented part, in my own code (e.g., for cropping or further processing of the segment in the original image)? What options are available in this case? Additionally, is there a way to prevent the code from printing the process and results in the terminal/console, especially when creating an .exe file using tools like PyInstaller or similar?

2. In the context of releasing an open-source freeware that includes YOLO model predictions, what are the copyright considerations? Should I include a version of the copyright notice obviously somewhere within the .exe file? Or is it sufficient to mention it once in the Python code?

3. After running the model, the images saved in the "predict" folder have lower resolutions compared to the input images. This issue doesn't occur when no class is detected in the image. Is there a way to retain the bounding box, label, and mask produced by YOLO on the image without any changes in image size, color quality, format, etc.?

Thanks.

[pderrenger]

@haqueshenas hello! I'm glad to hear that you're finding YOLOv8 useful. Let's address your questions:

1. Exporting Segmentation Data: When using YOLOv8 for segmentation tasks, you can access the pixel data of the segmented parts through the results object returned by the predict method. The results.masks.data attribute contains the mask data. To prevent printing to the console, you can redirect the output to os.devnull or adjust the logging level in your code. For example:

   import os
   import sys
   from ultralytics import YOLO
   
   # Suppress print statements
   sys.stdout = open(os.devnull, 'w')
   
   # Load model and perform prediction
   model = YOLO('yolov8-seg.pt')
   results = model.predict(source='image.jpg')
   
   # Access segmentation masks
   masks = results.masks.data  # This is a tensor with mask data
   
   # Restore print statements
   sys.stdout = sys.__stdout__

2. Copyright Considerations: When distributing open-source freeware that includes predictions from YOLO models, you should comply with the AGPL-3.0 license under which the models are released. This typically involves including the copyright notice and license in your software's documentation or about/help section. If you're embedding the software in a commercial product, consider obtaining an Enterprise License for more flexibility.

3. Retaining Original Image Quality: To retain the original image resolution and quality in the saved predictions, ensure that you're not resizing or altering the image within your code. YOLOv8's predict method should retain the original image size by default. If you're experiencing issues, check the imgsz parameter and make sure it matches the original image dimensions or is set to None. If the problem persists, please open an issue on the GitHub repository with a minimal reproducible example, and we'll look into it.

I hope this helps! If you have further questions or need assistance, feel free to reach out. 😊👍

[Enliqwe]

TypeError: 'NoneType' object does not support the context manager protocol I can't compile the pt file in engine? HELP!

[pderrenger]

@Enliqwe it seems like you're encountering an issue with a NoneType object when trying to compile a .pt file into an engine format, possibly for deployment. This error typically indicates that an expected object is None when it's being used within a with statement, which requires a context manager.

To assist you better, could you please provide more details on the exact command or code snippet you're using when this error occurs? Additionally, ensure that the file path to your .pt model is correct and that the file is not corrupted.

In the meantime, here's a general checklist to troubleshoot the issue:

1. Verify File Path: Ensure the path to your .pt file is correct and the file exists at that location.
2. Model Loading: Confirm that you're loading the model correctly using the YOLOv8 API.
3. Environment Setup: Make sure your Python environment has all the necessary dependencies installed and properly configured.
4. Context Manager: If you're using a custom script with a with statement, ensure that the object you're trying to use supports the context manager protocol.

If you continue to face issues, please provide the code snippet and the context in which the error occurs, and we'll be happy to help you further. 🛠️

[Anshuman-02905]

Hi I have a dataset resolution 1280x720. I want to do segmentation using YOLO v8. I am not able to do so any clues how to solve it.

[pderrenger]

@Anshuman-02905 for segmentation tasks using YOLOv8, you'll need to ensure that your dataset is properly formatted and that you're using a model configuration that supports segmentation. Here are some steps to help you get started:

1. Prepare Your Dataset: Make sure your dataset is in a format that YOLOv8 expects. For segmentation, this typically means having images and corresponding segmentation masks.

2. Select a Segmentation Model: Choose a YOLOv8 model that is designed for segmentation tasks. Models with a -seg suffix in their names are tailored for segmentation.

3. Modify Configuration: If necessary, adjust the model configuration file (YAML) to match your dataset's resolution. You can change the imgsz parameter to 1280, 720 to match your dataset resolution.

4. Train the Model: Use the train mode to train your model on the segmentation dataset. Make sure to specify the correct dataset and model configuration files.

5. Evaluate the Model: After training, use the val mode to evaluate your model's performance on the validation set.

Here's a concise example of how you might set up your training in Python:

from ultralytics import YOLO

# Load a segmentation model configuration
model = YOLO('yolov8n-seg.yaml')

# Train the model on your dataset
results = model.train(data='your_dataset.yaml', imgsz=[1280, 720], epochs=50)

Remember to replace 'yolov8n-seg.yaml' and 'your_dataset.yaml' with the actual paths to your model and dataset configuration files. If you encounter any specific errors during this process, please provide the error messages, and I'll be happy to assist you further. 🚀

[chandra-ps612]

I trained a model from the scratch using "yolov8n.yaml". Further, I want to tune hyperpararameters to get best performing model.
Will this code snippet be given below helpful to get desired results?

from ultralytics import YOLO
model = YOLO(checkpoint_path) # checkpoint_path= './best.pt'
model.tune(imgsz=416, data='custom_data.yaml', epochs=30, iterations=300, optimizer='SGD', plots=False, save=False, val=False)

[pderrenger]

@chandra-ps612 the code snippet you've provided is on the right track for hyperparameter tuning with YOLOv8. However, there are a few adjustments and considerations to make:

1. If you're starting from scratch, you should initialize the model with the YAML file rather than a checkpoint path. The checkpoint path is used to load a pre-trained model or resume training from a specific checkpoint.

2. The tune method is not a standard method in the YOLOv8 API. Instead, you would typically use the train method with specific hyperparameters. For integrating Ray Tune, you would need to set up a Ray Tune search space and run the tuning process outside of the standard YOLOv8 training pipeline.

3. Ensure that custom_data.yaml correctly points to your dataset and that the dataset is properly formatted for YOLOv8.

4. The imgsz, epochs, and optimizer parameters are correctly specified for training, but you'll need to define the hyperparameter search space for Ray Tune separately.

Here's a more appropriate way to start training a model from scratch with a YAML configuration:

from ultralytics import YOLO

# Initialize the model with the YAML file
model = YOLO('yolov8n.yaml')

# Train the model with your custom dataset
results = model.train(data='custom_data.yaml', epochs=30, batch=16, imgsz=416)

For hyperparameter tuning with Ray Tune, you would need to set up a Ray Tune experiment where you define the search space for the hyperparameters you want to optimize. You would then run the Ray Tune process, which would internally call the training routine with different hyperparameter combinations.

Please refer to the Ray Tune documentation and the YOLOv8 documentation for more details on setting up hyperparameter tuning experiments. 📘🚀

[MilenioScience]

i got error when i use this command, model.tune(data='coco8.yaml', epochs=30, iterations=300, optimizer='AdamW', plots=False, save=False, val=False)
ModuleNotFoundError: Tuning hyperparameters requires Ray Tune. Install with: pip install 'ray[tune]'
How to fix this?

[pderrenger]

@MilenioScience it looks like you're trying to use the hyperparameter tuning functionality in YOLOv8, which requires the Ray Tune library. To fix the ModuleNotFoundError, you need to install Ray Tune. You can do this by running the following command in your terminal:

pip install 'ray[tune]'

This will install Ray along with the Tune module, which is necessary for hyperparameter tuning. After installation, you should be able to run the model.tune() command without encountering the error. If you have any further issues or questions, feel free to check the Ultralytics Docs or ask here. Happy tuning! 😊🚀

[ClementFranger]

Hello, I have been trying to make the ultralytics ray tuning works, but to no avail...

Here is my code snipper :

yolo = YOLO(yolov8)
yolo.tune(data=os.path.join(dataset.location, "data.yaml"), use_ray=True)

I have multiples errors :

1. Every model except the nano one fails because it is too large for ray : The actor ImplicitFunc is too large (198 MiB >
   FUNCTION_SIZE_ERROR_THRESHOLD=95 MiB).
2. When switching to the nano model, i still run into this error :
   2024-03-28 00:06:33,495 ERROR tune_controller.py:1332 -- Trial task failed for trial _tune_6e994_00009 Traceback (most recent call last): File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\air\execution\_internal\event_manager.py", line 110, in resolve_future result = ray.get(future) ^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\_private\auto_init_hook.py", line 21, in auto_init_wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\_private\client_mode_hook.py", line 103, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\_private\worker.py", line 2667, in get values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\_private\worker.py", line 864, in get_objects raise value.as_instanceof_cause() ray.exceptions.RayTaskError(OSError): ray::ImplicitFunc.train() (pid=2888, ip=127.0.0.1, actor_id=b8d18b78f3c8a6510826493d01000000, repr=_tune) File "python\ray\_raylet.pyx", line 1889, in ray._raylet.execute_task File "python\ray\_raylet.pyx", line 1830, in ray._raylet.execute_task.function_executor File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\_private\function_manager.py", line 724, in actor_method_executor return method(__ray_actor, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\util\tracing\tracing_helper.py", line 467, in _resume_span return method(self, *_args, **_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\tune\trainable\trainable.py", line 334, in train raise skipped from exception_cause(skipped) File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\air\_internal\util.py", line 88, in run self._ret = self._target(*self._args, **self._kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\tune\trainable\function_trainable.py", line 53, in <lambda> training_func=lambda: self._trainable_func(self.config), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\util\tracing\tracing_helper.py", line 467, in _resume_span return method(self, *_args, **_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ray\tune\trainable\function_trainable.py", line 261, in _trainable_func output = fn() ^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ultralytics\utils\tuner.py", line 98, in _tune results = model.train(**config) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ultralytics\engine\model.py", line 336, in train self.trainer = (trainer or self._smart_load('trainer'))(overrides=args, _callbacks=self.callbacks) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ultralytics\engine\trainer.py", line 85, in __init__ self.device = select_device(self.args.device, self.args.batch) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ultralytics\utils\torch_utils.py", line 143, in select_device s += f'CPU ({get_cpu_info()})\n' ^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\ultralytics\utils\torch_utils.py", line 60, in get_cpu_info info = cpuinfo.get_cpu_info() # info dict ^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\cpuinfo\cpuinfo.py", line 2759, in get_cpu_info output = get_cpu_info_json() ^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\site-packages\cpuinfo\cpuinfo.py", line 2741, in get_cpu_info_json p1 = Popen(command, stdout=PIPE, stderr=PIPE, stdin=PIPE) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\subprocess.py", line 1026, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "C:\Users\Minifranger\miniconda3\envs\object-detection-dofus\Lib\subprocess.py", line 1538, in _execute_child hp, ht, pid, tid = _winapi.CreateProcess(executable, args, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ OSError: [WinError 87] Paramètre incorrect
3. When I do use_ray=False, i get the following error :
   UnboundLocalError: cannot access local variable 'ckpt_file' where it is not associated with a value

Any help would be appreciated...

[glenn-jocher]

Hey! 🌞

Sorry to hear that you've run into issues with Ray tuning. Let's try to resolve them step by step.

1. Model Size Issue with Ray: This is due to Ray's default limits. A quick workaround is to manually adjust FUNCTION_SIZE_ERROR_THRESHOLD in the Ray configuration or split your model into smaller parts if possible.

2. Error with Nano Model: This error seems a little more complex and specific to your system. Without full visibility into your setup, a common advice is to ensure your Ray environment is correctly set up and has sufficient resources. Additionally, verifying the integrity of your data and model paths might help.

3. Local Variable 'ckpt_file' Error: This appears to occur when use_ray=False is set, likely due to some misconfiguration in the tuning script. Double-checking the scope of ckpt_file or initializing it before use might resolve this issue.

Without more context, these suggestions are fairly general. If these steps don’t resolve the issue, could you share more details on your setup, or perhaps the complete stack trace for deeper insights?

Happy to help further!

[Githubprojectsacc]

Hello! I want to get the metric values like map score for object detection for hyper parameter tuning using yolov8 model. I used the same code the above for finetuning with ASHA and hyperopt search algorithm. In the training function I returned the evaluation score .But in the result I got the same metric values in every trial .Can u pls help me with this.

[glenn-jocher]

Hello! It sounds like you're experiencing an issue where each trial returns the same metric values during hyperparameter tuning with Ray Tune and YOLOv8. This could potentially be caused by not correctly updating the model's hyperparameters for each trial, or possibly an issue with how the evaluation scores are being captured and returned in your training function. 🤔

Without seeing your specific code, here's a general approach to ensure that the model's hyperparameters are correctly set for each Ray Tune trial. Ensure you're passing the updated configuration from Ray Tune's config dictionary to your model's training function:

from ray import tune

def training_function(config):
    # Example: Updating the learning rate for each trial
    lr = config["lr"]
    # Initialize your YOLOv8 model here and apply the learning rate `lr`
    # model = initialize_model(...)
    # model.set_lr(lr)
    
    # Proceed with training and evaluation, returning the evaluation score
    eval_score = model.train_and_evaluate()
    tune.report(eval_score=eval_score)

analysis = tune.run(
    training_function,
    config={
        # Define your hyperparameter search space here
        "lr": tune.grid_search([0.001, 0.0001, 0.00001]),
        # Include other hyperparameters you're tuning
    },
)

Ensure the model or function that's performing the training and evaluation is correctly utilizing the hyperparameters injected from the config dictionary. It's essential each trial runs with its unique set of hyperparameters for effective tuning.

If you're already following a similar pattern and still encountering issues, it may be helpful to verify the training and evaluation process—checking that the model is indeed training with the varied parameters and that the evaluation metric (like mAP score) is correctly calculated and returned after each trial.

Hope this helps to troubleshoot your issue! If you continue to face problems, providing a snippet of your training function could help diagnose further. Happy tuning! 🚀

[Githubprojectsacc]

Thank you!@glenn-jocher

from ultralytics import YOLO

import os
import ray
from ray import tune
from ray import train
from ray.tune import ResultGrid
from ray.train import Result
from ray.tune.schedulers import ASHAScheduler
from ray.tune.search.hyperopt import HyperOptSearch

def model_train(config):
    model = YOLO("yolov8n.pt")
    data = '/content/drive/MyDrive/data.yaml'
    momentum=config["momentum"]
    batch=config["batch_size"]
    for epoch in range(1):
     trained_model = model.train(data=data,
                            epochs=1,
                            batch=batch,
                            workers=8,
                            imgsz=640,
                            patience=1,
                            momentum=momentum,
                          )
     eval=model.val()
     result=eval.results_dict
     metrics=eval.box.mp
     newmetric=momentum*2
     metricsnew=eval.box.map75
     train.report({"_metric":trained_model.box.mp})


ray.init(ignore_reinit_error=True)

search_space = {
    
    "batch_size": tune.choice([8, 16, 32]),
    "momentum": tune.loguniform(0.7, 0.98, 0.3)
    "batch_size": tune.choice([8, 16, 32]),
    "num_layers": tune.choice([3, 4, 5]),
    "activation": tune.choice(["relu", "tanh"]),
    "optimizer":tune.choice(['SGD', 'Adam', 'AdamW', 'NAdam', 'RAdam', 'RMSProp']),
}


search_alg = HyperOptSearch(metric="_metric")
num_samples=4

tuner = tune.Tuner(tune.with_resources(model_train,resources={"cpu":1,"gpu":1}),
    tune_config=tune.TuneConfig(
        metric="_metric",
        mode="min",
        search_alg=search_alg,
        num_samples=num_samples,
    ),
    param_space=search_space,
)
results = tuner.fit()

# Clean up Ray resources
ray.shutdown()

In this code I got the metrics value same for each trial the same precision score.I checked with other way of passing newmetric value which is the customized value that i given,whereas it shows different results.But there is some problem with the training method that I had implemented.The evaluation metrics are returning same for each trial.Can u pls guide me regarding this

[glenn-jocher]

Hello! It looks like you're doing some fantastic work integrating YOLOv8 with Ray Tune for hyperparameter tuning. 👍 From your description, it seems the metric values remain constant across trials, hinting at a possible misconfiguration in the training process or metric reporting.

A quick check you might want to perform is ensuring that train.report() correctly reports varied metric values back to Ray Tune. Based on your code:

train.report({"_metric": trained_model.box.mp})

This line attempts to report a metric, but it should directly report the metric value you're interested in. For instance, if metrics or metricsnew is your evaluation metric, ensure to report that instead. Also, ensure that model.train() and model.val() modify the model's internal state or weights correctly for each trial. The model should be freshly initialized or reset at the start of each training function call to prevent state carryover between trials.

Also, there seems to be a duplicate entry in your search_space dictionary for "batch_size". This might cause unexpected behavior in defining your search space. Here's a corrected snippet:

search_space = {
    "batch_size": tune.choice([8, 16, 32]),
    "momentum": tune.loguniform(0.7, 0.98),
    "num_layers": tune.choice([3, 4, 5]),
    "activation": tune.choice(["relu", "tanh"]),
    "optimizer": tune.choice(['SGD', 'Adam', 'AdamW', 'NAdam', 'RAdam', 'RMSProp']),
}

Lastly, ensure your model.val() function fetches a fresh set of evaluation metrics after training, correctly reflecting the training's impact.

Give these adjustments a try, and hopefully, you should start seeing varying metric values across trials reflective of the changes in hyperparameters. Keep experimenting, and don't hesitate to reach out for more assistance! 🚀