Video Super-Resolution Quality Assessment Challenge 2024

Participate

Here you can register team and send your own metric to take part in this challenge.

Dataset

  • Train Set (Download; 37.04Gb)
  • Public Test Set (Download; 7.35Gb)
  • Private Test Set (Download; 16.59Gb)
  • Full Dataset can be also found on Google Drive (Link)

Important! On June 3rd, we changed level distribution in all sets. If you’ve already downloaded the whole dataset, just change the videos_info.json files for all three sets. You can download these file separately at the link above.

Each of the sets is stored in the following format:

├── DistortedVideos
|   ├── <Video Group Name>
|   |   ├── <Distorted Video from Current Group>
|   |   ├── ...
|   ├── ...
├── GroundTruthVideos
|   ├── <Video Group Name>
|   |   ├── 000000.png
|   |   ├── ...
|   ├── ...
├── videos_info.json

videos_info.json is a JSON file of the following form:

{
    ...
    <Level Name ("easy", "medium" or "hard")> : {
        ...
        <Video Group Name> : {
            "distorted_videos" : {
                ...
                <Path to Distorted Video from the Current Group> : <Subjective Score for Train Set or 0.0 for Test Sets>,
                ...
            },
            "gt_frames" : <Path to the Frames of the Ground-Truth Video corresponding to the Current Group>
        }
        ...
    }
    ...
}

Submitting

For deep-learned solutions only working on Python are sought. Other solutions will be accepted but will not be officially ranked.

  1. Process the input videos;
  2. Apply the code and training checkpoints for neural network models;
  3. Create a JSON-file with dictionary in the following format:
    {
     ...
     <Video Group> : {
         ...
         <Path to Distorted Video from the Current Group> : <Metric Value>,
         ...
     }
     ...
    }
    


    For example, for video “DistortedVideos/animation/RealESRGAN_x4_libx264_bitrate=1000.mp4” from the “animation” group and metric value equal to 0.54 output should be:

    {
     "animation": {
         "DistortedVideos/animation/RealESRGAN_x4_libx264_bitrate=1000.mp4" : 0.54
     }
    }
    
  4. Send this file to the submission acceptance form by clicking on the “Upload File” button.

Here you can download a template of the file you need to upload (Download; 40.8Kb)

Fields Description

“Metric Type” field must contain metric type in the following format: “{FR/NR}_{Image/Video}”.

  • NR - for No-Reference Metric (don’t need GT frames for evaluation)

  • FR - for Full-Reference Metric (need GT frames for evaluation)

  • Image - if Metric requires image input

  • Video - if Metric requires video input

By clicking on the “Upload File” button, you can upload a file with the results of the metric on the test set.

Evaluation

The evaluation consists of the comparison of the predictions with the reference ground truth scores obtained by pairwise subjective comparison.

We use the Spearman rank-order correlation coefficients (SROCC) as often employed in the literature.

Its implementations are found in most of the statistics/machine learning toolboxes. For example, the demo evaluation code in Python:

import scipy

# SROCC 
srocc = scipy.stats.spearmanr(ground_truth_scores, metric_values).correlation

This coefficient will be computed separately for each video sequence (by sequence we mean all videos with Super-Resolution methods applied, corresponding to the same source video, corresponding to the same difficulty level).

The final score for a particular difficulty level is calculated as the average of these coefficients over all video sequences corresponding to that level.

The final metric score is equal to (0.3 * (score for “Easy” level) + 0.4 * (score for “Medium” level) + 0.5 * (score for “Hard” level)) / 1.2

Results are evaluated separately on the Test Public and Test Private Sets. The results on the Test Public Set are visible to you immediately after you check the upload. The results on the Test Private Set will appear at the end of the challenge.

The final result of the metric on the whole Test Set is evaluated as ((Result on the Test Public Set) + 2 * (Result on the Test Private Set)) / 3.

You can find the baseline method (Q-Align for IQA) results on the “Leaderboard” page.

15 Jul 2024