Test set scores

Here you can download JSON file with subjectives scores of videos from the test set (Download; 105Kb)

File with subjective scores is a JSON-file of the following form:

├── <Level Name>
|   ├── dist_videos
|   |   ...
|   |   ├── path
|   |   |   ├── <Path to distorted video>
|   |   ├── subjective_score
|   |   |   ├── <Subjective score of the current video>
|   |   ...
|   ├── group
|   |   ├── <Video sequence name>
├── ...

Participate

Challenge has been finished. Server is inactive now.

Dataset

• Train Set (Download; 37.04Gb)
• Public Test Set (Download; 7.35Gb)
• Private Test Set (Download; 16.59Gb)
• Full Dataset can be also found on Google Drive (Link)

Important! On June 3rd, we changed level distribution in all sets. If you’ve already downloaded the whole dataset, just change the videos_info.json files for all three sets. You can download these file separately at the link above.

Each of the sets is stored in the following format:

├── DistortedVideos
|   ├── <Video Group Name>
|   |   ├── <Distorted Video from Current Group>
|   |   ├── ...
|   ├── ...
├── GroundTruthVideos
|   ├── <Video Group Name>
|   |   ├── 000000.png
|   |   ├── ...
|   ├── ...
├── videos_info.json

videos_info.json is a JSON file of the following form:

{
    ...
    <Level Name ("easy", "medium" or "hard")> : {
        ...
        <Video Group Name> : {
            "distorted_videos" : {
                ...
                <Path to Distorted Video from the Current Group> : <Subjective Score for Train Set or 0.0 for Test Sets>,
                ...
            },
            "gt_frames" : <Path to the Frames of the Ground-Truth Video corresponding to the Current Group>
        }
        ...
    }
    ...
}

Submitting

For deep-learned solutions only working on Python are sought. Other solutions will be accepted but will not be officially ranked.

1. Process the input videos;
2. Apply the code and training checkpoints for neural network models;
3. Create a JSON-file with dictionary in the following format:

   {
    ...
    <Video Group> : {
        ...
        <Path to Distorted Video from the Current Group> : <Metric Value>,
        ...
    }
    ...
   }
   

   
   For example, for video “DistortedVideos/animation/RealESRGAN_x4_libx264_bitrate=1000.mp4” from the “animation” group and metric value equal to 0.54 output should be:

   {
    "animation": {
        "DistortedVideos/animation/RealESRGAN_x4_libx264_bitrate=1000.mp4" : 0.54
    }
   }
   

4. Send this file to the submission acceptance form by clicking on the “Upload File” button.

Here you can download a template of the file you need to upload (Download; 40.8Kb)

Fields Description

“Metric Type” field must contain metric type in the following format: “{FR/NR}_{Image/Video}”.

• NR - for No-Reference Metric (don’t need GT frames for evaluation)

• FR - for Full-Reference Metric (need GT frames for evaluation)

• Image - if Metric requires image input

• Video - if Metric requires video input

By clicking on the “Upload File” button, you can upload a file with the results of the metric on the test set.

Evaluation

The evaluation consists of the comparison of the predictions with the reference ground truth scores obtained by pairwise subjective comparison.

We use the Spearman rank-order correlation coefficients (SROCC) as often employed in the literature.

Its implementations are found in most of the statistics/machine learning toolboxes. For example, the demo evaluation code in Python:

import scipy

# SROCC 
srocc = scipy.stats.spearmanr(ground_truth_scores, metric_values).correlation

This coefficient will be computed separately for each video sequence (by sequence we mean all videos with Super-Resolution methods applied, corresponding to the same source video, corresponding to the same difficulty level).

The final score for a particular difficulty level is calculated as the average of these coefficients over all video sequences corresponding to that level.

The final metric score is equal to (0.3 * (score for “Easy” level) + 0.4 * (score for “Medium” level) + 0.5 * (score for “Hard” level)) / 1.2

Results are evaluated separately on the Test Public and Test Private Sets. The results on the Test Public Set are visible to you immediately after you check the upload. The results on the Test Private Set will appear at the end of the challenge.

The final result of the metric on the whole Test Set is evaluated as ((Result on the Test Public Set) + 2 * (Result on the Test Private Set)) / 3.

You can find the baseline method (Q-Align for IQA) results on the “Leaderboard” page.