Terms and Conditions

Compressed Video Quality Assessment Challenge

These are the official rules (terms and conditions) that govern how the ECCV-AIM 2024 challenge on Compressed Video Quality Assessment Challenge will operate. This challenge will be simply referred to as the “challenge” or the “contest” throughout the remaining part of these rules and may be named as “ECCV-AIM” benchmark, challenge, or contest, elsewhere (our webpage, our documentation, other publications).

In these rules, “we”, “our”, and “us” refer to the organizers (compressed-vqa-challenge-2024@videoprocessing.ai: Maksim Smirnov, Anastasia Antsiferova, and Aleksandr Gushchin) of ECCV-AIM challenge and “you” and “yourself” refer to an eligible contest participant.

Note that these official rules can change during the contest until the start of the final phase. If at any point during the contest, the registered participant considers that can not anymore meet the eligibility criteria or does not agree with the changes in the official terms and conditions then it is the responsibility of the participant to send an email to the organizers such that to be removed from all the records. Once the contest is over no change is possible in the status of the registered participants and their entries.

Contest description

This is a skill-based contest and chance plays no part in the determination of the winner(s).

The goal of the contest is to predict the perceptual quality of an input video, tentatively compressed, and the challenge is called Compressed Video Quality Assessment Challenge.

Competition focus: a dataset customized to the specific needs of the competition will be provided. The videos are characterized by a broad coverage of compression artifacts. We will refer to this dataset, its section, and related materials as the ECCV-AIM Dataset. The dataset is divided into training, and test data (public and hidden). The goal is to achieve predictions with the best correlation with the ground-truth scores obtained from subjective video comparisons. Subjective quality scores were obtained by applying a Bradley-Terry model to the results of a pair-wise comparison of the dataset videos. The higher the score is the better is the quality. Participants will not have access to the ground-truth scores from the test data. Participants will be ranked according to the performance of their methods on the open and hidden test data. Before the final evaluation on the hidden test data participants must provide an archive with working method code written according to the template we provide (details on the “Participate” page). Participants should also provide information about the name and type of their method (Video/Image Quality Assessment, Full-Reference/No-Reference), team name, and members affilations.

Tentative contest schedule

The registered participants will be notified by email if any changes are made to the schedule. The schedule is available on the “Overview” page.

Eligibility

You are eligible to register and compete in this contest only if you meet all the following requirements:

• you are an individual or a team of people willing to contribute to the open tasks, who accept to follow the rules of this contest

• you are not an v challenge organizer or an employee of ECCV-AIM challenge organizers

• you are not involved in any part of the administration and execution of this contest

• you are not a first-degree relative, partner, household member of an employee or of an organizer of ECCV-AIM challenge or a person involved in any part of the administration and execution of this contest

This contest is void wherever it is prohibited by law.

NOTE: to submit a paper to the workshop the participants need to introduce some noticeable novelty compared to existing solutions.

Entry

In order to be eligible for judging, an entry must meet all the following requirements:

Entry contents: The participants are required to submit their method results on all videos from the open test set. Before the end of the challenge participants also have to submit the method script, weights checkpoint, and dockerfile to set up an environment. It will allow us to measure submission performance on the hidden test data. To be eligible for judgment participants should publicly release their code or executables under a license of their choice, taken among popular OSI-approved licenses, and make their code or executables online accessible for not less than one year following the end of the challenge (applies only for top ten ranked participants of the competition). This challenge will have a report paper published with AIM Workshop and in ECCV Workshop proceedings. All the participants are also invited (not mandatory) to submit a paper for peer-reviewing and publication at the ECCV Workshop.

Use of data provided: All data provided by ECCV-AIM are freely available to the participants from the website of the challenge under license terms provided with the data. The data are available only for open research and educational purposes, within the scope of the challenge. ECCV-AIM and the organizers make no warranties regarding the database, including but not limited to warranties of non-infringement or fitness for a particular purpose. The copyright of the images remains the property of their respective owners. By downloading and making use of the data, you accept full responsibility for using the data. You shall defend and indemnify ECCV-AIM and the organizers, including their employees, Trustees, officers, and agents, against any claims arising from your use of the data. You agree not to redistribute the data without this notice.

Hidden test data: The organizers will use the hidden test data for the final evaluation and ranking of the entries. It will not be made available to the participants during the contest. The hidden test data includes compression standards and codecs, which are not presented in the training data and the open test data.

Open test data: The open test data will be shared with the competitor along with the training data. It will be used to calculate the competitor intermediate rank during the challenge.

Training data: The organizers will make available to the participants the training dataset with ground-truth subjective scores.

Post-challenge analyses: the organizers may also perform additional post-challenge analyses using extra data, but without effect on the challenge ranking.

Submission: the entries will be online submitted via the videoprocessing.ai web platform. During the development phase, while the validation server is online, the participants will receive immediate feedback on the open test data. The final scores (on the hidden test data) will be released after the challenge is over.

Original work, permissions: In addition, by submitting your entry into this contest you confirm that to the best of your knowledge:

• your entry is your own original work

• your entry only includes material that you own, or that you have permission to use

Submission of entries

The participants will follow the instructions on the videoprocessing.ai website to submit entries (details on the “Participate” page)

The participants will be registered as mutually exclusive teams. Each team is allowed to submit only one single final entry. We are not responsible for entries that we do not receive for any reason, or for entries that we receive but do not work properly.

The participants must follow the instructions and the rules. We will automatically disqualify incomplete or invalid entries.

Judging the entries

We will be also the judges of the contest; all of us are forbidden to enter the contest and are experts in causality, statistics, machine learning, computer vision, or a related field. We will review all eligible entries received and select (three) winners based on the prediction score on test data. We will verify that the winners complied with the rules, including that they documented their method by filling out a fact sheet.

Our decisions are final and binding. In the event of a tie between any eligible entries, the tie will be broken by giving preference to the earliest submission, using the time stamp of the submission platform.