Home Background Rules Schedule Registration Submission Results Download

Results

As of 30 November 2003, 33 teams registered for the competition, with most of them from academia and a few from industry. Of the 33 teams registered, 16 teams eventually submitted their programs for Task 1 while 13 teams for Task 2 by the deadline (31 December 2003). Some teams entered submissions for both tasks. One team, which participated in both tasks, submitted a program that requires a licensed software to run it. Eventually this team withdrew from the competition. So we ended up having a total of 15 teams for Task 1 and 12 teams for Task 2. All teams are from academia from nine different countries (Australia, China, France, Germany, Korea, Singapore, Spain, Turkey, and United States).

The testing protocol is as follows. Each program was evaluated on two signature data sets. The first data set, called the training set, consists of genuine signatures and skilled forgeries for 40 users. It was released to the participants for system building and evaluation prior to program submission. The second set, called the test set, consists of genuine signatures and skilled forgeries for 60 users. This set was not released to the participants. For each user from either the training set or the test set, 10 trials were run based on 10 different random subsets of five genuine signatures each from files S01-S10 for enrollment. After each enrollment trial, the program was evaluated on 10 genuine signatures (S11-S20), 20 skilled forgeries (S21-S40), and 20 random forgeries selected randomly from genuine signatures of 20 other users. Whenever randomness was involved, the same random sets were used for all teams for fairness. We report the equal error rates (EER) and ROC curves separately for skilled forgeries and random forgeries.

The programs of some teams encountered problems during the evaluation process. In particular, they failed to report similarity scores for some input signatures during the training and/or testing phase. For fairness of comparison, EER statistics and ROC curves are not reported for these programs. Besides reporting the average EER over all users and all 10 trials for each team, we also report the standard deviation, maximum and minimum EER values. For both tasks, Team 6 (i.e., 106 and 206), from the Sabanci University of Turkey, gives the lowest average EER values when tested with skilled forgeries.

EER statistics

Task 1
Task 2

ROC curves

Task 1
Task 2


Home | Background | Rules | Schedule | Registration | Submission | Results | Download

Copyright 2003 All rights reserved.