DAS-Discussion: Datasets and Benchmarks

Back to [[DAS-Discussion:Index]]
{| style="width: 100%"
|-
| align="right" |

{|
|-
| {{Last updated}}
|}

|}
== DAS Working Subgroup Meeting: Datasets and Benchmarks ==
Authors:
* Dimos Karatzas (Moderator) – CVC Barcelona – IAPR TC-11 Data Curator
* Bart Lamiroy (Secretary) – CSE Lehigh University – IAPR TC-10 Data Curator

Further Participants:
* Henry Baird – CSE Lehigh University
* Igor Filippov – SAIC-Frederick, Inc.
* Albert Gordo – CVC Barcelona
* Dan Lopresti – CSE Lehigh University – IAPR TC-11 Chair
* Marcal Rossinyol – CVC Barcelona

=== Context === 
Availability, use and dissemination of benchmarks, datasets and ground truths in order to promote
subjective and reproducible assessment of document analysis methods, collaboration and exchange of
research results in the document analysis domain.
The IAPR TC-11 has launched the initiative to strongly encourage participants at the DAS 2010
workshop to contribute and publish their used datasets.
It is generally admitted that global and openly accessible benchmarking resources and datasets can be
very beneficial to research communities, and that the document analysis domain is lagging behind in
this domain compared to others.

=== Challenges and Main Issues ===
Initiatives of this kind have existed before and seem to be regularly recurring over the decades (cf.
introductory keynote talk by G. Nagy) not all are successful, some have a significant impact of the
community, all decline, disappear or become obsolete over the years.
The main challenges related to benchmark, dataset and general resource dissemination can be reduced
to :
1. guarantee sustainable access over time,
2. keep « entry cost » as low as possible,
3. keep them adapted to the technology advances, application needs, scientific focus shifts,
4. avoid lock-in syndromes by using open and adaptable representation standards and not
restricting new and creative uses that were not initially intended.

=== Potential Community Benefits and Uses ===
1. Access to reference datasets allows for
* fast prototyping and testing of one's own methods and algorithms.
* coherent and objective peer review of presented results
* reproduction and sharing of results
2. Access to reference algorithms allows for
* open peer-based performance evaluation
* self-assessment of one's own prototypes
The fact of guaranteeing an open, permanent and sustained access adds the possibility to allow for
continuously running contests or performance evaluation.

=== Pitfalls and Risks ===
The design of above mentioned repositories should remain in phase with the needs and the usages of
the community it is targeting. The most common pitfall is that its design coerces or limits the usage it
can have, for instance by imposing formats, limits, and constraints that could exclude potential users
for taking advantage of it or sharing their results.
On the other hand, if the repository is just a « bag of stuff » it provides no significant added value to its
users and will fail its mission.

== Report of Discussions ==
Given the previous context this section describes the discussions and exchanges that have taken place
during the DAS 2010 dedicated workshop sessions.

=== Dataset Coherence ===
* What to do with the datasets, and how make their access coherent ?
* Given the very wide variety of dataset contributions, how to guarantee the quality and their global usefulness to the community? Should contributions be peer-reviewed before been put on-line.
* How should contributions be referenced, indexed or annotated so that potential users can
rapidly and efficiently find what they need ?
* Should there be a specific submission protocol and form ? Is the existing form sufficient ?

=== What is Data? ===
Data that might prove useful for the document analysis community consists of
* Document Images
* Annex data (annotations, meta-data, acquisition conditions, task context ...)
* Ground truth (which is a specific case of the next point)
* Results from algorithms
* Programs-algorithms
* Papers
as well as all the links that can be expressed between them, (essentially provenance and generation
hierarchies of data produced by algorithms on data produced by other algorithms ...)

=== Specific Uses and Needs ===
Besides providing a repository for datasets and resources, such a platform can also offer higher level
services to the document analysis community, like, for instance hosting long standing contests, formal
tasks definitions for benchmarking and comparing methods and algorithms, capitalizing and
encouraging creative an new uses of common referencing and benchmarking resources.
To that extent it is important that the maintainers of such a platform make it as open as possible and
allow for inputs, suggestions and adaptations of the user community to take into account a wide range
of services.

=== Incentives ===
There's a general consensus that repositories and community resources are only useful when there is a
continuous contribution, correction, evaluation process that keeps it in line with the needs and
expectations of its potential users. This means that contributors should have an as great as possible
interest to participate in this process. Ideas of achieving this may consist in having a way of referencing
the datasets in a non-ambiguous way so that due credit is given (in the same way articles are cited, for
instance), provide statistics of “most used”, “most referenced”, “best regarded” contributions.
This will both require a stable repository, a flexible reference annotation, a way of publicly rank, rate or
review datasets as well as reviewing, ranking or evaluating the reviewers themselves.

=== Copyright, Intellectual Property, Sensitive and Personal Data ===
History has shown that copyright and intellectual property issues may significantly hamper and limit
uses and dissemination. The fact is, however, that correctly assessing the copyright of the data that is
commonly used in the Document Analysis community may be extremely complex and perhaps even
impossible in certain cases.
It is obvious that experimental data can be classified in three main levels:
* Open copyright or public domain data (either coming from clearly identifiable sources or of which the contributer is the copyright holder and who releases this right to the community). This includes Scientific Commons, copyleft, GPL or other similar licenses.
In order to avoid complex situations, it is probably a good idea to incite contributors to
release their data under a clearly specified license.
* Third party owned data is data that clearly cannot be distributed.
* Most of the data is very likely to be of blurry origin. Most certainly because either its origin has been lost, because the copyright holders cannot be clearly identified, or because the data is very unlikely to create litigation.
The bottom line is, unfortunately, that a non-profit research community does not have the resources of
legally establishing even a reasonable guarantee of origin of copyright or a risk assessment on
interesting and useful, yet possibly problematic data contributions.
It seems, however, that the Millennium Act offers academia the possibility to host and distribute data in
a legal safe way, as long as there is no clear intent to infringe on the copyright law, that the contributers
are knowingly responsible for its publication (and are identifiable) and that the platform has a
mechanism allowing legitimate copyright holders to signal litigious material and that the publishers
offer a means to efficiently and rapidly remove this material (cf. Youtube-like platforms).
It is not clear, however what happens with data that is distributable from a copyright point of view, but
that contains personal or sensitive information. While it is clear, from an ethical point of view that these
data should not be distributed, it is however less obvious as how to detect and handle situations where
it find its way in our repositories.

=== Data Persistence and Dependency ===
The need for taking into account possible copyright litigation and subsequent needs of removing data
from the repository calls for a very careful study of data persistence and dependency. Since successful
datasets will be largely used by the community they are likely to produce “derived products” in the
form of ground truths, analysis results, etc. In case of copyright litigation it will be necessary to retrace
the provenance of all derived or dependent data as to correctly remove the concerned data, without
affecting legitimate data.
This situation may extend to versioning and error-correction of dynamic datasets. Interdependence of
datasets can furthermore be considered with respect to, remote or partial hosting of parent data.

=== Operational issues ===
Submission process
Maintenance burden
Noise management (spurious, silly or malevolent contributions to the repository)
Guaranteeing data quality:
* reviewing/certifying
* starring
* reviewing or starring reviewers

=== About the Lehigh DAE platform ===
The DAE platform currently developed at the CSE department of Lehigh University
(http://dae.cse.lehigh.edu) offers a framework that looks very promising and seems to potentially
address all issues that have been discussed. It uses a open and document analysis centered data model
to represent data in a very flexible way, allows non-static notions of datasets, browsing and querying
data inter-dependence and provenance etc. We therefore suggest that the TC-10 and TC-11 data
repository initiatives experiment with the DAE platform for hosting their datasets.

== Conclusions and Concrete Actions ==
What specific actions can be sponsored, supported or done by the TC-11 ?
First and foremost, we feel that publicly available quality datasets are needed and beneficial for the
community. It therefore is our opinion that the TC-11 (and IAPR TCs in general) should sponsor and
promote make these datasets available. Possible incentives are:
* provide a way to reference and cite datasets with due credit to their authors
* make publishing and contributing dataset a reasonably hard requirement for article review and acceptation. In order to receive TC-11 sponsorship events could, for instance, require that authors publish their datasets to a public repository under an appropriate license, or clearly state why this is not possible in their context (e.g. sensitive personal data in medical document analysis)
* reward most cited or best reviewed datasets at major bi-annual international events.
Of course providing resources like datasets raises quite some legal questions. While it seems quite
obvious that our community does not have the resources to do an exhaustive legal study of this
question, it seems reasonably safe to assume that under the Millennium act, data hosted that is
contributed by identified users having acknowledged a copyright notice and/or provided data under an
appropriate license (e.g. but not exclusively, Creative Commons – http://creativecommons.org/). Help
from volunteers that can share knowledge in this domain should be called upon.
An open resource creates the risk of containing “noisy” data. TC-11 should therefor sponsor and
encourage a posteriori review of these data. This can, for instance, be done by providing a “starring”
mechanism, or a more extensive web 2.0 review mechanism, similar to community websites where
reviewers are evaluated on the quality of their reviews and where reviewer ranking can provide a
recognition within the community.
Last but not least, it is our conviction that the TCs should play a leading role in promoting and inciting
for the use of collectively shared resources, but that it is not the TCs (and TC-11 in particular) role to
decide how these resources are actually used. Experience shows that longstanding contests and
benchmarks are generally very useful and successful, but require a high level of individual commitment
of the organizers. We see the role of the TC-11 more as a facilitator, counselor and service provider
which would lower the burden on the organizers. This would probably be done under the form of
providing advice, best practices and support (or even provide) on-line back-office services for running
contests (upload and storage, output format formalization and comparison ...).