Newsletter Archive - Discussion on quality control & IAPR
|
Contents
TC-11 Opinion Corner
FFT on conference-quality control in the near future
The abbreviation FFT in the title refers in this case, not to the useful Fast-Fourier Transform but to: Food For Thought. The upcoming conference season has stimulated a few thoughts among members of the TC-11 community on a worrying topic: the value of conference publications in this day and age.
Increasingly, research output is quantified in strict manners by university faculty deans and by corporate R&D managers. As one university dean put it: If quality is low, one must strongly hope that quantity is likewise. At the same time, the work of (spamming) commercial conference organizations is causing a serious devaluation of the concept of "conference paper" as a solid product of research.
Under these conditions, it becomes increasingly important to maintain quality standards within our own community, within TC-11 and within the IAPR as a whole. Already, the IAPR is considering possible scenarios for quality measurement and assurance. Please consider the following sketched outline, which may just serve as a stimulus for the formation of your opinion.
Journal articles will remain to be our ultimate target platform, and dedicated journals such as the IJDAR will need to obtain ISI status. For the conferences, the IAPR might install a method of quality stamps, or "stars". Each number of stars corresponds to specific criteria with respect to the rigor of the reviewing procedure and the nature of the organizational components of a conference event.
In such a system (Table 1), the ICPR conference would represent a four-star event, with an output value per article which is comparable to a journal contribution. Medium-scale regular conferences such as the ICDAR would represent three stars. Satellite conferences and workshops with a stable and long history and sufficient quality control would obtain two stars, whereas improvised low-threshold workshops sponsored by IAPR would obtain one star. The regulations should make explicit the detailed criteria for paper and poster contributions, for each level of event prestige.
(not entailing a judgment on existing quality levels)
| Stars | Conference | Goal |
|---|---|---|
| **** | ICPR | Disseminate findings relevant to PR at large |
| *** | ICDAR | Solid findings & designs in TC10+TC11 |
| ** | stable satellites | Experimental work, PhD student maiden voyages |
| * | improvised workshops | Focused new developments, round-table discussions |
These views are not only my own, but are based on numerous discussions. We need to form our opinions and design a solution in view of the ongoing changes in research management. Whatever comes out of this, clear results have already emerged: The criteria and measuring sticks are becoming increasingly clear and explicit. This will be beneficial to all of us, PhD student and senior researcher alike. In the presence of an IAPR quality assurance mechanism, we will have our clear and testable answers ready when a skeptical instance asks us individually "why we needed to fly to X last year", where X denotes an element of the set of attractive conference destinations. More importantly, we will be able to defend our basic conviction that conference papers constitute an essential component of scientific productivity in technology research. Only then we will meet a willingness at high administrative levels to include an IAPR/TC-11 conference paper in the research-finance spreadsheets, with a sufficient and reasonable weight attached to it, in recognition of the quality of our work. Under such conditions, the negotiation of your travel budget including current conference fees will be greatly facilitated.
Lambert Schomaker, chair of TC-11
21 May 2004
Comments
Dear colleagues,
we have obtained a number of reactions on
my provocative opinions on conferences and
quality within IAPR & TC-11. All in all the
commentators agree with the observed problems,
but there does not seem to be much enthusiasm
for the proposed cure. All commentators but one
have expressed a preference to remain anonymous.
Therefore, I have decided to enter all comments
anonymously.
Lambert Schomaker
Mon May 31 13:57:11 CEST 2004
Commentator A (USA)
I think quality is very important in everything, including conferences. I know I have my own internal ranking of how good the conferences I attend and have contact with are. The same with journals. I use that when I evaluate the people I meet at these conferences, and for journals when we interview job candidates Your idea of having a rating, is good, but I don't know if it can be implemented, or if it were to be implemented whether it would have any validity. I'm sure that you are aware that some papers presented in lower ranking conferences and workshops are better than some at high end workshops. I went to ICPR for the first time 2 years ago (..). While there were many good papers, I felt on a whole very disappointed given the reputation of the conference. You have your table of rankings on the TC-11 opinion corner webpage, and you list ICDAR as lower than ICPR, but on a whole I found the ICDAR papers last year better (on average) than the papers I saw at ICPR 2 years ago. I think I can say the same with the papers at ICDAR 01 also. Also a bigger threat to quality is repetition of papers. 2 years ago I saw one paper at ICPR and saw almost (I mean very almost - same slides, 100% match in meaningful content, certainly no new results- the written documents aren't verbatim, but match more than 80%) the identical paper at (workshop). So combining these two effects (...), my ranking of ICPR went way down from what I had been lead to believe from hearsay (and my ranking of that author really plummeted). [Actually, the observation that paper versions were submitted with a very limited edit distance is one of the triggers of the ongoing discussion (LS)] Also if you are going to look at ranking, I'm not sure how ranking within one sponsor organization will help overall when the real problem is so many conferences across a range of sponsoring organizations. Good luck on this project. If quality isn't discussed, it will only be attained by chance.
Commentator B (India)
I wish to add my two bits worth for the FFT. Recently in our field conferences like ICDAR want registration fee to be paid so that papers get included into the conference proceedings. I think this will reduce the academic worth of the proceedings. My apprehension is that proceedings contents will be dominated by the papers of richer authors. Papers from [financially] poorer authors shall fail to find expression. Therefore I would give a higher star rating to a conference which would publish the *best* papers.
Commentator C (Europe)
[edited (LS)] Conference Proceedings that are published by Springer LNCS already have higher status than a limited-edition proceedings without ISBN which is printed locally by the organizers. This is a status-enhancement solution by many workshops. [yes..., but the actual quality control has to be determined by the criteria for quality which exist in our actual community, the IAPR (LS)] I consider [favorite workshop] a 'stable satellite'. I will tell my dean and director (...) that in order to remain visible in this field I HAVE to visit [favorite workshop]. Since such a specialized event is visited by all other Experts of the World in that area and at least three Experts review each paper, the status should be at least three stars.
Commentators D+E (Europe, they sent in a joint comment)
In general, we support your idea, however there are some questions:
(1) who is the "Michelin" to decide on four stars?
[The IAPR, that is us! (LS)]
(2) who is willing to disregard his event as a two-stars-only thing
in advance?
[there seems to be a need for quality and status as well as for
informal discussion and low thresholds for beginners in a field (LS)]
(3) what are the criteria - number of submissions, number of published pages,
number of reviwers?
[these could be made explicit by our central organization, the IAPR (LS)]
(4) what about costs/ressources/quality balance when organizing a meeting?
(5) do you intent to propose this issue to IAPR? - so it needs to
propose/organize a formal procedure as well, like a voting-page using php.
- we think, for given reasons, one can not decide on quality freely.
It is a long-term process.
- someone might think that the "stable satellites" should be four star events,
and not the mass meeting.
[indeed: they often are, but it might imply that the big conferences
are losing their status. There is a need for centripetal forces to
keep the field together. Concentration of quality could be a solution (LS)]
- in order to ensure quality assurance, we might better use new technologies
more extensively., e.g., providing authors with the possibility to inspect
and comment reviewers comments before the PC hammer falls (as it was seen at
ECCV'04) , or to check CRPs whether reviewer comments have been considered.
- computer-science conferences seem still to have higher quality procedures
than conferences in other fields, e.g., not like deciding on abstracts
in the manner of "quickly submitting my abstract before leaving to the airport"
which luckily not yet established firm grounds in computer science conferences.
[I completely agree with this latter point. However, it is exactly for this
reason why deans and directors do not easily believe that our conference
papers are of higher quality than is the case in more lenient/slack
scientific arenas (LS)]
Summary
There seems to be a concensus that quality control is an important issue. At the same time, existing problems are becoming more clear. Those who have invested in workshops as their favorite platform will note the advantage of focused discussion, which is accompanied by an increasing level of quality in some areas. However, I would like to stress (knowing that I am on ice here) that it is important not to trick PhD students into believing that all is extremely well if their thesis consists essentially of one or two workshop contributions. The presence of an intermediate level of status between workshop paper and journal article could be provided by high-status IAPR conferences, constituting a stepping stone towards the journal paper.
A possible solution would consist of reformatting ICPR and ICDAR as umbrella events for workshops organized at the same, or very close to the same, venue. New options would consist in positioning the best papers submitted to the workshops as plenary oral presentations of the larger event. This approach would also prevent double submissions of papers. However, workshop organizers have complained that such a solution would cream off the best contributions and would reduce their freedom in selecting attractive small-scale venues with lively discussions in "forced isolation".
Conclusion
We cannot produce a clear-cut solution to these difficult problems within the framework of this TC-11 Opinion Corner. Still, I think the exchange of opinions has been very fruitful, making the important issues more explicit. Therefore, I would like to thank all commentators. [LS, May 31 2004]
This page is editable only by TC11 Officers .