Newsletter Archive - Discussion on quality control & IAPR

From TC11
Jump to: navigation, search
Last updated: 2010-004-18


TC-11 Opinion Corner

FFT on conference-quality control in the near future

The abbreviation FFT in the title refers in this case, not to the useful Fast-Fourier Transform but to: Food For Thought. The upcoming conference season has stimulated a few thoughts among members of the TC-11 community on a worrying topic: the value of conference publications in this day and age.

Increasingly, research output is quantified in strict manners by university faculty deans and by corporate R&D managers. As one university dean put it: If quality is low, one must strongly hope that quantity is likewise. At the same time, the work of (spamming) commercial conference organizations is causing a serious devaluation of the concept of "conference paper" as a solid product of research.

Under these conditions, it becomes increasingly important to maintain quality standards within our own community, within TC-11 and within the IAPR as a whole. Already, the IAPR is considering possible scenarios for quality measurement and assurance. Please consider the following sketched outline, which may just serve as a stimulus for the formation of your opinion.

Journal articles will remain to be our ultimate target platform, and dedicated journals such as the IJDAR will need to obtain ISI status. For the conferences, the IAPR might install a method of quality stamps, or "stars". Each number of stars corresponds to specific criteria with respect to the rigor of the reviewing procedure and the nature of the organizational components of a conference event.

In such a system (Table 1), the ICPR conference would represent a four-star event, with an output value per article which is comparable to a journal contribution. Medium-scale regular conferences such as the ICDAR would represent three stars. Satellite conferences and workshops with a stable and long history and sufficient quality control would obtain two stars, whereas improvised low-threshold workshops sponsored by IAPR would obtain one star. The regulations should make explicit the detailed criteria for paper and poster contributions, for each level of event prestige.



Table 1. Possible future system for quality control
(not entailing a judgment on existing quality levels)
Stars Conference Goal
**** ICPR Disseminate findings relevant to PR at large
*** ICDAR Solid findings & designs in TC10+TC11
** stable satellites Experimental work, PhD student maiden voyages
* improvised workshops Focused new developments, round-table discussions


These views are not only my own, but are based on numerous discussions. We need to form our opinions and design a solution in view of the ongoing changes in research management. Whatever comes out of this, clear results have already emerged: The criteria and measuring sticks are becoming increasingly clear and explicit. This will be beneficial to all of us, PhD student and senior researcher alike. In the presence of an IAPR quality assurance mechanism, we will have our clear and testable answers ready when a skeptical instance asks us individually "why we needed to fly to X last year", where X denotes an element of the set of attractive conference destinations. More importantly, we will be able to defend our basic conviction that conference papers constitute an essential component of scientific productivity in technology research. Only then we will meet a willingness at high administrative levels to include an IAPR/TC-11 conference paper in the research-finance spreadsheets, with a sufficient and reasonable weight attached to it, in recognition of the quality of our work. Under such conditions, the negotiation of your travel budget including current conference fees will be greatly facilitated.

Lambert Schomaker, chair of TC-11
21 May 2004




Comments

Dear colleagues,



  we have obtained a number of reactions on 

my provocative opinions on conferences and

quality within IAPR & TC-11. All in all the

commentators agree with the observed problems,

but there does not seem to be much enthusiasm

for the proposed cure. All commentators but one

have expressed a preference to remain anonymous.

Therefore, I have decided to enter all comments

anonymously.



     Lambert Schomaker

     Mon May 31 13:57:11 CEST 2004






Commentator A (USA)


I think quality is very important in everything, including conferences.

I know I have my own internal ranking of how good the conferences I

attend and have contact with are. The same with journals. I use that

when I evaluate the people I meet at these conferences, and for journals

when we interview job candidates



Your idea of having a rating, is good, but I don't know if it can be

implemented, or if it were to be implemented whether it would have any

validity. I'm sure that you are aware that some papers presented in

lower ranking conferences and workshops are better than some at high end

workshops. I went to ICPR for the first time 2 years ago (..). While there 

were many good papers, I felt on a whole very disappointed given the 

reputation of the conference. You have your table of rankings on the TC-11 

opinion corner webpage, and you list ICDAR as lower than ICPR, but on a 

whole I found the ICDAR papers last year better (on average) than the 

papers I saw at ICPR 2 years ago. I think I can say the same with the papers 

at ICDAR 01 also. 



Also a bigger threat to quality is repetition of papers. 2 years ago I

saw one paper at ICPR and saw almost (I mean very almost - same slides,

100% match in meaningful content, certainly no new results- the written

documents aren't verbatim, but match more than 80%) the identical paper

at (workshop). So combining these two effects (...), my ranking of

ICPR went way down from what I had been lead to believe from hearsay

(and my ranking of that author really plummeted).



[Actually, the observation that paper versions were submitted with

a very limited edit distance is one of the triggers of the ongoing

discussion (LS)]



Also if you are going to look at ranking, I'm not sure how ranking

within one sponsor organization will help overall when the real problem

is so many conferences across a range of sponsoring organizations.



Good luck on this project. If quality isn't discussed, it will only be

attained by chance.






Commentator B (India)


I wish to add my  two bits worth for the FFT. Recently in our field conferences

like ICDAR want registration fee to be paid so that papers get included into

the conference proceedings. I think this will reduce the academic worth of the

proceedings. My apprehension is that proceedings contents will be dominated by

the papers of richer authors. Papers from [financially] poorer authors 

shall fail to find expression. Therefore I would give a higher star rating to a

conference which would publish the *best* papers.






Commentator C (Europe)


[edited (LS)]

Conference Proceedings that are published by Springer LNCS already have

higher status than a limited-edition proceedings without ISBN which is

printed locally by the organizers. This is a status-enhancement solution 

by many workshops.



[yes..., but the actual quality control has to be determined by the

 criteria for quality which exist in our actual community, the IAPR (LS)]

  

I consider [favorite workshop] a 'stable satellite'. I will tell my dean and 

director (...) that in order to remain visible in this field I HAVE to 

visit [favorite workshop]. Since such a specialized event is visited 

by all other Experts of the World in that area and at least three Experts 

review each paper, the status should be at least three stars. 






Commentators D+E (Europe, they sent in a joint comment)


In general, we support your idea, however there are some questions:



(1) who is the "Michelin" to decide on four stars?



[The IAPR, that is us! (LS)]



(2) who is willing to disregard his event as a two-stars-only thing 

    in advance?

    

[there seems to be a need for quality and status as well as for

 informal discussion and low thresholds for beginners in a field (LS)] 

     

(3) what are the criteria - number of submissions, number of published pages, 

    number of reviwers?

    

[these could be made explicit by our central organization, the IAPR (LS)]

    

(4) what about costs/ressources/quality balance when organizing a meeting?



(5) do you intent to propose this issue to IAPR? - so it needs to 

    propose/organize a formal procedure as well, like a voting-page using php.



- we think, for given reasons, one can not decide on quality freely. 

  It is a long-term process.



- someone might think that the "stable satellites" should be four star events,

  and not the mass meeting.

  

[indeed: they often are, but it might imply that the big conferences 
 
 are losing their status. There is a need for centripetal forces to 
 
 keep the field together. Concentration of quality could be a solution (LS)]  



- in order to ensure quality assurance, we might better use new technologies

  more extensively., e.g., providing  authors with the possibility to inspect

  and comment reviewers comments before the PC hammer falls (as it was seen at

  ECCV'04) , or to check CRPs whether reviewer comments have been considered.



- computer-science conferences seem still to have higher quality procedures

  than conferences in other fields, e.g., not like deciding on abstracts

  in the manner of "quickly submitting my abstract before leaving to the airport" 

  which luckily not yet established firm grounds in computer science conferences.



[I completely agree with this latter point. However, it is exactly for this

reason why deans and directors do not easily believe that our conference

papers are of higher quality than is the case in more lenient/slack

scientific arenas (LS)]


Summary

There seems to be a concensus that quality control is an important issue. At the same time, existing problems are becoming more clear. Those who have invested in workshops as their favorite platform will note the advantage of focused discussion, which is accompanied by an increasing level of quality in some areas. However, I would like to stress (knowing that I am on ice here) that it is important not to trick PhD students into believing that all is extremely well if their thesis consists essentially of one or two workshop contributions. The presence of an intermediate level of status between workshop paper and journal article could be provided by high-status IAPR conferences, constituting a stepping stone towards the journal paper.

A possible solution would consist of reformatting ICPR and ICDAR as umbrella events for workshops organized at the same, or very close to the same, venue. New options would consist in positioning the best papers submitted to the workshops as plenary oral presentations of the larger event. This approach would also prevent double submissions of papers. However, workshop organizers have complained that such a solution would cream off the best contributions and would reduce their freedom in selecting attractive small-scale venues with lively discussions in "forced isolation".

Conclusion

We cannot produce a clear-cut solution to these difficult problems within the framework of this TC-11 Opinion Corner. Still, I think the exchange of opinions has been very fruitful, making the important issues more explicit. Therefore, I would like to thank all commentators. [LS, May 31 2004]



This page is editable only by TC11 Officers .