Minutes: International Scientific Committee Meeting 9 Time/Date: 10:10 -- 12:20, 22 August 2003 Place: Univ. of Wisconsin - Parkside, Molinaro Building, Kenosha, WI, USA Status: Approved (Revised April 2004) Present (alphabetical order): Hal Burch Host n (leaving member) Kyung-Yong Chwa Host n-1 (leaving member; left after item 4) Greg Galperin Host n Michalis Hatzopoulos Host n+1 Marcin Kubica Host n+2 (new member after IOI'03) Ian Munro Elected (leaving member; left after item 4) Jyrki Nummenmaa Chair ITWG (new member after IOI'03; left at 11:00) Djura Paunic Elected Iphigenia Founta Host n+1 (new member after IOI'03) Xenofon Papadopoulos Host n+1 (new member after IOI'03) Ismail Hakki Toroslu Elected (new member after IOI'03) Tom Verhoeff Re-Elected (chair) 1. The meeting is opened by Tom Verhoeff. 2. The proposed agenda is approved without changes. 3. The minutes of the previous meeting (7th, in Korea) are confirmed. 4. Tom informs the members that he will most likely not be present at IOI'04 in Greece (the dates for IOI'04 are not yet established), but that he is willing to chair the committee until that time. Jyrki proposes to decide now how to handle the situation when Tom will not be available. No decision is made. The committee appoints Tom as chair for the next period. 5. The leaving members, Kyung-Yong, Hal, and Ian, are thanked for their contributions to the committee, and especially for their involvement in the preparation and execution of the IOI'03 competition. They take the opportunity to say a few words. 6. Three documents were discussed: * The definition of the IOI Technical Working Group (ITWG), as written in the newly accepted IOI Regulations. * The proposal by South Africa to allow the Java programming language * A letter from Mark Cieliebak (Deputy Leader Switzerland) and Peter Kese (Team Leader Slovenia) about their investigation of the relation between IOI contestant preparation and their scores at the IOI. Concerning the ITWG, Jyrki briefly informed the committee of his plans as its first chair. He intends to contact developers of software tools that are being used in the IOI and to inform them and seek their cooperation. He also requested names of other technical experts, within and outside the IOI community, whom ISC members felt he should contact. He will draw up a more detailed plan later. He expressed his concerns about the two similar but different programming development environments currently in use at the IOI. Jyrki had leave early to catch his flight back. * The proposal about Java consists of a 6-page (undated, unsigned) document, which was distributed by the proposers to delegations at IOI'03. It addresses a number of issues that naturally arise when considering the introduction of a new programming language in the IOI. The proposal drew a number of comments. * It seems that when using an appropriate Java compiler, the execution speed difference with the current IOI programming languages (viz. C/C++, Pascal) can be considered small enough to avoid the introduction of separate limits on the execution time for Java programs. On the other hand, such a compiler is currently not available for the Windows platform. But this is expected to change in the future. Of course, the claims have to be verified. * It will require that the contest support system handles submissions of programs spanning multiple source files. * The USACO will extend their contest support system to handle Java programs next year. * Marcin indicated that the Polish national olympiad and the CEOI might be good opportunities to carry out some experiments with Java. * It is suggested that someone develops Java solutions for the IOI'03 competition tasks. * There is the question of which IDE to use, and whether this should be the same for Linux and Windows. * Greece is not in favor of allowing Java at IOI'04, because it will only add to the technical burden. * Some found the motivation for adding another programming language not convincing. Java resembles C++, and even Object Pascal, very much from a scientific point of view. All are object-oriented imperative programming languages. However, the burden of supporting an additional programming language on two platforms must not be underestimated. * Is it possible to drop something else when allowing Java? For instance, Windows as development platform? * The IOI'03 competition survey has some questions about Java, and the results should be taken into account when discussing Java further. We have steered clear from programming language battles, and differing opinions on how to approach the teaching of computing science and, in particular, programming. But these do play a role. At some point, these matters will have to be faced. It is clear that there is only limited support for the Java proposal. The decision to adopt another programming language should not be taken lightly. Time will tell how many other countries really "need" Java at the IOI. The only way to proceed seems to be to await (written) reports on further experiments with Java in competitions similar to the IOI. * The letter from Cieliebak and Kese requests details on the scores for the contestants at IOI'03, and if possible also at IOI'01 and '02. These scores will only be used to correlate preparation practices and IOI performance. Only aggregate data will be published, so that no information about individuals is disclosed. Special permission is needed to obtain this information from the organizers, because the IOI has decided in the past that results of non-medal winners must not be published. It is suggested that the authors could have requested the results on their survey form, because leaders are free to supply that information if they wish to contribute to the investigation. The ISC wishes to support the investigation and seeks a way to provide the requested information under the following condition. The ISC requires that any data derived from the confidential results and intended for publication is submitted to the ISC for prior approval. 7. Tom briefly reminds those present about the need for confidentiality. Confidential communications need to be encrypted. For that purpose, the ISC uses PGP. Public PGP keys of members are available on the web. Those new to the committee are requested to provide a public PGP key. For some ISC members there may occasionally arise a conflict of interest, when they are also involved in the organization of a national or regional olympiad. It would be best to refrain from such involvement, but we also understand that in some countries or regions, there may not always be a good alternative available. To avoid misunderstandings members are requests to report their involvement in other activities (training camps, competitions, etc.) where (potential) IOI candidates participate. We trust that all ISC members understand their position in this respect, and that they will know how to handle delicate situations properly. In case of doubt they should ask the ISC for advice. 8. We briefly discussed the IOI'03 competition, which was just completed. Two of the six competition tasks were contributed externally. One of these had to be adapted considerably. Three tasks for the practice session were available before the IOI. The schedule was a bit tight on the day with the practice session. In particular, the GA meeting to be held after the practice session but before the GA is separated from the contestants was not scheduled. It had to be squeezed in. The task approval and translation started earlier than in previous years. This meant less time spent on excursions and less stress in the night. Most delegations were done translating before midnight. The casual attitude at IOI'03 was appreciated by everyone, and contributed to the friendly atmosphere. Security was rather relaxed. In fact, a bit too relaxed, because a contestant simply got hold of confidential material via the "pigeonhole mailboxes" (and subsequently turned it in to an official). This incident was reported to the GA. ISC members agreed that draconian security measures might provide some psychological reassurance, but would not improve real security significantly. They also stated that it should be reasonable to trust leaders, but that contestants should not be trusted in the same way. It is advised that badges are designed to make the IOI-role clearly visible from a distance (e.g. by a simple color code). Furthermore, it should be well-defined for all involved, both in the written program and by on-site signs, which areas are supposed to be contestant-free for what period of time. Both competition days started on time. One unfortunate incident involved a contestant who had a very high score on the first day, but who failed to submit any programs developed on the second day, resulting in a zero score for that day. That person actually did not log into the system on the second day at all. We should seek ways to minimize the opportunity for such incidents. It is true that the IOI is a competition, but the IOI is intended as a personal scientific challenge to display skills in algorithmic problem solving, rather than a recreational game. The contest support system could be monitored, and contestants who did not log into the system after some time could be checked upon and notified. There might be some kind of problem related to the use of the system. The contest support system is an organizational artifact, whose side-effects on the competition should be minimized. For task Code, the test data had the property that a simple and incorrect program (viz. writing the minimum of the lengths of the two input programs) obtained a 25% score. In the future, the ISC should be sure that they understand the test data set and the scoring that various implementations would achieve on that set. The GA is not in a position to do such an inspection, and it is the duty of the ISC to know about this. As an aside, the mean score on Code was below 14 points, less than half of the other tasks; and of the 165 people who scored any points on this task, 95 of them scored less than 25 points. The web-based competition services (submit, print, backup/restore, test run) worked very well. It was based on Rob Kolstad's systems used at IOI'01 and for the many USACO Internet contests. The 'analysis mode' was a new feature for IOI. Grading reports were printed in summary form, with details available through the web. In analysis mode, delegations could submit programs for evaluation after the competition, as an aid to understand the results, but without affecting them. Some further facts about the IOI'03 competition: * There were 14 grading appeals this year, three of which related to the student who did not log in. This is many fewer than in previous years (at least recent). The analysis mode on the grading system is believed to have caused this low count. If that is true, similar features should be included in the grading system for future IOIs. * No grading appeal resulted in a score change. * The top score was 455.4 out of 600 (76%), the lowest (in percentage terms) of the IOIs for which the top score was published (1992 - 2002, excluding 1997 and 1998). The closest was 1999, where the top score was 480 of 600 (80%). The Host SC expected a higher top score. In addition, several members of the GA expected much higher medal cutoffs than occurred (one coach expressed concern that a student may not get any medal, even though they had earned a mid-silver score). No student got 50% or more on all six tasks, and only three got 50% or more on five of the tasks. Despite this low top score, medal cutoffs were clean and easily made, with gaps of 7 (gold), 4.8 (silver), and 1.1 (bronze). * There were several people that got hit by the UNIX/MsDOS newline differences. In such cases, their programs would work correctly under MsDOS, but fail under Linux. This is an unfortunate side-effect of allowing contestants to program under an operating system different from the OS of the grading system. It is not clear what would have happened if their programs, compiled under Linux, would have been given input files with MsDOS newlines. If we could detect when such errors are made (during the test run), we could either grade their submission with input files using MsDOS newlines or (preferably) inform them of the nature of the error programmatically. * Several people were confused on Guess which Cow, because the input came both from a file and standard input. In such cases, they expected the file input to come on standard input. It is not certain what their programs received as response, but it was, at least initially, time limit exceeded, as their program failed to output a question within the real-time limit. Either such tasks should be avoided in the future, or this "dual input" should be made more explicit. * Announcing median scores for each task does not seem to give the GA excessive information in determining medal cutoffs. * The graphical representation used for medal cutoffs (verticle bars, spaced according to score) well-conveyed the relative gap sizes, while making it difficult to determine the gaps themselves. In the past, the constentants and the GA have demonstrated that giving the point gaps numerically often gives them enough information to determine the medal cutoffs before the closing ceremony. This may only be true if no ties occur in the vicinty of cutoffs. For IOI '03, although some of the top and bottom scores displayed for individual cutoffs had more than one contestant with that score, this was not communicated to the GA, so no ties were communicated. At IOI'03, non-integer scores were used (in contrast to at least the preceding 10 years, where only integer scores were awarded). This was a surprise for the GA, but also for the ISC. It had not been discussed beforehand, and was not approved by the ISC. Further discussion of this issue was postponed. The Host SC conducted a survey among the participating delegations. It resembles last year's survey and is intended to get feedback on, among other things, the quality of the compilers (GCC, FreePascal), development environments (Linux, Windows, IDEs, tools, etc.), and the contest support system. The results of the survey are expected in January 2004. In contrast to the previous two years, a publication with more details about the tasks, including complete source code listings of solutions was not available at the closing ceremony. A publication and further analysis would be done later. Some ISC members expressed their concern about the low median score (173, with maximum possible score 600). This is higher than last year, but it should be higher still. Some had in fact expected higher scores, because this year's tasks were not all designed to be hard to solve perfectly (as had been the case at IOI'02). A reasonable aim seems a median of 200+ (with 600 as max.). You probably cannot expect much more without losing the distinctions at the top (differentiating between gold and silver). It would be best if a bronze medal would require the complete solution of at least one task (at 95% or more) and significant points from two or three other tasks (at 30% to 50% or more). [As a side note: We did not discuss the issue of "ideal" scores in any detail. It concerns the difficulty level of the tasks, the choice of test data, and the accompanying scoring rules.] 9. Concerning the preparations for IOI'04 (Greece), the following remarks were made. * The ISC document "Guidelines for IOI Competitions" was brought to the attention of the IOI'04 Host SC members. * Review meeting: about three months in advance (i.e. June 2004). 10. General issues. * Why were nondeterministic programs disallowed at IOI'03? Answer: To avoid dilemmas when regrading a program. It does not exclude randomizing algorithms, as long as they are deterministically seeded. It is the duty of coaches to educate their contestants about this. * What about timing nondeterminism? Yes, that is an issue, though much less so than nondeterministic choices in programs. Also the choice of test inputs plays an important role. They should stay away from "gray" areas, where some "typical" to-be-expected algorithms will perform close to the time limit. * At the Polish National Olympiad, gradual scores are given, depending on the actual running time. That is, there is no discontinuity near the time limit. However, this does not solve the problem of determining a scientifically justified score, because the choice of programming language and certain low-level implementation decisions affect the timing details as well. This should be analyzed further. * The Linux kernel can be patched to provide more accurate timing. This patch as used at IOI 2003 and worked very well. There are still variations on the order of milliseconds when running the same program on the same input. 11. Tom closes the meeting.