This Web-based package provides capabilities for
Activities related to submissions and administration support three basic constituencies: general users, conference leadership, and system/database administrators. The documentation that follows attempts to address the needs of these constituencies in successive sections.
Updating of this material is an on-going process; the status of this work is as follows:
| Section | Title | Revision Date | Comment(s) |
|---|---|---|---|
| I | Introduction/Orientation | 20 July 2008 | Revision completed |
| II | User Activities | 20 July 2008 | Revision completed |
| III | Conference Chair/Program Chair Activities | 6 August 2008 | Revision completed |
| IV | System Administration | 19 August 2008 | Revision completed |
This Web-based package represents an on-going project to support conferences by SIGCSE and other organizations. Started in 1999 for SIGCSE 2000, the Web pages, scripts, and supporting documentation have evolved dramatically over the years. A Documentation History for the Proposal Submission and Reviewing Process is available for those interested in reviewing the history of this effort.
This system supports the submission and reviewing of proposals to conferences, including these basic capabilities:
Scripts support most capabilities, so that conference leadership will not need to program.
Past experience indicates that attempts by conference leaders to be helpful by downloading scripts via ftp, editing, and uploading revised script files can cause significant problems that require many hours of work by developers/maintainers to correct.
The main Web-index page for a conference contains several types of links:
The files for this system are located on two servers:
With this structure, if the Windows database server is down, a symbolic link on www.cs.grinnell.edu (e.g., submission.shtml, revieweRegistration.shtml) can point to pages that notify users of the problem, and all conference references to (e.g., to submission.shtml, reviewerRegistration.shtml) remain valid. Since the Linux server has been extremely reliable over the years, this two-part approach has worked well.
Conference leadership will likely want to have Web links for the conference to the www.cs.grinnell.edu .shtml pages, but you will never need to log into that system. (I used to give the password to that account to conference leaders. However, they never used the account for 4-5 years, so distribution of that password no longer seems needed.)
The administrative script dbAdmin.asp provides support for most leadership tasks. Thus, many of the common tasks are supported directly; scripts identify and update relevant tables. There are two situations, however, when conference leadership will want to edit specific database tables. The first of these can use dbAdmin.asp through the "View Database" option, but the second requires conference leadership to download the full database via ftp for editing.
Using dbAdmin.asp, conference leadership can edit many database tables. Passwords for submission chairs are in the SubmissionTypeMap table. This can be edited using the "ViewDatabase" option within dbAdmin.asp.
Once this set up is done, there will be times when conference leadership wants to close the submission of papers, open up reviewing, etc. All deadline-related activities are monitored through the Permissions table — again using dbAdmin.asp. For a wide range of tasks, a field in the table is set to "Yes" when the task is allowed and "No" when it is not. For example, submission of papers is allowed when the field paperSubmissionsAllowed is "Yes". Thus, when the time comes to stop paper submissions, this is accomplished by using the "Edit Database" capability of dbAdmin.asp to change the paperSubmissionsAllowed entry from "Yes" to "No".
Once the system is set up, the only editing of the database required with dbAmin.asp will involve changing permissions.
Once the full program has been determined (e.g., decisions are completed regarding acceptance and rejection, and the schedule is determined), the acceptance/rejection information and the scheduling data must be entered into the database. Unfortunately, this is tedious — there is just a lot of material to enter, and use of dbAdmin.asp is extremely awkward. For this task, conference leadership likely will want to download the database, use Access to enter the data, and then upload the revision.
Originally, a third task also required work that went beyond the scripts.
To send e-mail to reviewers or submitters, the process involved putting a letter template onto the db.grinnell.edu server. This template contains variable designations for inserting material for a particular reviewer or author. A script actually sends the letter, but the template must be on the server. Historically, getting the template file to the server required ftp. However, in July 2008, an additional function was added to dbAdmin.asp for this purpose, and ftp is no longer required for sending e-mail.
The following documentation has two main sections, organized according to function.
Computer science educators are involved with conferences primarily as authors/proposers of sessions, reviewers, and attendees. A general description of such activities is available through the conference home page. Notes here describe the basic scripts for:
Contributors submit proposals to the conference through a common interface, submission.shtml. In reality, this URL is a symbolic link to one of the following:
By changing the link submission.shtml, contributors are directed to the relevant page listed above. Of the actual pages, submissionDown.shtml, submissionEarly.shtml, and submissionLate.shtml are single pages with no further processing functions.
As already noted, submissionRegular.shtml is a symbolic link to submissionRegular.asp on db.grinnell.edu. This .asp script reads the database (table submissionTypeMap) to list the various submission categories. This .asp script then initiates processing flows, as shown in the following figure.
As suggested in the figure, the form submissionRegular.shtml (and its target script submissionRegular.asp) provides two basic forms, one for the submission of a new proposal and the second to view and/or modify an existing proposal.
Within submissionNew.asp, the contributor provides information regarding the proposal, supplies a password to allow later access to proposal data, and provides the relevant pdf file(s).
Upon submission of this information, submissionNew.asp calls submissionAction.asp that creates new records for the proposal, submitter, proposalSubmitter, and proposalSubject tables (as defined in the SubmissionTypeMap table).
Since the scripts log in as a new user, users cannot view the database over the World Wide Web. Also, proposals are stored under the ownership of a private user, so they are not available either over the World Wide Web or by a database user.
submissionUpdate.asp provides four basic capabilities:
Submission of contributor/proposal data uses the same submissionAction.asp script used for new submissions.
When storing or updating contributor/proposal data, submissionAction.asp confirms updates by sending e-mail to the submitter. Other scripts return relevant information to the screen, but do not send e-mail.
Interested individuals register to become reviewers. The conference leadership then assigns proposals to reviewers and notifies reviewers of these assignments via e-mail. Reviewers view electronic versions of the proposals through a Web-based interface, and they submit their reviews electronically through another Web-based form. After the reviewing process is complete, reviewers may view all reviews for submissions they have reviewed.
From a reviewer's perspective, work proceeds from three entry points:
The following subsections describe these capabilities in some detail.
Each conference maintains its own database tables for reviewers. Thus, when one conference is over, the reviewer tables are copied to the next conference. For annual conferences, this process is straightforward. However, when two conferences (e.g., the SIGCSE symposia and the ITICSE conferences) wish to utilize the same reviewer database, consistency issues can arise. At present, the transition is accomplished through the following pages:
The following figure illustrates the basic flow for active reviewer registration.
When the current database is active for reviewer registration, the start for processing comes from reviewerRegistrationRegular.shtml.
Once reviewers have registered, conference leadership can establish reviewing assignments, set permissions to allowing reviewing to take place, and e-mail the reviewers with a listing of proposals for consideration.
The main starting place for reviewer activity is reviewerViewing.asp, found on db.grinnell.edu. The following figure shows the main flow for this processing.
reviewerViewing.asp (on db.grinnell.edu) asks for a reviewer ID and password. Successful login leads to the reviewer base page, reviewerAssignments.asp. Capabilities for this page are determined by the Permissions table: the reviewing of proposals, submission of reviews, and viewing of other reviews are all controlled by fields within this Permissions table. As suggested by the figure, each of these capabilities are supported by scripts:
The review form asks a reviewer to supply comments for each ratings area. If the reviewer leaves the comment fields blank, JavaScript in the form requests that comments be entered — up to 2 times. If the reviewer persists in not entering comments, the data are sent to the database anyhow.
Actual update of the review occurs in a ratings table, accomplished by the script proposalUpdateReviews.asp.
As noted on the above figure on Reviewer Activity Scripts, review forms can be displayed directly — without a proposal ID or reviewer ID. Since the content of forms depends upon the category of submission, the display of blank forms proceeds in two steps:
Normally, a reviewer would complete a review, starting with reviewerViewing.asp. In that case, when a reviewer requests a review form (via reviewerAssignments.asp), proposalReviewForm.asp preloads a proposal number, title, reviewer number, and any comments/ratings previously submitted. However, a reviewer also could use proposalReviewForm.asp directly (from selectReviewForm.asp) to complete a review. In such a situation, submission of the review calls script proposalUpdateReviewReviews.asp, that first checks the validity of the proposal ID and reviewer ID. Reviews are only allowed from reviewers assigned by the conference leadership for each submission.
The system allows attendees to view the conference program in several ways:
The interaction of these scripts is illustrated in the following figure.
Documentation for Conference and Program Chairs has three parts:
The organization and planning for a conference involves the completion of many tasks. Guidance for the performance of these tasks is available from several sources. For example, ACM provides a Conference Manual that outlines in some detail tasks related to ACM approvals, committee organization, finance, registration, publicity, operations, the technical program, proceedings, exhibits, etc. Scheduling these various tasks is somewhat tricky, due to the differing scale of conferences, the extent of geographic dispersion of committee members, and the flow of everyday life at varying times of the year.
The following table presents two general schedules, based on SIGCSE conferences.
For each schedule, the following table identifies required tasks. Many task labels serve as links to detailed documentation that appears later in this Part.
| General | Feb/March Conference | June/July Conference | WHAT: Activities |
|---|---|---|---|
| 9 months ahead | early June | early October |
|
| 8 months ahead | July | November |
|
| 7 months ahead | August | December |
|
| 6–7 months ahead | August–September | December–January |
|
| 6 months ahead | September | January | |
| 5–6 months ahead | September–October | January–February |
|
| 5 months ahead | October | February | |
| 4.5 months ahead | early November | early March |
|
| 4 months ahead | November | March |
|
| 3 months ahead | early December | early April | |
| 2–3 months ahead | December–January | April–May | |
| 1 month ahead | February–March | June–July |
Some general principles may be useful in guiding conference leaders as they work with this system. These principles fall into four general areas:
Script dbAdmin.asp provides a common interface for many of the administrative tasks within the system. The page itself is organized into two main parts.
The system allows letters to be sent to submitters, reviewers, and session chairs. In each case, the first step is to develop a template that gives the text to be sent to individuals and that specifies where individual data will be inserted. The details of these letters depend upon their intended recipients (submitters, reviewers, session chairs). However, these letters all follow some common guidelines.
Elements in common for all types of letters include the following.
Creation of this letter template may be accomplished in either of two ways:
Passwords are assigned for several categories of administrative users:
Except for the conference-level password, a password allows the Proceedings' Chair and submission chairs access to materials for their work, but these people cannot access the full range of scripts.
For any type of password, the system will remember a password (via a session variable), once it is given. Thus, if a chair has several tasks, the chair can give the relevant password once at the start of a session. Thereafter, the system will remember the password during the session, and the chair need not enter the password again.
Users are granted capabilities for various activities in this system through a series of permission fields, detailed in a Permissions table within the database. In the database, each permission field is set to either "Yes" or "No" to allow or disallow a specific capability.
At many conferences, different categories of submissions have different deadlines, and reviewing schedules depend upon the submission category. To accommodate differing submission and reviewing deadlines, the Permissions table has separate fields for each category of submissions and reviewing. Other fields relate to the availability of accepted proposals through the online program and to various reviewing and revision activities. The following table identifies these fields within the Permissions table.
| Field(s) in the Permissions' Table | Capability Description | Suggested Time for Granting Permission | Suggested Initial Value |
|---|---|---|---|
| panelSubmissionsAllowed
paperSubmissionsAllowed specialSessionSubmissionsAllowed workshopSubmissionsAllowed etc. | allow proposers to submit materials | allowed through the submission period | Yes |
| ReviewerAssignmentsAvailable | reviewers have been assigned to submitted proposals, and reviewers are allowed to view those assignments | allowed during the reviewing period and thereafter | No |
| panelReviewsAllowed
paperReviewsAllowed specialSessionReviewsAllowed workshopReviewsAllowed etc. | reviewers are allowed to submit reviews for their assigned
proposals
| allowed only during the reviewing period | No |
| ReviewViewingAllowed | proposal submitters and reviewers of proposals are allowed to read all reviews of their proposals | allowed after proposers are told about the acceptance or rejection of their submissions | No |
| FullUpdateAllowed | submitters of accepted proposals may update the publication version of their proposals | allowed after acceptances mailed until proceedings deadlines | No |
| AcceptedSubmissionDisplayAllowed | the publication version of accepted proposals may be viewed through the online program | allowed after the proceedings deadlines until a month or so after the conference (when the final papers and proposals appear in the ACM Digital Library) | No |
Conference leaders can set any of these fields through the Web administrative interface dbAdmin.asp using the option to "View Database Tables" within the administrative script dbAdmin.asp. The basic steps follow:
The following sections describe each of the administrative tasks in some detail.
As described under the system administration section of this documentation, Web-based materials for this software package involve both shtml and asp pages. Details for preparing files depends on the nature of the pages involved.
Preparation of these conference Web materials involves six basic types of activities:
The following notes relate to conference-related information that might be supplied by conference or program chairs. Additional information may be found under the notes for System and Database Administration.
The conference leadership determine the basic look and feel of conference Web pages. A uniform style and page format is achieved through style sheets and header files. Often, headers and styles will include graphical images. (For historical reasons, no general footer files are used, so the body of script pages cannot be in a div tag — there is no way to close a div tag at the end of each page.)
While the concepts of style sheets and header files extend to all files, some details differ for shtml pages and for asp pages.
Both shtml pages and asp pages contain many conference specific elements, such as conference name, conference abbreviation (e.g., SIGCSE 2008, ITiCSE 2009), dates, URLs, leadership names and titles, leadership e-mail addresses.
For asp scripts, all of these details are collected together as variables in file sigcseBasic-include.asp. While the list of variables is extensive, assigning variables once in this file provides the needed information for all asp scripts.
System-level passwords have several distinct purposes and are specified in separate include files on db.grinnell.edu
The passwords for the overall conference leadership and for the Proceedings Chair are set in collaboration with the leadership; the password for script access is reserved for system administrators.
Conference leaders may tailor this system to support various categories of submissions. All asp scripts are table driven, based on a database table, SubmissionTypeMap. In this table, one record contains all relevant information for a category of submissions. While many of these fields may be set by system and database administration, the conference leadership must decide on the following elements:
(In addition, the database administrators will need to identify various database tables that contain records for elements of each submission.)
Since this description may seem somewhat abstract, the following information provides examples for many of these elements for recent SIGCSE Symposia.
| Category: | Panel | Paper | Special Session | Workshop |
|---|---|---|---|---|
| Submitter Title: | Moderator | Author | Proposer | Proposer |
| Collaborator Title: | Panelist | Co-Author | Leader | Leader |
| Collaborator Title (plural): | Panelists | Co-Authors | Leaders | Leaders |
| Minimum group: | 3 | 1 | 1 | 1 |
| Initial Collaborators Displayed: | 5 | 5 | 5 | 3 |
| Maximum group: | 5 | 20 | 12 | 5 |
| Chair Title | Panel Chair | Program Co-Chairs | Special Session Chair | Workshop Chair |
| public submission name (with appropriate capitalization for a title) | Panel | Paper | Special Session | Workshop |
| commentary to describe publication version | public description of the panel | publication version (with normal author/institution information) | publication description of the special session | publication version (with full leadership/institution information) |
| anonymous version required | no | yes | no | yes |
| text-based abstract required | no | no | no | yes |
| commentary to describe anonymous version (if required) | anonymous version with all references to the author(s) removed | reviewer version with equipment needs and participation limits | ||
| full review form | no | yes | no | no |
| proposal table: | Panel | Paper | SpecialSession | workshop |
| submitter table: | Panelist | Author | Leader | Organizer |
| Submission/person table: | PanelPanelist | PaperAuthor | SpecialSessionLeader | workshopOrganizer |
| table for proposal subject | panelSubject | paperSubject | specialSessionSubject | workshopSubject |
| table for subject given by reviewers | panel/paper/specialSession/workshopReviewerAssignedSubject | |||
| [tables for recording accepted proposals, rejected proposals, and those identified as being appropriate for posters] | ||||
| table for reviewer ratings | panelRatings | paperRatings | specialSessionRatings | workshopRatings |
Although the system and database administrators may handle details of the database tables, the conference leadership must determine category types and their basic parameters.
As already noted, most forms and scripts include information about a conference, such as the conference number (e.g., the 31st Annual SIGCSE Symposium), dates (e.g., March 8-12, 2000), location (e.g., The Renaissance Hotel, Austin, Texas), and Program Chair (e,g, Henry Walker, sigcse@cs.grinnell.edu).
Since scripts retrieve such information from the sigcseBasic-include.asp file, the good news is that all of this material may be specified once through assignments to variables.
The bad news is that shtml pages are not generated dynamically, and all non-header information must be hard-coded. Thus, a conference and/or program chair must review each shtml document. These pages may be organized into two families:
Each of these files must be updated with conference-specific information.
As the system is set up and tested, new submission categories may be added, other categories may be adjusted, test proposals may be submitted, reviewers may be assigned to submissions, and reviews submitted. When this preliminary work is complete, entries in the submissionTypeMap table should be checked, and the database should be reinitialized.
Within dbAdmin.asp, the option to "Set up the Database at the Start of a Conference" performs the following tasks:
Since this script option removes data and cannot be undone, this script should be used with care. To help prevent accidental erasure of data, the scripts also ask for explicit confirmation before performing these tasks.
Called from dbAdmin.asp:
Users are granted capabilities for various activities in this system through a series of permission fields, detailed in a Permissions table within the database.
When a conference site becomes live, permission should be granted for proposals for each category of submissions. This is accomplished by setting SubmissionsAllowed fields in the Permission table to "Yes" for each category of submission. For example, fields paperSubmssionsAllowed, panelSubmissionsAllowed, etc. should be set to "Yes".
All other permissions (e.g., for reviewing and the online program) should be set to "No".
An overview of the Permissions table and details on changing specific permissions are given in the section above on Deadline Enforcement and Permissions.
During initial preparation for a conference, the leadership may wish to remove some past reviewers. This typically arises under any of three circumstances.
Since conferences rely on reviewers, it is important to remove non-productive records. However, it also is important to maintain records of as many productive reviewers as possible. Thus, paring the database should be done sparingly.
Details of removing a reviewer are given in the section on reviewer deletion. Before considering that action, however, two notes are in order:
The first two of these may be handled directly through dbAdmin.asp, the primary administrative page for this system. The third type of query can be handled from the normal submission page submission.shtml. Some details follow.
Finding a proposal is handled within dbAdmin.asp by the first proposal-related option (in the second half of the page). Upon specifying the proposal type and administrative password, boxes ask for either the proposal ID or the submitter's last name. The response shows proposal information for the given ID or outlines data on all proposals for which the submitter or a collaborator has the specified name.
Removing a proposal begins with the second proposal-related option on dbAdmin.asp. After supplying the administrative password, boxes ask for the proposal ID or the submitter's last name. For safety, a follow-up page asks you for confirmation before any proposal is deleted. Also for safety, proposals can be deleted only one at a time, so the process must be repeated multiple times when several deletions are needed.
Finding Submission Information: The main source for all submission information (proposers, titles, subjects, descriptions) is submission.shtml.
Notes:
Individualized e-mail messages may be sent to collections of reviewers, as follows:
In the scripts' directory, create/edit a text file containing the common content of your message.
The process for uploading file templates is common for letters to submitters, reviewers, and session chairs. Details are in the section above on creating/uploading letter templates.
Individualized content
At any point in the message where individualized content is to be inserted, create a new line and select and type one of the "flag strings" listed below.
NOTE: The dollar sign MUST be in the first position of the new line and the string identifying the type of content must be typed exactly as it appears below. There can be NO SPACES in the string or between the '$' and the string identifier.
At successive points where individualized content is to be inserted, create a new line and again insert an appropriate "flag string" on a line by itself.
The currently available "flag strings" are:
(A sample message text file can be found at "reviewerUpdateRequest.txt", in the scripts' directory.)
Send the message
Notes:
For tracking purposes, a message displays as each e-mail message is sent.
During reviewer registration, the most common reviewer queries involve
Reviewer deletion is covered in in the next section of this document.
Finding reviewer information, given a last name or number, may be
accomplished by the third option within dbAdmin.asp, the main
administrative interface to the system. Just supply the administrative
password and submit to obtain a form for the name or ID, and click submit
again for the desired reviewer information.
From time to time, a reviewer asks to be removed from the SIGCSE database. Potentially, this requires removing records from several database tables: Reviewer, ReviewerSubject, Panel/Paper/SpecialSession/WorkshopRatings, and Panel/Paper/SpecialSession/WorkshopReviewerAssignedSubject. (In most cases, only the first two of these tables are involved, as a reviewer is unlikely to asked to be removed immediately after completing reviews.)
Script deleteAReviewer.asp performs all these updates. To use this form, begin with the "Delete a Reviewer" option on the administration form dbAdmin.asp. The script then asks for a last name or reviewer number. Supplying this information yields a table entry for checking. Clicking the "Delete" button then performs the operation.
Notes:
When several reviewers have the same last name, entering a last name displays all matching candidate records, each with its own "Delete" button. Deletions can be made only one reviewer at a time, using the individual's "Delete" button.
The deletion operation is not reversible once the "Delete" button is clicked for a specified reviewer. The database is not changed when an individual is displayed; the removal occurs when "Delete" is clicked.
After a proposal deadline passes, submissions should be disabled. This is accomplished by changing the SubmissionsAllowed field for the relevant categories (e.g., paperSubmissionsAllowed, panelSubmissionsAllowed, etc.) from "Yes" to "No".
An overview of the Permissions table and details on changing specific permissions are given in the section above on Deadline Enforcement and Permissions.
Since information on all papers and all potential reviewers is available in the database, the assignment of reviewers to papers can be done online. The process typically follows two steps:
Both the tentative assignment and the review may be done using forms and scripts accessible from successive options on the administration form dbAdmin.asp.
To tentatively assign reviewers to proposals, use the "Assignment of Reviewers to Proposals by Chair" option on dbAdmin.asp.
Reviewer Pools: The assignment of reviewers can draw from two pools of people:
Subject Categories: SIGCSE 1999 Program Chair, Bob Noonan, and others have suggested that proposal assignment begin with proposals from categories with few reviewers. (There is considerably more flexibility when proposals fit into subject categories with many reviewers.) To start, click on the "View data" option on the assignment page. This displays the number of proposals and the number of reviewers available for each subject area. Clicking on a subject then displays all proposals, which have been classified by the submitter as fitting in that subject area.
Assignment for a Specific Proposal: Clicking on a specific proposal gives some information about the proposal and basic information regarding potential reviewers. The listing of potential reviewers is ordered as follows:
In addition to a reviewer's name, the assignment form displays the reviewer's institution, state/country, and number of previously-assigned reviews, together with a flag ("yes", "no") showing if the reviewer has previously been assigned to the current proposal. As one might guess, Button "Add" allows a new reviewer to be assigned the current paper to review, while Button "Delete" removes the assignment of a current reviewer of a paper.
After assignments have been made or modified, click the "Update" or "Update and Next" button at the top of the page to make the changes in the database.
Various assignment algorithms can work well in picking reviewers for proposals. Here are some suggestions.
Guidelines used for SIGCSE 2004:
Additional Suggestions
Since the assignment form includes both institution and state/country information, checking the guidelines during the assignment process is easy.
Once assignments have been made, it may be useful to check them for adherence to the above guidelines. Also, one might want to check that reviewing has been spread out over the reviewers available. Both of these tasks may be accomplished with dbAdmin.asp.
To check how many reviews have been assigned for each proposal, use the "Obtain Data on Proposals" option on the administrative form dbAdmin.asp. The first option indicates gives the number of assignments per proposal in ascending order.
To check specific assignments, use the "Review proposal assignments" option on dbAdmin.asp. After submitting your password, specify an initial proposal or reviewer to review. Buttons then allow sequential review by proposal or reviewer (respectively).
To check the breadth of workload for each reviewer in the database, use the "View Database" option from dbAdmin.asp, and view the "Reviewer" table. Clicking once on the numRevs field will sort the reviewers in ascending order by the number of reviews assigned to them. Click a second time orders in descending order.
While each Program Chair will make her or his own policy decisions, it seems highly desirable for each reviewer to have at least 1 or 2 reviews, so no reviewer feels left out.
After all reviewing assignments have been made and checked, the Permissions table should be updated to allow reviewers to view those assignments and to enter their reviews. This is accomplished as follows:
An overview of the Permissions table and details on changing specific permissions are given in the section above on Deadline Enforcement and Permissions.
Once reviewers are assigned to proposal, the e-mail to reviewers script may be used to notify reviewers of their assignments via e-mail. Reviewers access the proposals themselves as part of the reviewerViewing.asp form.
All reviewing is done online, with reviewers using an online reviewer form. Scripts handle the recording of the reviews as they come in.
Toward the end of the reviewing process, conference leadership may want to check whether or not each proposal has received a minimum number of reviews. If some proposals are short, additional reviewers may be assigned. Note that most reviews come in close to the reviewing deadline; several may arrive a few days late, if review submission is still allowed. This suggests that the assignment of any additional reviewers may need to be done late — after the reviewing deadline.
To determine how many reviews have been submitted for each proposal, use the "Obtain Data on Proposals" option within dbAdmin.asp. The first two options for reviewer data available include:
The option to "Obtain Data on Proposals" includes several additional listings as well related to ratings — useful after reviews are in and decisions must be made regarding the acceptance and rejection of proposals. These listings are discussed later in this document under Summary Reports of Proposal Ratings.
After the reviewing deadline passes (perhaps by a few days for safety), review submission should be disabled before the conference leadership discusses proposal acceptance and rejection. In particular, when reviews can be submitted, reviewers can change their current reviews and ratings. Since acceptance decisions should be based on final reviews, disabling the submission of reviews prevents possible changes after acceptance discussions begin.
To disable submission of reviews, edit the Permissions table, replacing "Yes" by "No" for the relevant ReviewsAllowed field (e.g., paperReviewsAllowed, panelReviewsAllowed, specialSessionReviewsAllowed, etc.)
An overview of the Permissions table and details on changing specific permissions are given in the section above on Deadline Enforcement and Permissions.
In recent years, the SIGCSE community has engaged in an extensive dialog regarding reviewing and the process for deciding which proposals should be accepted. Unfortunately, misinformation abounds, particularly as some vocal individuals apparently believe some of their rejected submissions are better than some accepted submissions of others.
At a basic level, acceptance decisions must depend upon data and upon a high-level vision of constraints for having a balanced program. (To illustrate the complexity involved, note that an uncritical policy based solely upon ratings might result in the acceptance of many CS1 papers, but few papers for upper-level courses — many more CS1 papers likely are submitted than those for upper-level courses.) Ultimately, analysis of submissions draws from three basic sources:
Within a submission category, conference leadership or the Chair for that category can obtain a listing of all submissions, with links to the submissions themselves, using the "View Index of Proposals" option within dbAdmin.asp.
This option also provides a mechanism to check that the submitted files are in pdf format. (Although the full pdf is not parsed, the file header is examined for basic pdf format.)
A complete report of all reviews for a proposal is available from dbAdmin.asp under the option, "Review Ratings and Comments of Reviews". For better or worse, using these reports is largely a labor-intensive process. The following listing provides some suggestions to support this process.
Using the "Review Ratings and Comments of Reviews" on dbAdmin.asp, start at submission 1 and iterative print each review. (While this is somewhat tedious, it goes reasonably fast and provides a helpful basis for what follows.) The reviews for each submission should be on a separate sheet of paper.
Read each review for consistency. Sometimes a reviewer will say wonderful things about a proposal and give a low rating. Sometimes the opposite happens. In either case, one might hypothesize that the reviewer reversed the meaning of the ratings.
Read each review to identify any obvious errors or misconceptions. For example, a reviewer may indicate that one subject is not appropriate for a conference, although the conference leadership would like to promote that topic. As a second example, a submitter may omit citations in a bibliography if those might identify the submitter in a blind-review setting. (If papers are to be anonymous for reviewing, names and citations might be removed.) In some cases, a reviewer might then fault the proposal for leaving such references out. Such anomalies must be identified by reading reviews.
Summary information on proposals may be found through the "Obtain Data on Proposals" option of dbAdmin.asp. Since one outlier rating can affect averages significantly, the system provides listings based on several computed averages:
In addition to averages, several listings highlight submissions for which ratings seem to have a high variability. These submissions may require special attention:
Additional attention also may be due to reviews in which reviewers wrote confidential comments to the Program Committee or for which a reviewer reported low familiarity with the topic:
Ratings Variability: An analysis of ratings for SIGCSE 2000 (and reported at ITiCSE 2003) indicates a natural variability in overall ratings between 0.8 and 1.2 on a scale from 1 to 6. More precisely, when a proposal is generally considered rather good or rather bad (either extreme), reviewer ratings vary statistically about 0.8. For intermediate proposals, the variability is statistically about 1.2.
Statistical analysis (based in part on the ratings of 10 papers sent to 100 reviewers each) suggests that fine variations in proposal ratings are likely not significant.
Selecting Proposals: After adjusting ratings to reflect anomalies (as described earlier), a common practice is to divide proposals into categories: accept, maybe, and reject. Based on the statistical study, the "accept" and "reject" categories should have average ratings 1.2 or more apart. Beyond this initial division, acceptance often involves the program's balance, session coherence, and other factors.
Sometimes the conference leadership (or a submitter) may decide that a proposal may work better as a different type of session. For example, a panel might be reclassified as a special session or a workshop. This is accomplished as follows:
Select the "Change Type of a Submission" option in the top part of the administrative form dbAdmin.asp.
Indicate the current number of the proposal, its current category, and the desired category. Then click "submit".
Although conversion of materials in tables is automatic, there are two potential difficulties:
The organization of papers, panels, etc. into sessions is a manual process.
Final session descriptions should be entered manually into the DayTime, Session, and SessionPaper tables within the database. The following details reflect practice from recent conferences. Some changes are anticipated to reflect the new table-driven nature of the conference software.
Download the full database:
Download the full database using ftp to the server (e.g., db.grinnell.edu).
Contact the system administrator regarding the username and password.
With your copy of the file, you will need to revise numerous tables. In some cases, data from past conferences may provide a template for what appropriate entries might look like.
While the timeId system is a bit tedious, this provides a definitive way to determine the sequencing of time slots.
The event field can be any text -- as you wish it displayed in a browser. Thus, you can add html links to Web pages, if you wish. (This is how Workshop and Keynote information are sometimes included.)
Here are some additional notes regarding the connection of the DayTime table with the Session table:
For example, consider the following database entries:
| timeId | weekday | startTime | endTime | dateInMonth | event | location |
|---|---|---|---|---|---|---|
| 120 | Thursday | 10:30:00 AM | 11:45:00 AM | 3/8/3007 | Technical Sessions | As assigned |
| sessionId | timeID | Room | sessionTitle | reviewerId | type |
|---|---|---|---|---|---|
| 120 | 120 | Ballroom A | Teaching Tips We Wish They'd Told Us Before We Started | 0 | Panel |
| 121 | 120 | Ballroom B | Learning Solutions for the First Year | 0 | Paper |
| 122 | 120 | Ballroom C | Recruitment and Retention of Computing Students | 0 | Paper |
| 123 | 120 | Ballroom D | Artificial Intelligence | 0 | Paper |
| 124 | 120 | Ballroom E | Vendor Session | 0 | Vendor Session |
| 125 | 120 | Meeting Rm 1-3 | Funding Opportunities in Computer Science Education at the National Science Foundation | 0 | SpecialSession |
| 126 | 120 | Meeting Rm 4-5 | Web-based Technologies | 0 | Paper |
| 127 | 120 | Meeting Rm 6-8 | Mechanics of Undergraduate Research at Liberal Arts Colleges -- Lessons Learned | 0 | Panel |
These tables produces the following sample entry in the on-line program (from the SIGCSE 2007 Symposium, although links not shown):
| Thursday, March 8, 2007 10:30 AM to 11:45 AM | ||||||
| Paper Session: | Room: Ballroom B |
Learning Solutions for the First Year |
| Paper Session: | Room: Ballroom C |
Recruitment and Retention of Computing Students |
| Paper Session: | Room: Ballroom D |
Artificial Intelligence |
| Vendor Session: | Room: Ballroom E |
Vendor Session |
| Special Session: | Room: Meeting Rm 1-3 |
Funding Opportunities in Computer Science Education at The National Science Foundation |
| Paper Session: | Room: Meeting Rm 4-5 |
Web-based Technologies |
| Panel Session: | Room: Meeting Rm 6-8 |
Mechanics of Undergraduate Research at Liberal Arts Colleges — Lessons Learned |
For example, if the session table contains records for 10 workshops in a particular timeId -- each with a different sessionId, then only one entry for these workshops will appear on the main online program (with the online program entry coming from the session with the lowest sessionId). That single entry will be a list that will contain a link to a listing of the various workshops. To be more specific, consider the following DayTime and Session tables that use the word "Concurrent" as the first word of the entry field of the DayTime table (also from the SIGCSE 2007 Symposium, with links omitted):
| timeId | weekday | startTime | endTime | dateInMonth | event | location |
|---|---|---|---|---|---|---|
| 50 | Wednesday | 7:00:00 PM | 10:00:00 PM | 3/7/3007 | Concurrent Workshops | As assigned |
| sessionId | timeID | Room | sessionTitle | reviewerId | type |
|---|---|---|---|---|---|
| 50 | 50 | Meeting Rm 1 | Workshops 1-10 | 0 | Workshop |
| 51 | 50 | Meeting Rm 2 | Workshops 1-10 | 0 | Workshop |
| 52 | 50 | Meeting Rm 3 | Workshops 1-10 | 0 | Workshop |
| Et cetera | |||||
| Wednesday, March 7, 2007 7:00 PM to 10:00 PM | |||
| Workshops: | Room: As Assigned |
Workshops 1-10 | |
The ReviewerId field identifies the person who will chair this session (e.g., the chair of a paper session).
Entries in the "Type" field indicate the submission category. Possibilities include:
SessionId's should be unique, and sequencing within a time slot will determine the order that sessions are displayed in the online program.
Once the program is entered, script program.asp produces an online summary of the program, together with information about individual sessions. Separate links to listings of panels, special sessions, or workshops should be inserted into the script.
Since a draft program is available online as soon as it is entered, the conference committee might be asked for scheduling feedback before the program is announced to the community at large.
After acceptance decisions have been made, submitters and the reviewers of proposals should be allowed to see the reviews of those proposals.
To allow the reviewing reviews, edit the ReviewViewingAllowed field of Permissions table, replacing "No" by "Yes".
An overview of the Permissions table and details on changing specific permissions are given in the section above on Deadline Enforcement and Permissions.
After acceptances are mailed, authors and proposal submitters have a some time to revise their work, based on reviews, recent research, reflection, new experiences, etc. Typically, the acceptance letter contains a deadline for inclusion of revisions within the Proceedings.
To allow the updating of accepted submissions, edit the FullUpdateAllowed field of Permissions table, replacing "No" by "Yes".
When the deadline for revision for the Proceedings is reached, of course, this field should be set back to "No".
An overview of the Permissions table and details on changing specific permissions are given in the section above on Deadline Enforcement and Permissions.
In principle, once each proposal is marked as Accepted, Rejected or Poster, a script could use that information to send e-mail to proposers. However, such an approach might result in a proposer getting the incorrect response due to a simple typographical or data-entry error. Thus, the system utilizes an extra step to allow checking.
Notification of submitters regarding each proposal's acceptance proceeds in four main steps:
For each type of proposal (e.g., panels, papers), an administrative script generates tables of accepted proposals, rejected proposals, and proposals which might be resubmitted as posters.
The conference leadership reviews special database tables in order to review the categorization of proposals as accepted, rejected, or poster candidates.
The conference leadership updates permissions in selected tables, so that authors and reviewers will be able to view the reviews, and so that authors can review their papers.
An administrative script allows the actual sending of e-mail. One letter can be sent to submitters with accepted proposals, a second letter to submitters with rejected proposals, and a third letter to those whose proposals are identified as being good candidates for posters. In all cases, the e-mail sends notices to the primary submitter only; collaborators do not receive notification.
While snail-mail acceptance letters might be sent to primary submitters upon request, these letters are not sent automatically.
Details of each of the above steps follow:
Update Accepted, Rejected, and Poster Proposal Tables:
Using the bottom half of the administrative form dbAdmin.asp, select the type of submission, enter the password, and choose the option to "Update Accepted, Rejected, and Poster Proposal Tables". Using the resulting Web interface, select each type of table and click "Update Table". The overall task requires an update of the "accepted-proposal", "rejected-proposal", and "poster-proposal" tables for each category of submission (e.g., panel, paper).
Check the Accepted, Rejected, and Poster Proposal Tables:
Use the top option in the administrative form dbAdmin.asp to "View Database Tables." Then select "View Database" (rather than "update or delete").
A separate table appears for each category of submission and for accepted sessions, rejected sessions, and poster-encouraged sessions. Thus, a conference leader checking papers would check the AcceptedPapers, RejectedPapers, and PosterPapers tables; and a leader checking panels would check AcceptedPanels, RejectedPanels, and PosterPanels.
If errors are found, the basic database tables should be corrected, using the scripts regarding Organizing Accepted Proposals into Sessions. After making corrections, rerun the scripts to update accepted, rejected, and poster proposal tables. Do not try to edit the AcceptedProposal, RejectedProposal, or PosterProposal tables, as these do not correct the fundamental errors in the database.
Use the View Database Tables of dbAdmin.asp to allow viewing of reviews by proposers and reviewers, and to allow proposers to revise the full versions of their submissions. (In the case that a category utilizes blind reviewing, anonymous versions cannot be changed, so the system can maintain a record of the version that actually was reviewed.
Notify Proposers of Acceptance or Rejection: The actual sending of e-mail to proposal submitters proceeds as follows:
Create a text file template stating the content of your message.
The process for uploading file templates is common for letters to submitters, reviewers, and session chairs. Details are in the section above on creating/uploading letter templates.
Individualized content:
At any point in the message where individualized content is to be inserted, create a new line and select and type one of the "flag strings" listed below.
NOTE: The dollar sign must be in the first position of the new line and the string identifying the type of content must be typed exactly as it appears below. There can be no spaces in the string or between the '$' and the string identifier.
At successive points where individualized content is to be inserted, create a new line and again select and type one of the "flag strings". The currently available "flag strings" are:
Note: For obvious reasons, this option may be used only for
accepted proposals.
Using this option for other proposals will cause the script to crash.
Send the e-mail:
On the bottom half of the administration form dbAdmin.asp, select the option to "Notify Proposers of Acceptance or Rejection". The resulting form will ask you to provide the name of the file containing your message; in most cases, this will be in the conference's main directory. You will also need to enter your name, e-mail address (for the "from" line of the notification e-mail), and the e-mail subject.
The form contains additional boxes for "Begin sending at ID number" and "Stop sending at ID number". To send to all reviewers, leave these fields BLANK. If for some reason the script does not finish, you can re-start the script by entering the appropriate ProposalID number in the "Begin sending at ID number" box.
Near the bottom of the form, select whether letters should go to those with accepted proposals, rejected proposals, or proposals that might be posters. This selection specifies the group to receive the e-mail (based on the corresponding AcceptedProposal, RejectedProposal, and PosterProposal tables).
The very bottom controls message distribution: the message may be sent to the sender (for testing) or to the actual submitters of proposals.
For tracking purposes, a message displays as each e-mail message is sent.
Several scripts support the work of the Proceedings Chair.
program.asp produces an online version of the program, giving the schedule of all events, allowing the Proceedings Chair to view this material online as soon as it is entered.
tailoredProgram.asp produces an online program, including title information for presentations for each session.
dbAdmin.asp provides several capabilities.
"View Proposals" returns a listing of all submitted materials, together with a check that they are in pdf format and a link to each submission.
"Generate a List of all Paper Session Chairs" returns a list of chairs for all paper sessions.
"Obtain Index and Links for all Accepted Sessions" provides an index for all accepted submission, with links to all accepted sessions.
"Obtain Id, Title, and Author/Leader Information for All Accepted Sessions" lists accepted sessions by category, with information on the ID number, the title, and the primary author/submitter.
"Obtain Id, Title, Primary Author/Leader, e-mail Information in Spreadsheet Format for All Accepted Sessions" lists accepted sessions by category, in a comma-separated format that can be saved for use by a spreadsheet.
"Obtain Data on Reviewers" lists all reviewers that have completed at least one review, in a format appropriate for the Proceedings.
Several links from program.asp provide listings of all accepted proposals by submitter or by title within a category (e.g., all accepted panels, all accepted papers).
While session chairs may be selected following many criteria, there may be some advantage in identifying those not otherwise on the program who need some status in order to get travel funds. In this regard, reviewers or submitters of rejected proposals may be given some priority.
To record a Session Chair in the database, the individual must be registered as a reviewer.
Historically, most folks considered for session chairs had already registered as reviewers, so consultation of the reviewer table kept data entry to a minimum.
If a new session chair is not a reviewer, then the person must register as a reviewer using the normal reviewer registration process. If the person does not want to review for future conferences, the person can indicate "No" in the reviewer availability columns.
To designate a person as a Session Chair, the person's reviewer ID must be entered in the reviewerID field of the Session table. This allows the Session-Chair assignment to become visible in the online program.
The actual sending of email to Session Chairs follows these steps:
Create a text file template containing the common content of your message.
The process for uploading file templates is common for letters to submitters, reviewers, and session chairs. Details are in the section above on creating/uploading letter templates.
Individualized content:
A letter to Session Chairs may contain general information and information
specific to the session or chair. At any point in the message where
individualized content is to be inserted, create a new line and select and
type one of the "flag strings" listed below.
NOTE: The dollar sign must be in the first position of the new line
and the string identifying the type of content must be typed exactly as it
appears below. There can be no spaces in the string or between the '$' and
the string identifier.
The currently available "flag strings" are:
$session-time [This outputs the line "Your session is scheduled for xxxx at yyyy." where xxxx specifies the date of the session and yyyy gives the time.]
$session-name [This outputs the line "The title of your session is xxxx." where xxxx is the session title as it appears in the online program.]
$session-chair-entry [This outputs the line "xxxx, yyyy" where xxxx is the person's full name and yyyy gives the person's institutional affiliation.]
$rev-ID-password [This outputs the following:
Your Reviewer ID/Session-Chair ID Number is:
xxxx.
Your password is: #########"
where xxxx is the reviewer ID number of the person receiving the message
and ######### is their associated password.]
$rev-profile-data [This sends the current reviewer profile information]
Send the message.
Use the "Send letter to Session Chairs" option within dbAdmin.asp to
send the message. (This option is location in under "General Database
Administration Tasks" in the top half of the dbAdmin.asp menus.)
Enter the name of the file containing your message and press the "Send
email to specified session chairs" button.
Enter your name, email address and an email message subject.
The "Begin sending at Session ID number" and "Stop sending at Session ID
number" boxes may be useful for testing purposes. To send to all session
chairs, leave these fields BLANK.
If for some reason the script does not finish, you can re-start the script
by entering the appropriate Session ID number in the "Begin sending at
Session ID number" box.
Indicate if you want the message to be distributed to you only (as part of
testing for you to review the letter as it will be received),
or to all session chairs within the range of numbers you specified.
For tracking purposes, a message displays as each email message is sent.
(A sample message can be found at "sampleChairSampleNotification.txt", viewable either via ftp access to the conference home folder on the server or via the template creation facility described above.)
After authors and submitters of accepted proposals have revised their materials, and after session chairs have been determined, the system allows potential conference attendees to view individual papers and session descriptions through the online program. Experiences at SIGCSE conferences and reports from other SIGs suggest that the ability to view these materials generates interest in the conference and promotes attendance.
To enable links to accepted papers/submissions through the online program, edit the AcceptedSubmissionDisplayAllowed field of Permissions table, replacing "No" by "Yes".
After the conference, when the full Proceedings becomes part of the ACM Digital Library, this field should be set back to "No".
An overview of the Permissions table and details on changing specific permissions are given in the section above on Deadline Enforcement and Permissions.
In past years, conference leaders printed transparencies for each session. This allowed display of presenters in each room.
In recent years, the use of transparencies has been inconsistent, but the need for announcing the title and presenters for each room continues. Sometimes a poster is prepared for posting outside each room; sometimes a transparency is prepared for a session chair to display; sometimes an electronic image is created for display by a data projector.
Whatever format is used, session information (title, session chair, presentation titles, presenters) must be created for each room. One approach is to utilize the individual session information available on the Web through the program.asp script.
If a poster will be displayed outside the room, the poster might be created as a magnified form of the online session display. The posters themselves likely will need to be created by a copy center.
If a transparency will be created, the output of the online program may be adequate, either as is or magnified slightly with a copier. Once produced, the slide should be sent to the registration folks for stuffing into the registration envelopes of Session Chairs/Moderators.
Two capabilities within dbAdmin.asp support communication with session chairs for paper presentations.
If an electronic slide will be projected, the output of the online program might be adjusted using a common style sheet.
This part describes files created and/or used in the project. Conceptually, these files may be placed in three basic categories:
Since the shtml pages are static, they can run on any type of Web server. Historically, this has been a Linux server, partially because of the remarkable reliability of the network maintained for the Mathematics and Computer Science Department of Grinnell College by John Stone. However, any Web server could be used, as long as include files are enabled.
All pages and scripts rely heavily on header files and style sheets to maintain a uniform look and feel to the materials. The pages are organized as follows:
shtml include files
Include files: Most scripts expect the following files in an
includeFiles subdirectory
| header.inc | gives formatting for all page headers | |
| sigcse-style.inc | provides style-sheet information for all pages | |
| validateIDs.js | a JavaScript file with libraries to validate ID numbers | |
| validateSubmission.js | a JavaScript file with libraries to validate various type of submission fields |
Image files largely depend upon the styles and headers. However, the .shtml files require images valid-html401.gif and valid-css.png, so that the pages can display that they conform to various standards of the World Wide Web Consortium. Standard include files are now as follows:
| Include file | Overview | Programming Notes | |
|---|---|---|---|
| sigcseBasic-include.asp | sets commonly used, conference-specific variables | usually must be included first | |
| sigcseStyle-include.asp | specifies look and feel of the pages | ||
| header-include.asp | specifies conference page header information | ||
| subject-Display-include.asp | processes subject areas in forms | ||
| dbConnection-include.asp | establishes a connection to the database,
by first logging on as a hidden user
also creates a UserData object for handling user data from forms | must be included before any scripts that require data from the database | |
| setSubmissionType-include.asp | connects with the database to determine information about the current proposal/submission category; sets variables related to submissionType table | should NOT be included for reviewer processing, as their information is not stored in specific proposal/submission files. However, for all proposal/submission processing (e.g., my files for submitting proposals and your scripts for administrative processing of those materials), this must be included after dbConnection-include.asp | |
| security-include.asp | checks the conference leaders' passwords, but does not distinguish among proposal/submission types. | draws on password-include.asp, which contains those administrative passwords. | |
| security-proposal-include.asp | checks passwords for proposal and submissions tasks, allowing chairs of specific sessions (e.g, panels' chair) to access your scripts for those sessions, but not the materials for other types of sessions. Of course, the conference chairs can access anything. | Since the passwords of the specific chairs are given in the database and since the passwords are retrieved as part of setSubmissionType-include.asp, security-proposal-include.asp should be referenced late. | |
| dbClose-include.asp | closes and wraps up variables opened by dbConnection-include.asp | placed at the very end of processing; there should not be other statements of the form databaseConn.close previously. | |
| adovbs-include.asp | sets variables for database manipulation | ||
| vbFileManager-include.asp | sets more variables for database manipulation | ||
| AspUser-include.asp | sets variables needed for file manipulation | ||
| password-include.asp | contains security password information | ||
| privateUser-include.asp | contains private user for file uploads |
Previous sections of this documentation describe various user and administrative capabilities of the system. This section pulls together the various processing flows, identifying which scripts are used for which tasks.
Earlier versions of this documentation presented composite diagrams for submissions, reviewing, and administration, and the reader is invited to view these diagrams in past documents (e.g., see Part III of the 2005 documentation listed in the index of earlier versions).
This current documentation organizes scripts into relatively small categories by function:
Each script category has one or two starting pages that form entry points to an entire family of scripts. The following figures provide an overview of the calling sequence(s) of these various scripts.
This document is available on the World Wide Web as
http://www.cs.grinnell.edu/~sigcse/2009documentation.shtml
|
created 9 July 1999
by Henry M. Walker
with Weichao Ma and Dorene Mboya last revised 21 October 2009 by Henry M. Walker |
| |
| For more information, please contact Henry M. Walker at sigcse@cs.grinnell.edu. |