Peer Assessment

Peer Assessment is an application in the MU IS implementing the pedagogical assessment process where students evaluate each other. In the English-language literature, this concept is known as "peer assessment" or, less precisely, "peer review".

The basic difference between the above-mentioned terms lies in the choice of the so-called evaluators, i.e. the persons responsible for assessing the submitted work and drawing up an opinion. Whereas in the peer assessment process, the submitter and the assessors are typically students and often come from the same set of individuals, in the case of peer review, the assessors are carefully selected from among experts in the field with the aim of providing a qualified and impartial review of the submitted scientific work while ensuring a high degree of relevance. The basic mechanism for the collection and distribution of files and results is identical for both use cases.

There are a number of reasons to incorporate student peer assessment into teaching. The following list outlines at least some of them:

• In the process of producing their assessments, students develop their critical thinking skills, their ability to look objectively at others' work and to produce constructive criticism.
• Students get feedback not only from the teacher, the authority figure, but also from colleagues - their criticism can be more easily accepted by students.
• Evaluators can also be students of a completely different course, e.g. from a higher year, for whom it may be part of their preparation for a teaching career or an opportunity to improve further professional competences.
• The application also allows specifying individual persons as assessors, so that the teacher can distribute the correction of assignments among his/her doctoral students or colleagues who do not teach the course .
• The teacher can easily get more assessments of any number of papers as the number of assessors increases with the number of students.

Peer assessment is a course-specific application, i.e. it is tied to a specific course or courses. The teacher can find the necessary links in the Teacher agenda, in the Teacher's Notebook.

The student has his links in the Student agenda, tab Teaching.

A person who is an evaluator and is not a student or a teacher can find the link on the front page of the MU IS in the E-learning section.

Help for students (submitters and assessors) and for assessors who are not students or teachers of the course is available here.

In the Peer Assessment application

The setting up of the peer assessment is divided into five points (or steps):

1. Name of the peer assessment: Filling in basic information such as the name, instructions for submitters, date the peer assessment will be made available to the submitters (i.e. the date when the peer assessment is displayed in the MU IS to submitters, not when the distribution of files takes place or the assessment begins), setting up the sending of informational emails, specifying the hidden identity of the submitting student(s) or evaluator(s), etc.

   

2. Submission: setting up a new homework vault or selecting from existing ones. In the case of setting up new homework vault, you need to set a date by which students can submit their files. Only students who have uploaded at least one file to at least one of the selected homework vaults are considered as submitters during distribution.

   

3. Evaluator selection: in the simplest case, just turn on the "Submitters are evaluators" mode. Alternatively, you can specify the evaluators as students of courses or seminar groups, or individually specified by UIN (učo). Only those courses are offered for which the teacher has the right. The IS remembers the settings and evaluates the real persons only at the moment of distributing the files to the evaluators. Peer assessment can therefore be set up at the very beginning of the semester.

   If one student submits multiple files (even to multiple homework vaults), the evaluator will have all files from that student. In other words, evaluators are "assigned" submitters, not individual files.

   

4. Set assessment attributes: the teacher must specify the start and end of the assessment and how many times one submitter should be assessed, or how many submitters should be assigned to one assessor. In the "Enable the evaluation of instructor's file" section, it is possible to set the teacher's assessment files (see question here). All settings must be feasible at the time of distribution or the distribution will fail. This step can also be used to set a later date for the results to be published to students than the end of the assessment. If the results are transferred to notebooks, the results will be published on this date.

   

5. Create evaluation form: The teacher has an advanced WYSIWYG (informal "what you see is what you get") evaluation form editor. Here, the teacher can specify a series of questions that he/she then guides the evaluator through in creating their evaluation. There are questions with one or more possible answers, free text, marking and scoring. For each question, the teacher can specify whether its value is to be stored in the student's notebook.

   

Peer assessment allows for the following basic combinations:

• The submitters and evaluators are groups
• Submitters are groups, evaluators are individuals
• Submitters are individuals, evaluators are groups
• The submitters and evaluators are individuals

In the application

• Without sorting into groups – Individual students are the submitters or evaluators, not groups.
• Sort students randomly – Peer assessment will set up a topic list with a sufficient number of groups (topics) and assign students to them randomly.
• Students sign in on their own – Peer assessment will set up a topic list with enough groups (topics) but will not sign up any students.
• Sort students into groups manually – The teacher (i.e. the peer assessment administrator) creates groups by entering a list of students‘ UIN (učo) into the peer assessment editing window. He/she can also directly edit the names of the future groups.
• Použít existující rozpis (i.e. Use an existing topic list) – The teacher selects a topic list that he/she has previously created in the Topic Lists application for the course. The topic list will be linked directly to the peer assessment. A single topic list can only be linked to one peer assessment using this option, and only topic lists that can be linked in this way are offered.
• Copy existing groups – Here the teacher selects a topic list that he/she has already created in the course in the Topic Lists application. However, the peer assessment will not directly link this topic list, but a one time copy of it. If groups of an existing topic list are to be used for peer assessment a second or more time, this option must be selected.
  Any changes to the group names, student sign in, etc. must be made on these copied groups, not on the original topic list, if they are to be reflected in the peer assessment. If students are allowed to modify the existing topic list, they must be informed.

To divide students into groups, peer assessment uses the Topic Lists application in the course Teacher's Notebook, where the groups are the topics in the course topic list and the students registered for the topic are members of the group so named. In most cases, however, there is no need to set up anything in Topic Lists.

Members of the groups can only be students of the course in which the topic list is located or of the courses between which it is shared. If peer assessment is set up for multiple courses, groups are created (or set up) so that students from all courses can be enrolled in them (details below).

At the time of the start of the evaluation, the topic lists with groups linked to the peer assessment are locked. This means that they can no longer be edited and will only be available in view mode.

Similar to peer assessment for individuals, the setting that the groups submitting are the same as those they are evaluating can be used for groups. Both the submitters and the evaluators assigned to the groups can be students from different courses or seminar groups.

The specifics of the groups submitting:

• If the submitters are grouped, it is highly desirable to check the option for notifying the submitter by email in step 1 "Name of the peer assessment", because only then will the application send the students the information that, they have been signed in a group for peer assessment or that they should sign in one.
• Submission groups are formed on the date and time that the peer assessment is published to students and students are subsequently informed of their signing in or obligation to sign in if this is set in point 1. It is therefore desirable to set the date of publication of the peer assessment well in advance of the submission date.
• A student is considered a member of a group by the application if he/she joins the group (is enrolled) no later than the start of the evaluation of the files.
• Application-generated submission groups are created for courses whose students are allowed to upload into one of the linked homework vaults (if they are still allowed to upload) or whose students have a file loaded in one of the homework vaults (if it is an already closed homework vault). The homework vault must be located in the study materials of the course for which the peer assessment is created. If the peer assessment is to be a group assessment for multiple courses, it must be shared, see Is it possible to create a peer assessment that is common to multiple courses?
• A group is considered a (successful) submitter by the application if at least one member submits at least one file to at least one of the connected homework vaults. Such a file is considered to be the work of the whole group.

The specifics of the evaluator groups:

• The groups of assessors created by the "Sort students randomly" and "Sort students into groups manually" options are created only at the time of distribution of the files to the assessors. Evaluator groups created using the "Students sign in on their own" option are created at the date and time the peer assessment is published to students.
• A student is considered a member of a group by the application if he/she joins the group (is enrolled) no later than the start of the evaluation of the files.
• The whole group of evaluators has one common evaluation form and it is sufficient for one member of the group to enter the evaluation, as it is assumed that it is the result of joint work. It is recommended that the whole group meets for the evaluation at one computer (or via video call) and one member enters the evaluation for all. The evaluation form does not support simultaneous editing by multiple members of the same group.

Yes, when setting up a peer assessment, you can create one or more new homework vaults or use existing ones (even closed ones). When using existing but not closed homework vault, it is necessary to make sure that the deadlines for the submission of the files and the start of the evaluation are set correctly.

When using already closed (and therefore files-filled) homework vaults, it is advisable to cancel sending an email to the submitters (a setting in the first step of setting up the peer assessment), which no longer makes sense in this case.

Students who do not submit their first file until after the distribution of files has taken place are not considered submitters. If students are allowed to change their file after the assessment has started, the assessor will always be offered the version of the file that is current at the time he/she decides to assess it.

Once the results are posted, they will be found both in the Peer Assessment application in the submitter role and in the notebooks if the teacher has set the results to be transferred to them.

Almost anyone who is active in the MU Information System and is logged in. Therefore, the entry for evaluators is also from the front page outside the student or teacher agenda. Persons who are not students of the course or seminar group need to be entered through their UIN (učo).

Help for assessors who are not students or teachers of the course is available here.

In peer assessment, it is possible to hide the identity of both evaluators and submitters. Setting pseudonymity will ensure that the name and UIN number (učo) of the evaluator/submitter is not displayed anywhere in the application and that the submitted files are renamed so that they cannot be linked to the file name in the homework vault. However, the Peer Assessment application does not delete metadata from the submitted files or check their content. Therefore, if the submitter signs directly into their file or leaves metadata in the file (e.g., the author of a Word document), even with pseudonymity set, the evaluator can still reveal the identity of the submitter in peer assessment.

Course teachers always know the identity of submitters and assessors.

Pseudonymity is set when a new peer assessment is created:

or when editing the peer assessment:

Yes, after the distribution is made, clicking on the peer assessment tile in the administrator role will bring up the "Overviews" link, which lists submitters and evaluators. This displays summary information on how submitters and evaluators are fulfilling their formal role responsibilities. These lists can be exported and further processed by the teacher. At the same time, the teacher can see, for example, links to the assessments themselves, the file submitted and, where applicable, the totals of marks awarded or received.

In addition, you can see how they have evaluated or been evaluated via the "Submitter" and "Evaluator" links in read-only mode. All these options offer the same data, just grouped differently. The teacher can always see the identity of the assessors and submitters here, regardless of the peer assessment setup and his/her possible role in it.

The teacher cannot change the assessment for the assessor. However, if the results are exported to notebooks, e.g. for the purpose of calculating the average and the final classification, he/she can modify these notebooks directly and at his/her discretion.

Yes, by selecting "Delete" on the peer assessment tile (as an administrator). This way you can cancel any incorrect or ongoing peer assessment. However, it cannot be deleted after the end date of the evaluation. If a peer assessment has already been published to the submitters or is already in progress, an informational email will be sent to submitters or evaluators after cancellation.

Canceled peer assessments can be found via the "Create a copy" option on the "Not saved" tab. The settings from a deleted peer assessment can thus be used as a basis (template) to create a new one, and various unsaved versions can be found in a similar way.

No, you can't. Peer assessments cannot be cancelled once the evaluation is completed. Disclosure of results to students can be temporarily delayed, but this setting can only be changed for upcoming or ongoing assessments.

Here it makes a difference whether all students should have peer assessment in one (foreign) language or whether some students should have the same assessment in Czech and some in English.

If all students are to have one language version of the peer assessment, whether in English or in another language, it is sufficient to fill in the version of the form otherwise designated for Czech (i.e. the main language of the peer assessment) in the required language, see question 4 above. All setting items, including the evaluation form, can also be filled in languages that use a font other than the Latin alphabet. (The numbers in the distribution priority setting or in the points allocated must always be Arabic.) Similarly, evaluators can use different languages.

If some students are to have their peer assessment in Czech and some in English, depending on which language they have switched to in the IS, they need to click the "+EN" icon in step 1 of the setup and fill in the items as described. The form for the evaluator in step 5 should also be filled in both in Czech and English.

Information emails to submitters and assessors are sent in the language version to which they have switched the IS.

Yes, if this is set in the peer assessment using the attributes "[ ] Evaluate the files of the instructor" and "[ ] all evaluators" (Step 4 of Set assessment attributes). If neither of these two attributes is checked, files uploaded by teachers to linked homework vaults are ignored during distribution. That is, the teacher cannot upload a file on behalf of a student (e.g., sent by email) into the homework vault.

If the attributes are checked as follows:

If the attributes are checked as follows:

If only teacher files are to be evaluated, the frequency should be set to 0 times as the distribution priority.

For example, the requirement for assessors to be provided with a model assignment can be addressed in this way.

The current application capabilities do not yet allow to solve the requirement when the teacher wants to distribute x files among the students so that one student always receives just one file from the teacher (each evaluator is always assigned all files from the teacher).

You can change the date up to which an assessment can be made (in step 4 when setting the assessment attributes) for a peer assessment, but only if the end of the assessment has not yet occurred. That is, beware you cannot extend the interval after the evaluation end date has already passed. The date can only be changed to one in the future and this change is irreversible. Along with this, the time of publication of the results must be adjusted accordingly.

Other settings related to the evaluation cannot be changed once the evaluation has started.

When creating a peer assessment, you can also create a peer assessment using the "Create a copy" button. The teacher can then select from the existing peer assessments of the course (tab "Proposed") and from the peer assessments in his/her other courses (tab "By term"). In the created copy, besides the name and the date of publication (step 1), it is always necessary to modify the connection to the homework vaults (step 2) and the data related to the evaluation (step 4). The "Not saved" tab allows you to use the settings of deleted or unsaved (i.e. in-progress) peer assessments.

Yes. If it is necessary to copy all peer assessments from the previous semester (or another course) to the new semester, you can use the "Sharing and copying study materials" option.

The target course must have empty course materials, so this option is typically used at the beginning of a new semester. Peer assessments copied in this way will be displayed as incorrect/incomplete in the target course after copying.

Peer assessments are shared between all courses that are selected in the Teacher's Notebook when they are created or modified.

These courses can be selected in the Teacher's Notebook via the "other courses" option, then need to be checked all of the courses among which the peer assessment is to be shared and confirmed with the "Selected" button. All peer assessments that the teacher creates (or clicks "Edit" and then saves) with these selected courses will be shared between them. If the teacher selects "Close without saving", sharing between courses will not be set up.

Sharing of peer assessment between courses cannot be abolished. If sharing is not desired, it is possible to delete the peer assessment and create a new peer assessment (i.e. a copy of it) from its settings. However, this is not always necessary. The submitters are always students of the course (in which the peer assessment is located) who have submitted at least one file to the attached homework vault (or have the right to the attached homework vault), and the evaluators are determined by the settings in step 3. Other students do not have access to the peer assessment and do not receive any informational emails, they can only see it in their Student agenda, under the Teaching tab.

All types of files that can be uploaded to the homework vault in Study Materials can be used for peer assessment. There are no other limitations to peer assessment.

When evaluating a video (or audio recording) created by students, the format they use may be problematic if it cannot be played directly in a browser, or if it can only be played on devices with a particular operating system, etc. As a precaution, we recommend asking students to convert the recorded video (or audio) to HTML5 format, which can be played directly in the browser without downloading, even in peer assessment. If necessary, the teacher can also do this conversion (but not upload the file in the homework vault).

The procedure for conversion to HTML5 format is described in detail in the Help here. Conversion can be done for multiple checked files at the same time. It should be taken into account that the conversion may not take place immediately, but may take some time.