marking pdf assignments

Marking essays and assignments online with A.nnotate

If you are going paperless, "less paper", or paper-light, then A.nnotate can help with the marking!

A.nnotate provides an easy and efficient way to mark essays and assignments online. Documents can be uploaded or emailed in as PDF file, MS Word documents, or in OpenOffice formats. It can also be integrated with Moodle and other learning management systems to provide feedback for online assignments.

You can select text and add a tag or write a note to give feedback. It is just like using a highlighter pen or writing notes in the margin, except you do it online, from any browser with no software to install.

Several tutors can mark the same document if necessary, even at the same time. And when it is ready, the student can be given access to the annotated on-line version.

A.nnotate is a low-cost standalone system that is easy to use and can be used today for online assessment. It can also be incorporated in existing document management systems - just get in touch for details.

Sign Up Now - all new accounts come with 150 free credits per month (enough for 30 online pages). Find out more... about other ways to use a.nnotate.

The A.nnotate Document annotator runs in the browser. There are no plugins or extra software to install, and it works seamlessly on Windows, Mac OSX and Linux. You do not need to wait for Adobe Reader , Acrobat or even Flash to start up as the A.nnotate PDF Annotator simply requires standard web browser technology. Everyone can annotate the same shared read-only copy of the PDF document online, so you can eliminate the usual problems associated with PDF comments - no need to merge and keep track of different edited versions send as attachments.

Other benefits and applications

A.nnotate is not just for online marking - you can use it give or provide feedback on drafts of documents and in everyday web research.

Your Personal Index of Annotations

Annotate text with notes, tags and replies, the a.nnotate pdf annotator displays pdfs in high quality.

Grade Assignments Using Annotate PDF

This tutorial shows how to create an assignment that uses the annotate pdf tool which allows instructors to grade assignments without downloading them..

Note:  It is not recommended using this tool to provide feedback for an assignment that students will need to revise, work on further, etc. This is because students will receive the feedback in the form of a PDF, which they won't be able to edit.

To create an Assignment with Annotate PDF:

• Follow the instructions to Create an Assignment
• When you get to the settings choose Submission Types, Feedback types, Grade settings, as described below:

Submission types

• Make sure the File Submissions  box is selected.
• In the  Accepted file types area, add Documents which includes .pdf files. (Note Annotated PDF grader will also allow you to annotate Word docs).

Feedback Types

• After you've finished setting up the Assignment, click to view it.

Marking with Annotate PDF

Graders can highlight, write, type, and add stamps and post notes in a PDF document submitted by a student when Annotate PDF is enabled. They can use a mouse, trackpad, or WACOM tablet to make handwritten markings when using a computer or an iPencil, stylus, or finger when grading on an iPad or other tablet.

• Click the assignment you would like to grade from within your course

The submission(s) will open one at a time in an annotation window as shown below. Navigate through the student submissions using the grading tools to annotate directly on the assignment submitted.

Select a tool from the annotation toolbar at the top of the window to mark the submission.

• If the document in annotation window is blank, the conversion to PDF failed (for a file submitted in a different format) or Annotate PDF could not read the PDF file. You will need to download the file using the link in the pane to the right of the annotation window and add annotations outside of Moodle.
• To save a copy of your annotations and feedback outside of Moodle, click the Download Feedback PDF link beneath the Feedback Comments box.
• The Annotate PDF feature works with PDF and Word doc files . If students submit files in other formats, Moodle will try to convert them to PDF so that you can mark them, but this conversion process does not work reliably. For best results when using Annotate PDF, have your students save their assignments as a single PDF file for upload.

Tip Deleting a Stamp

• Click on the stamp you want to delete. The stamp will highlight a little trash can icon appears.
• Click the trash can icon to delete the stamp.

Another perspective - Moodle: Grade assignments with Annotate PDF

Award-Winning!

Digital Grading Assignments Made Easy

Digital grading 101: how to use a digital worksheet maker to create easy-to-use auto-graded assignments and assessments.

Are you feeling weighed down by the stack of grading you have? Free up nights and weekends by automating your grading in your classroom. We’ve come up with a guide to jumpstart your digital grading journey. 

What is Digital Grading?

A digital-graded, or auto-graded assignment, is an assignment that is instantaneously graded. Often teachers use tools that automate the grading process. They create an assignment and then put in a key that the app uses to grade submissions. Self-graded assessments and worksheets can be partially or fully graded by auto-grading software. TeacherMade converts all of your assignments to digital worksheets and gives you the option to auto-grade interactive digital activities. 

What types of assignments and assessments can be Digital graded?

• Homework : Teachers use homework for practice. So why not automate it? You can even set up your assignment options so that students can get instant feedback, so they know how they are doing at the moment.
• Worksheets : Are worksheets dominating your grading time? Much like homework, teachers spend a lot of time grading worksheets. Free up your time grading and focus on things that matter more: instruction, meaningful feedback , and relationships. 
• Formative Assessment : Do a quick check with your students and get instant feedback on their learning journey.
• Quizzes : Most quizzes can be automated. Just put in your key with TeacherMade . 
• Tests and Assessment : Multiple-choice tests are perfect for self-grading. But you can also grade portions of tests and use self-grade for the rest.
• Bellringers : Start your day off in the right way with auto-graded bell work. It’s easy to have students log in to their digital bell work, and then you can quickly receive the feedback.

Benefits of Digital graded Assignments

Once you implement auto-scoring in your classroom, you will be amazed at all of the benefits you will experience. 

Save time with self-graded assignments and assessments

Many teachers are reluctant to start using digital assignments that self-grade because of the learning curve. It can be frustrating to learn a new app or software only to realize it’s not the right fit in your classroom. And each time you use it, there is time spent getting it set up just right. But here’s the thing, you’re already spending time typing and editing assignments that you use all the time. You’re also spending time copying the assignment, handing it out, and grading it. Why not cut this time out with some setup time?

With TeacherMade , our app makes it easy to set up auto-grading on all of your assignments. If you can set up a PowerPoint, you can set up a self-graded assignment. Try TeacherMade today . 

Digital worksheets are the key to implementing auto-graded Assignments in your classroom today

If you want to simplify grading with auto-grading, the best way to do this is to regularly use digital worksheets in your classroom. It’s easy to create a TeacherMade assignment. 

• Step 1: Upload your worksheet or assessment file. ( We accept these file types. )
• Step 2: Add fields for student responses.
• Step 3: Include answers to questions that work for self-scoring.
• Step 4: Send links to assignments to complete.
• Step 5: Allow our auto-grade feature to do all of the work from there!

Get immediate data that informs instruction

Your results show up instantly on TeacherMade. Take these results and immediately modify your lesson for an impact that will reach your students. With traditional grading, you may have to wait over a week before getting a clear picture of your students’ knowledge. It can feel too late to correct your course of action by then. 

Intervene faster with faster results

Connect with students that need intervention faster with TeacherMade . You will instantly know which concepts students struggle with while using our self-grading feature. Stop wasting time grading, and connect with students when the material is still fresh on their minds. 

Why use TeacherMade as a digital worksheet maker over Google Forms or Slides?

You may want to use Google Forms or Google Google Slides because it is already connected to Google Classroom. But TeacherMade syncs up with Google Classroom and provides an intuitive and feature-rich app for creating digital worksheets. 

• Stop limiting students with basic online worksheets from Google Forms. With TeacherMade, your online activities will look and feel like traditional paper assignments and tests. 
• Add multimedia like your voice or a video clip to your traditional PDF assignments.
• Grade assignments instantly. TeacherMade allows you the flexibility to edit your answer key as you grade– just like when you grade paper assignments and tests. 
• Our app connects to all of the major LMS platforms: Google Classroom, Canvas, and Schoology.

Use TeacherMade as a Digital grading worksheet and assignment resource

Regain your nights and weekends with TeacherMade. When you digitize your assignments, you gain the advantages of self-grading. This leads to seeing results faster and adjusting your instruction in the moment. Become a more organized and informed teacher with our auto-grading features today. 

TeacherMade transforms teaching materials into interactive digital lessons and saves teachers time with instant autograding. Our platform integrates seamlessly with major systems for personalized student engagement.

© 2024 All Rights Reserved.

Learning and Teaching

• Assessment & Feedback

How to mark an Assignment online in Moodle

Published on: 31/07/2023 · Last updated on: 31/07/2024

Preparing to grade an Assignment

Before beginning marking it’s important to understand how your assignment has been setup, so that you can ensure that you are not showing marks and feedback to students before you are ready for them to see them .

By default, Moodle assignments hide grades and feedback from students through the ‘Marking Workflow’ option.

When you create an Assignment, the ‘Marking Workflow’ setting will be enabled. This automatically hides any marks and feedback that you upload for students until you take specific steps to release the marks and feedback. This is a ‘safer’ option for the requirements of most Teachers .

Guidance on  releasing Marks and Feedback, and sending a notification to students  with a few clicks, using marking workflow can be found below.

If you prefer instead to turn off the marking workflow option you can do so within the Assignment’s settings, under the ‘Grade’ settings.  Note that doing this will mean that any marks and feedback added to Moodle will be immediately visible to students!

If you have turned off Marking Workflow, but you  also  don’t want students to see marks and feedback as soon as they are uploaded, you will need to hide the Assignment link and Gradebook item until you are ready to reveal the marks and feedback.

The guidance below shows how to turn off Marking Workflow (if you wish) and how to hide the Assignment link and Gradebook item:

• Find your Assignment and select Settings

• Scroll down to ‘ Grade ’ settings and expand them
• Set  use marking workflow  to  no
• Save changes

Note: Any marks and feedback you add will now be immediately visible to students!

To manually hide marks and feedback from students you will need to hide the Assignment’s  Gradebook item  and the  Assignment link  (which is often not practical where there are late submissions)

To hide the Gradebook item:

• Go to the  Course Homepage  and choose the Grades tab
• From the Grader Report drop-down menu, choose Gradebook Setup .

• In the row for the Assignment that you wish to hide, in the  Actions  column, choose  Edit  then  Hide .

• Save Changes .
• The Gradebook item will now appear to be greyed out to show that it is not visible to students.

To hide the Assignment link from students:  

• Navigate to the Assignment and, from its  Edit  drop-down menu, choose  Hide .

• You can now begin marking.

Note : When are ready to release marks and feedback to students, don’t forget that you’ll need to make the  Gradebook item  and  Assignment  link visible again.

Lock submissions to prevent changes

Before you begin marking it’s a good idea to lock submissions. This will prevent students from making any changes to their submissions once marking is underway. To do so, navigate to your Assignment and View all submissions .

• Tick the box at the top of the ‘select’ column to select all user submissions.
• From the ‘ With selected…’ drop-down menu choose Lock Submissions .

Note: If you have a large number of students you will need to select the option under the table to ‘show all students on one page’.

Moodle grading interface overview

Moodle has a built-in grading interface that allows you to annotate student assignments, add summary comments and award marks. Student submissions are displayed in PDF form and include a link to the Ouriginal similarity report (where activated).

The image below gives an overview of the built-in grading interface:

• Use the next and previous arrows to move to the next page of the student’s document or choose a page from the drop-down menu.
• Add an annotation to comment (with option to choose background colour).
• Use the arrow tool to select, move or delete existing annotations.
• Other tools such as the pen (squiggly line) for free form drawing or writing, shape tools, line, highlighter)
• Stamp tool/x tool.
• Anchor for an annotation (if you hover over it, it expands to reveal the annotation)
• An annotation
• Click the sub-menu icon in annotation (top right corner) to delete or add annotation to a ‘quicklist’ for the assignment).
• Use navigation to see Previous/Next student’s assignment or filter.
• Add grade here in the right-hand pane.
• Summary feedback can be entered here
• Don’t forget to save any changes made!
• Click to cycle through the options for screen layout

The video below provides an introduction to the grading interface, shows how to annotate student scripts, view Ouriginal reports and award marks. It also details how you can lift anonymity (where anonymous marking is being used) at the end of the marking process.

Marking Assignments Online

Marking a Group Assignment

Where one student has uploaded a file on behalf of her Group, the file will appear against all Group members, and any of these can be marked. On the individual student grading page, an option to ‘ Apply grade and feedback to all members of group ’ will appear. If ticked the grades and feedback will be applied to all group members. 

Allocate Assignments to another Marker

You can use the  Marking Allocation  feature to assign student submissions to a specific marker in your Moodle space.

• Marking allocation  – this allows allocation of markers to work after the deadline.  Marking allocation  depends on  Marking workflow  being enabled.

Marking Workflow  and  Marking Allocation  must be enabled in your assignment settings.

Once these have been enabled, you can allocate assignments to a marker.

• Click on your Assignment link
• Click on  View all submissions

• Tick the box to the left of the student whose assignment you want to allocate, then choose  Set allocated marker  and click  Go .
• At the prompt click  Ok , then on the next page select your allocated marker from the drop-down menu and click  Save Changes .

How to lift anonymity in an Assignment

If you are using Anonymous Marking (blind marking) in Moodle, once you have completed marking, it is necessary to lift anonymity in order to release grades to the Gradebook. Please note that lifting anonymity can only be done once, (you cannot return the assignment to an anonymous state once it has been lifted), so ensure that you have completed all marking before going ahead. 

To lift anonymity.

• Click on the assignment link, then click on  View all submissions
• From the grading actions menu choose  Reveal student identities:

This video also shows the process for lifting anonymity: https://uniofbath.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=dbdb18a7-e4f2-40af-a7fe-acd800a3803c&start=0

Lifting Anonymity

Related Articles

• Anonymous marking in Moodle
• Advanced Grading in Moodle
• Group Peer Review for Teachers
• How to mark an Assignment offline and upload feedback
• Using Ouriginal
• How to create an Assignment

Article Contents

Search form

• Travel & Maps
• Our Building
• Supporting Mathematics
• Art and Oxford Mathematics
• Equality, Diversity & Inclusion
• Undergraduate Study
• Postgraduate Study
• Current Students
• Research Groups
• Case Studies
• Faculty Books
• Oxford Mathematics Alphabet
• Oxford Online Maths Club
• Oxford Maths Festival 2023
• It All Adds Up
• Problem Solving Matters
• PROMYS Europe
• Oxfordshire Maths Masterclasses
• Outreach Information
• Mailing List
• Key Contacts
• People List
• A Global Department
• Research Fellowship Programmes
• Professional Services Teams
• Conference Facilities
• Public Lectures & Events
• Departmental Seminars & Events
• Special Lectures
• Conferences
• Summer Schools
• Past Events
• Alumni Newsletters
• Info for Event Organisers & Attendees
• FAQs / Guides
• Digital marking

Digitally marking students' work

• Accessing RFS storage
• Permissions
• Backup Schedules
• Manipulating PDF files
• Annotating PDFs in Adobe Acrobat Reader DC
• Annotating PDFs in Preview (Mac)
• Annotating PDFs in Okular (Linux)
• Annotating PDFs in Foxit
• Phone scanning
• E-signatures
• Restoring Files
• Version Control
• Network File Transfer
• Dos To UNIX
• Making Archive Files
• Annotating slides in Keynote for iOS
• Annotating slides in PowerPoint
• Accessing home folders remotely from Windows
• Downloading files using Safari

There are several ways you can use to mark students' work if it has been sent to you in PDF format .

Traditionally one might print it out, mark it and then scan  it back in again to send back to the student.

An option more suited to a digital world, particularly one in which people are working remotely, is to mark it directly on (a copy of) the PDF file via a PDF annotation program. This can be done on the screen using a keyboard/typing and a mouse , say, or with a stylus pen on a graphics tablet or a normal tablet if you have one.

There is a large selection of tablets out there which could be used for this purpose, but if you also want to, say, present mathematics with it over a remote teaching session as well, then you will probably get most mileage out of an Android or iOS tablet (e.g. Huawei MediaPad (sub £200 with a precision stylus), Apple iPad, Samsung Galaxy Tab etc).

A further advantage is that many annotation applications integrate with OneDrive which the University provides for free. So you could setup virtual pigeon holes for your students to put their work into, and they will automatically appear on your tablet, for you to mark it, and then the student can take the marked file straight out again. Please note that whilst the university has signed OneDrive off for information security purposes, other cloud services may not be compliant, so please consider the information security implications when storing personal data, such as students' work, in them.

On Android an application which works well for annotation and can also upload into OneDrive directly is Xodo PDF Editor (it also exists for the iOS, but does not seem to work as well). On iOS an equivalent is PDF Expert  - note that the free app gives you the functionality you need, you do not need to buy the costly monthly subscription.

Intercollegiate class work submission and return

Intercollegiate classes are now all run through Moodle where tutors collect and return digital work through the Moodle system.

Submitting and returning work through OneDrive (aka. the "virtual pigeon hole") - college teaching only

A convenient way to get and to return students' work is through OneDrive, as each student has a OneDrive account provided by the University (and associated with their SSO account). This would only apply to college teaching as all departmental work submission and return is now integrated into our Moodle system on courses.maths.ox.ac.uk . An example of how this could be done is:

• Login to OneDrive.
• Create a folder for your teaching, the year, the relevant courses underneath etc, as you see fit.
• In the relevant course folder create a folder for each of your students which will be their virtual pigeon hole.

• It should say "People you specify can edit", and you can now start entering your student's name and it will come up with suggestions for you to select. That will allow the relevant student to upload to the relevant folder and also send them a message that the folder has been shared with them.
• You can now decide whether to mark the work directly in those folders (which would give students "live" access even as you are changing the file), or you could create a separate a folder which is only accessible to you where you can copy all the files you need to mark, do the marking, and then move the files into the students' folders under a slightly different name. This way you will still have the originally submitted work and the marked work in each folder.
• Please note that you should remember to delete the students' work from all your devices and any cloud services like OneDrive once you have finished processing it.

Marking work on a Desktop/Laptop

• You may wish to install the OneDrive client for Windows or Mac  or use the OneDrive Free Client for Linux  (which is packaged for Ubuntu 20.04, but it is recommended it is built from source for earlier versions). You would need to login using your Maths email address and your SSO account.
• Installing the client will sync your students virtual pigeon holes automatically to your computer.
• With or without OneDrive you can mark/annotate the work eg. in Adobe Acrobat DC on Windows/Mac, Preview on Mac,  Okular on Linux or  Foxit Reader on Windows, Mac & Linux or  Xournal++ / Xournal on Windows/Mac & Linux.
• Once you are done you can save/move the files back into the pigeon holes for people to collect, either by putting it in the correct folder, if you have OneDrive installed, or uploading it back to OneDrive through the browser.

Using a Wacom graphics tablet

• If you have a Wacom graphics tablet (e.g. Wacom CTL-672 ONE Medium , typical cost £50), we recommend using Xournal++  (Please download the .zip/.deb/.AppImage and not the source code; you may need to click on the assets link on an older entry on the page to reveal the relevant binary version for your system), or for marking students work and other PDF annotation or simply for taking notes. It works on Windows, Linux and Mac.
• Once installed please connect your Wacom tablet and then start Xournal++.

• You may find it very handy to assign the Wacom Pen buttons to functions under Edit -> Preferences-> Stylus. For example you could choose for "Button 1" to be the eraser and "Button 2" the highlighter or similar.
• You can now open the PDF you need (you can combine it with OneDrive "pigeon holes" above) and start annotating it, by first hovering with the stylus to get the pointer where you need it and then writing on the tablet as you normally would.

Marking work with Xodo and OneDrive (Android)

• Download Xodo PDF Reader & Editor
• If you want to use OneDrive to receive/send the files easily please also make sure the OneDrive app is installed and you are signed in to the university's OneDrive by entering your email address and then using the SSO account.
• Start it and give it the necessary permissions to access the storage.
• In the menu on the top left you can now access your OneDrive files by navigating to the relevant OneDrive folder.
• You can now mark the work with your stylus.
• When saving it will offer to overwrite the original file or to create a copy. Usually it is best to create a copy.
• You can now mark any work with your stylus, and it will be automatically saved to OneDrive for students either to collect directly or for you to copy it into their "virtual pigeon hole" later.

Marking work with PDF Expert and OneDrive (iPad - iOS)

• Download PDF Expert
• Start it. It will offer a trial/subscription for the PRO edition, which is not necessary for the required functionality, so just close that dialog by pressing the cross on the top left.
• Select OneDrive
• Enter your Maths email address
• Select "Work or school account"
• Sign in with your SSO account
• Now you should be able to select your OneDrive account on the left and browse to the relevant folder. For quick access in the future you can click on the 'three dots' next to each folder and add it to favourites. That would make it appear in the left sidebar directly.
• You can now mark any work with your Apple Pencil, and it will be automatically saved to OneDrive for students either to collect directly or for you to copy it into their "virtual pigeon hole" later.

eMarking Assistant

helping teachers assess, grade, & provide feedback on student assignments using Microsoft Word

Grading software or marking software for teachers.

This article reviews several grading software or marking software packages which can be used to mark assignments or grade papers. Most of the packages have advantages and disadvantages (highlighted in fuchsia).

This review focuses on grading software for teachers which an individual teacher can install on their computer without needing to buy an institutional site license or integrate the software into the Learning Management System or install software on a web server.

People may also want to refer to these guides:

• eMarking tools matrix from the University of Queensland
• eMarking software from Transforming Assessment

1. Word Add-ins, Toolbars or Macros

• Website: http://emarkingassistant.com/emarking-assistant/
• Type: Word addin that adds a floating toolbar to Word
• More information and demo video:  http://emarkingassistant.com/emarking-assistant/
• allows user to easily  create, rate, and insert comments containing text, images, tables, formatting, links
• comes with several comment banks e.g. APA style comment banks, academic writing comment bank
• can contain unlimited comments
• create, save and use eRubrics which automatically rescale, total marks and convert the mark to percentages, weighted marks and grades
• other tools allow you to highlight a phrase and do a good search or highlight the phrase throughout the assignment
• can be used to grade a paper based or presentation by writing numbers on the paper and inserting comments into a feedback document
• Windows Office 2000, 2003, 2007, 2010 or 2013 (32 bit or 64 bit)
• no Macintosh version of eMarking Assistant
• Rubric-O-Matic has only the rubric component and works on Windows or Macintosh
• $USD 40 per user per year reducing to $USD 20 per year for a 5 year license
• site licenses are also available
• apply for a free 6 month trial for your university, school or company

Rubric-O-Matic (was eRubric Assistant)

• Website: http://emarkingassistant.com/rubric-o-matic/
• More information and demo video:  http://emarkingassistant.com/rubric-o-matic/
• Windows Office 2000, 2003, 2007, 2010, 2013 (32 bit or 64 bit), 2016 (32 bit or 64 bit), or Office 3656
• Macintosh Office 2004, 2011 and 2016 or Office 365

T.A. Toolbar

• Website: http://tatoolbar.com/
• Type: Word addin that adds a ribbon to Word
• Demo: http://tatoolbar.com/about-2 no longer working
• Allows you to insert comments into a word document by clicking a button
• User can add more comments
• comments can only include text (no images, links, audio, tables, formatting)
• Microsoft Word 2003, 2007, and 201
• Microsoft Window XP, Vista, and Windows 7
• You must have administration rights on your computer to install it
• Cost: $USD 149.95

Gradeassist

• Website: http://www.pappasvolk.com/gradeassist/
• allows you to insert comments into a Word document
• Comes with preloaded with either an APA, or MLA or Turabian comment bank ( each additional comment bank costs $USD 99 )
• Cost: $USD 99 for a 1 year license
• Vista, Windows 7, Windows 8 ( no Windows XP )
• Office 2007, 2010, 2013 32-bit and 64 bit (no Office 2003)
• Website: http://www.11trees.com/home.html
• allows you to insert comments into Word document by clicking a button
• Windows Microsoft Word 2007, 2010, 2013 ( no 64 bit version )
• Microsoft Word 2007, 2010, 2013 ( no Word 2003 version )
• Macintosh Word 2011

2. Software to create a feedback file or document

• Type: prepare feedback document
• Website:  http://www.cict.co.uk/markin/index.php
• Cost: approx $US40 for a single license with discount for multiuser license
• Import the assignment and then insert comments into the text of the assignment by selecting existing comments or inserting new comments
• Insert feedback at the end of the assignment
• Availability: Windows 2000or later

Electronic Feedback

• Website:  http://www.ljmu.ac.uk/ITHelp/software/feedback.asp
• Cost: free by applying on the above web site
• Allows you to construct a feedback file by picking comments from a comment bank in Excel
• Availability: Windows MS Office only

CAFAS: Computer Aided Feedback and Assessment System from UniSA

• Website:  http://cafas.pbworks.com
• Demonstrations:  http://cafas.pbworks.com/w/page/4430555/Training%20Videos
• It is unclear if this system is still available

Essay Grader from Gatsby’s Light

• https://itunes.apple.com/us/app/essay-grader/id376946246?mt=8
• http://www.gatsbyslight.com/essay-grader.php
• https://www.youtube.com/watch?v=pevpoZVP2kk
• displays a PDF document and you pick prewritten comments from a tab and checkbox interface
• iPad, iPhone, or IPod touch

3. Software to complete rubrics

• Website:  http://www.mark-rite.com.au/
• Features: allows the grader to complete a rubric when observing an Objective Skills Competency Exam (OCSE) which are commonly used in Medical Education.
• Requires: any mobile enabled mobile device
• Website:  http://www.isocrates.org/
• create and use rubrics
• comments from a bank of up to 25 comments can be added to the rubric
• Windows (no Win 7 or 8 version)

EssayTagger

EssayTagger is a web-based application which individual teachers can license and use. to insert comments and rubrics.

• Website: http://essaytagger.com
• Student or teacher can upload the assignment
• Teacher can insert rubrics into the assignment and add a comment to the rubric
• can add other comments
• comments and marks are stored on their server to they can give you an analytic report
• Student can retrieve the file from the system or you can send it to them
• Assignment cannot contain images or formatting and only text
• need to be online to use it
• comments can contain only text (no links, images, tables, formatting
• Software as a service
• $20.00 per teacher per month

4. Grading systems that allow you to record audio or a screen recording

These systems either record audio or the screen as you are marking the assignment.

Jing or Camtasia or other screen recording software

• Allows you to record audio and a screen recording and then stored the resulting video file with a password so only the student can retrieve it when you send them the link and the password

Sounds good or Audacity or other audio recording software

• Audacity  is available from  http://audacity.sourceforge.net/
• Sounds Good  is available from  https://sites.google.com/site/soundsgooduk/Home

5. PDF Annotation tools

This type of software is a digital version writing comments on the paper using a red pen or perhaps adding stamps or text comments. Not reviewed further as these systems are simply a digital red pen and they do not allow efficiencies such as reusable comments or additional features of the computer. Examples of this type of software include

• Adobe Acrobat: Windows or Macintosh

Some PDF annotation systems also allow other feature e.,g.

• RemarksPDF allows typing comments and reusable comments
• iAnnotate (iPad) allows voice comments

RemarksPDF is a PDF annotation tool that also allows you to create and store reusable comments

• Website: http://remarkspdf.com
• draw over a PDF document
• insert text comments into a PDF document
• text comments can contain only text (no links, images, tables, formatting)
• Windows XP, Vista, Windows 7 ( no Windows 8 version )
• Mac OS X 10.4 of later
• iPad ( no Android version )
• $AUD 25.00 per license for a one year license

PDF XChange Viewer (Windows only)

Despite being called a viewer this application includes very capable tools for annotating and editing PDF documents. This includes inking, drawing, typing stamps and other annotation tools. In addition you can split and merge PDFs.  And is is free for Private, Commercial, Government and all other uses as long as it the software is not distributed for profit.

• Website:  http://www.tracker-software.com/product/pdf-xchange-viewer
• Windows XP, Vista, Windows 7, Windows 8
• no Mac OS X 10.4 of later
• no iPad (no Android version)

6. Tablet based tools

Iannotate pfd.

• Website:  http://www.branchfire.com/iannotate/
• pen and ink writing over the assignment
• a highlighter
• record audio
• you can create a comment bank
• you can create and store rubrics
• iPad (9.99)
• Website:  http://uqmarkup.ceit.uq.edu.au/blog/about-uqmarkup/
• unclear if this is available to other universities

7. Systems that require an institutional license or to be installed on a web server

Rubrics and Marking guide within Moodle

Rubrics and marking guide are now standard components of Moodle 2.3 or higher.  get from  http://flo.flinders.edu.au/mod/book/view.php?id=263709&chapterid=24175#rubric3

Grademark used within Turnitin

• Website: http://turnitin.com/en_us/features/grademark
• Allows user to create, save and insert reusable text comments
• comments can contain only text (no links, images, tables, formatting)
• create and use marking rubrics
• web-based system and you must be online to use it
• you do not have access to all the features of Word when using it
• Purchased from Turnitin as an institutional license
• Windows, Macintosh, iPad

inline Grading for Assignments within Blackboard Learn

• Website:  https://help.blackboard.com/en-us/Learn/9.1_SP_10_and_SP_11/Instructor/070_Assignments/003_Grading_Assignments/Inline_Grading_For_Assignments
• free with Blackboard Learn 9.1+ SP10+
• Displays Word, Excel, PowerPoint and PDF files (Bb maintains that most formatting is retained)
• Allows comments to be entered or picked from a reusable comment bank

Lightwork within Moodle

• Website:  http://lightworkmarking.org/
• Demo:  http://lightworkmarking.org/Video/LightworkPresentation.html
• Availability: requires system administrator to install software into the Moodle server
• create rubrics to reduce inconsistency
• takes care of the workflow to distribute assignments to markers, moderate them and return them to the student

Glenco Online Essay Grader from McGrawHill

• Website:  http://www.glencoe.com/sec/glencoewriting/
• allows to you grade assignments using a rubric
• requires institutional license

8. Grading software to automatically mark or grade assignments

There is much debate about this e.g.  http://www.nytimes.com/2013/04/05/science/new-test-for-computers-grading-essays-at-college-level.html

Not reviewed further as current evidence is that these systems are not as good as human markers except in certain constrained environments. Generally not relevant to individual teachers as they are generally licensed to institutions or states.

9. The following grading software and marking software is no longer available

• WritePoint software was provided to University of Phoenix lecturers to help them grade papers. This WritePoint software presentation provides a good overview of its use. eMarking Assistant provides similar functionality.
• Mark my Words from http://mws.ust.hk/mmw/index.php
• Semi Automatic Grader
• Markers Assistant 3.2 from http://www.surreal.com.au/index.php?option=com_content&view=article&id=46&Itemid=56
• Markin from http://www.cict.co.uk/markin/index.php
• reMarksPFD Word add-in which was available from  http://remarkspdf.com

The packages include eMarking Assistant, Grade Assist, T.A. Toolbar, GradeAssist, Annotate, Electronic Feedback, Essay Grader, iSocrates, EssayTagger, ReMarksPDF, GradeMark, and LightWork. Systems that are no longer available e.g. Writepoint, Mark My Words, and Markin are also listed. In future a grid will be added which will rate the grading software against the following features:

• Word add-in: is it a Word Add-in that lets you use existing Word tools (e.g. revision marks or spell checker) and is familiar to most teachers and students
• Reusable comment banks: can you create and use comments from a reusable comment banks. Can be any type or only text
• Inline comments: Can the comments be inserted inline or in the margin
• Feedback document comments: Can the comments be inserted at the end of assignment comments or in a new file
• Drawing: can you draw over the assignment or use stamps
• Complete a rubric: can you complete a rubric by selecting cells and calculating totals
• Document types: Which of the following document types can you mark Word, PDF, Excel, PowerPoint
• Hosted web service: is it a hosted web service that allows assignment submission and return
• Audio comments: can audio comments can be recorded and inserted
• Moodle integration: Is it standard or does it requires an installation and plugin
• Blackboard integration:
• Turnitin integration:
• Platform: what platform does it run on Windows, Macintosh, iPad, Android
• Cost: what is the cost

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Yes, add me to your mailing list.

Enter coupon code "50%off-2691" to get 50% off eMarking Assistant (6 yrs for $60) or Automated Analytic Rubrics (6 yrs for $30) ! Dismiss

This browser is no longer supported.

Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.

Marking Assignments on teams

Hi, I'm a teacher and I have recently trialed the function assignments with a class. My aim is to be able to mark all their submitted work through teams, yet I wasn't able to write comments (on ipad with ipen) into pdf or jpeg documents. The only document that I was able to write on and add comments to was a word document. To me, that is a bit dodgy, since I was also able to edit the actual document and thus the student's work during the process of marking it. So my question is, can one only mark and return word documents through teams, without the added hassle of downloading the document first and mailing it seperately to the students, or are there options to allow online editing in teams of other file formats? Thanks a lot for your time. Vera

I also require the ability to easily markup PDF documents. Here's my work around.

• Set all PDF documents to open with Microsoft Edge
• For students to submit an assignment, they must go to their Homework section in the class notebook, create a page for the assignment, insert the PDF as a printed attachment
• I open the notebook in OneNote as it has better editing features
• Double click the PDF attachment and it will open in MS Edge
• Do markups and click save.
• The file is saved back to OneNote for students to open at any time

In my experience, there is no built-in editor in Microsoft Teams if you want to edit PDF in Microsoft Teams. The only format we can edit it in Teams are Word, Powerpoint, Excel and Visio files. If your format is pdf, we recommend you convert it to words and then you can add comments and edit it in Microsoft Teams.

If the workaround is not helpful to you, we recommend you give your feedback in Teams UserVoice. ( https://microsoftteams.uservoice.com/ )Microsoft will always focus on customer’s feedback and experience, some new features would be added to the services based on customers' feedback in the future, and your good ideas will be very helpful for them to improve the service.

2 additional answers

Hi, I provided one solution to teachers of my school to mark assignment on Teams or for Microsoft Teams assignment or online paper corrections ( pdf format) using Acrobat Reader feature. Please check this video created by me regarding the same.

https://youtu.be/nXWuxOrk3Qs

Regards, Suparna

This is an unbelievable feature ommision. I've used Google Classroom extensively for the past several years in which students can submit Docs and I can simply tap the markup button and I get a PDF copy to markup right in Classroom.

I've recently moved to a college that disallows Classroom in favour of Teams, now I find out that you cannot markup student work on Teams? This is WEAK. I might as well not bother setting assignments on Teams anymore.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

E-marking Tools Guide

Related Papers

Jibril Muhammad Adam , Umar Shafiu

Lecturers spend an awful lot of time and effort to manually mark assignments. When marking assignments, lecturers go through a repetitive process of opening files, working through checklist, calculating grades, recording them, etc. It would be easier for lecturers to follow good-practice in assessment if some of this leg-work was done for them. This onerous task needs to be addressed so lecturers can quickly and easily mark assignments and provide more useful and qualitative feedback. With the advancement of technology, there have been attempts to overcome this burden through eMarking tools. With this in mind, this paper describes the design and implementation of an eMarking tool that will aid at automating the repetitive processes involved in marking so that the heavy load caused by these burdensome activities will be lifted off the shoulders of lecturers.

Syllabus fall2002, …

ACM SIGCSE Bulletin

Michael Luck

Australasian Journal of Educational Technology

Bruce Granshaw

15 Abstract— In the near future, a pervasive digitization environment can be expected based on the recent progresses and advances in computing and programming technologies. Next generation of evaluation system is transformed from manual evaluation process to digitization evaluation process. The digitization evaluation process is called E-marking system. This E-marking system is designed for digitization of the evaluation process so that we can reduce the errors in the evaluations process and can release the results in more easy way. This paper describes how the digitization is done to evaluation process by giving its related research background including the concept, features, status, and applications of E-marking system. Some of the technical challenges that have been faced during the development process of E-marking system are also presented.

Proceedings of the Canadian Engineering Education Association

Peter Ostafichuk

The use of online team marking has the potential to both simplify and expedite the process of marking exams, papers, and other artifacts. An online team marking tool (Crowdmark) has been piloted at UBC in Mechanical Engineering (125 student midterm) and two common first year introduction to engineering courses (840 student final exam, and 730 student midterm and final exam).Crowdmark, the particular software tool used, printed a unique QR code on each page of each exam and then exams were written by students in a conventional pencil-andpaper fashion. After the exam, papers were digitized and uploaded to the Crowdmark system. Following a brief training and orientation session, all marking took place by teaching assistants through the Crowdmark interface. Overall grader preference was positive, with the majority of graders expressing a strong preference for the Crowdmark system over conventional paper-based grading. In MECH 223, extensive historical data for marking time was availabl...

Assessment & Evaluation in Higher Education

Alistair Campbell

Paul Blayney

This paper describes the development of a spreadsheet-based marking system and its use in first and second year accounting courses at the University of Sydney in 1994 and 1995. A live demonstration of the system is conducted as an integral part of the paper. The paper begins with a relatively brief depiction of the two-fold motivation for the system’s development: (1) the desire to provide large numbers of students with timely and accurate feedback on assignments in a resource constrained environment and (2) to foster use of the personal computer as a problem solving tool through development of expertise with a computer spreadsheet. The first motivation is in essence an integral part of one institution’s strategy to deal with both the current economic environment and the needs of its customers (students and employers). The second motivation is a direct response to the changing needs of today’s students in that the degree of data analysis required for much of their work cannot, in pr...

Yorgos Eliopoulos

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED PAPERS

Dr.Jenny Lane , Alistair Campbell

Denise Whitelock

Mathew Hillier

Lincoln Wood

Encyclopedia of Artificial Intelligence

Dimitris Rigas

Yvonne Breyer

Eva Heinrich

Research in Learning Technology

Jorge J Villalon , Manuel Matas

Johannes Magenheim

Birgit Loch

To be presented at the IEEE International Conference on Advanced Learning Technologies, ICALT 2013, Beijing, China

Jorge J Villalon

Polly Kobeleva

Educational Studies

Geoffrey Lautenbach

Zaara Barhoumi

International Journal of Instruction

Alexander B. Swan , Ani Ma

Prof Stephen E Colbran

Journal of Engineering Research and Reports

Stuart Shaw

pejman sajjadi

Ralf Klamma

EURASIA Journal of Mathematics, Science and Technology Education

Tzu-Hua Wang

Computers & Education

Fernando Fonte

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

pdf.gradex®

Mission accomplished …..

• Post author By timothy
• Post date 24 June 2020
• No Comments on Mission accomplished ….
• Sticky post

We’ve finished processing the exams in my organisational unit for this exam diet. It all went well. So we’re done. Finally! Here are the stats on what the system achieved (with the help of 48 combined cores of processing and two additional staff doing processing for around the last five weeks, working together to support our the exams team). This system was also used by another organisational unit as well, as their default option.

I’ve learned plenty about our behind-the-scenes exam processes and the limitations of PDF – this was definitely the right solution for us, for the emergency situation we were in, despite the immense amount of work to create it. It de-risked the process by keeping everyone’s work safe from edit conflicts between students and staff, and between staff members assessing it, as well as being backwards-compatible with the old Mk1 Human so we could revert to the pain of a fully manual process at any time if we hit insurmountable problems (which we didn’t in the end).

Working with the PDF ecosystem reminded me why I prefer browser-based solutions – you can interact with the user and keep them on track. Me

An online electronic marking system is the way to go for us in the future. That’s possible for us now that we’ve taken that first step. Using a digital process that mapped exactly onto our existing workflow avoided a possible barrier to adoption of a new approach. Now our staff have seen the benefit of a digital workflow, we can have a chat about adopting a fully online system – especially since colleagues ran limited trials with other systems (interestingly, using this system to pre-process files before feeding them into the competition). The issues I ran into with the PDF ecosystem itself mean I don’t think a better user experience overall is possible no matter how good the underlying library that you use to work with PDF – you are still at the mercy of the interpretation of the specification for PDF by the viewing tools used by the staffs. We topped out on the quality of that experience, so time to move back to my favourite domain – the browser.

So, as you might guess, eyeing up the browser takes us smack bang into the territory already served by some existing tools. To compete or not to compete? This was not an easy decision because there is a really strong ethical benefit to operating assessment software from an open-source academic team. It provides an opportunity to fund the development from teaching income, and avoid the allure of monetising the student data. By the way, the reason that is even an issue for the edtech sector is that if you have shareholders, you have a legal obligation to do right by them, and that makes it rather difficult to leave money on the table (money you might need to stay in business, to grow, or to make a suitable exit for the founders). We can get around that by using a different funding model.

On the other hand, is it worth the effort to develop a tool which doesn’t monetise student data, but still (only) achieves the same (limited) academic outcomes? And how do you feel about that when you step back and realise you’d really like to see a bunch of change in the way things are done in that educational sector? Is that a reason to aim for a seat at the table now, just so you can try and make change later? Or would that be so much effort you’d never get to the transformative stuff? And is the context in which you’d be operating sufficiently flexible that you can slot the transformative stuff into whatever you are doing, or might it not fit?

There’s a really excellent discussion of the difference between migration online, and transformation online, in assessment systems here – an article suggested to me by Jen Ross, during a conversation we were having with Tim Fawns, who also added great stuff to my thinking (which I am not re-iterating here simply because it needs a proper writing out!)

There is no bald statement of the right path in that paper, but there is certainly a strong indication of the risk of getting bogged down in migration instead of doing transformation. What sort of development journey will evaluative judgement, contract grading, accept-revise, student-in-control grading, and various other practices need in order to flourish – and what role will digital assessment play – if any? Can you make a logical sequence of technical development moves to get from online “commodity” grading to the transformative stuff? Or are they paths that do not cross? In which case, what happens if you try to start with migration? What if you think there might be a way to do all of this without any traditional grading at all? (Health warning: people of a traditional disposition, look away now. Too late? Did I already say that? Sorry not sorry). Ok that’s clearly an extreme provocation. But it is a thought process I went through to offset the dangers of sunk-cost fallacy and train-track thinking (i.e. trying to avoid the pitfall of “I’m doing this, so I shall do more of this”).

How to resolve this in the case of future tools in the gradex ecosystem? Since academics who care about teaching, care about improving the experience for students, it seems a greater good can be served by focusing immediately on new practices that lead to better experiences for students and putting up with short term ethical concerns around any usage of existing commercial tools that fulfill a need. So gradeX development work is going to go on hold, while I bury myself in my keyboard again to deliver an open-source remote laboratory infrastructure over the coming months. I’ve a few other things up my sleeve as well … but more about those in good time!

• Tags academic , assessment , blog , edtech , education , gradex , monetisation , open-source , PDF , reflection , summary

pdf.gradex™

• Post date 10 April 2020
• 2 Comments on pdf.gradex™

oh… this will be just like marking on paper, maybe even quicker an academic colleague

pdf.gradex ™ started life in April 2020 as a rapid response to the challenge of getting 50 colleagues to remotely mark 65,000 pages of scanned exams. When wrestling with the best way to get the job done in double-quick time, a bunch of exciting possibilities surfaced – why stop at a one-off patch when we can really benefit in the long term, as an academic community, if we build the tools that suit our local context and help others to do the same?

Edit: just over two months on (June 11), this project is nearing completion of the first phase of development – just a few features to go and a few exams to finish off processing. Here’s an animation that is taken from the git repositories associated with the project, giving an idea of how I managed risk by developing critical features in isolation, then bringing them together in the integrated tool:

What’s in a name?

The X in gradex ™stands for

• X as in “multiply your marking power” (automate tedious stuff);
• X as in “you can mark anything” (on paper now, others later)
• X as in connectivity networks between different marking tools;
• X as in “no to surveillance” (you control all your data);
• X for the variable page size in the first tool pdf.gradex ™.

This pdf workflow is just the first “sub-domain” in a broader vision.

But – solved problem, right?

Yep, commercial marking systems are available, but there are good reasons not to settle for them. One of them is that sharing student work with third parties is a sensitive topic. This project aims to make it possible for institutions to keep that data in-house, where it belongs, yet still make significant efficiency, quality and equality improvements to their process. That’s keeping the lights on. What about getting the party on? By putting our code where our mouths are, we can explore the tensions between the latest world-leading research in education and the practical implications of those philosophies and values, without having to somehow keep a boardroom happy too. And how do we invite everyone to party in a way that suits them, or a no-party if that’s your preference? Digital tools always bake-in cultural norms, whether you are aware of it or not. Open-source (with a diverse community ) is a necessary condition to overcome that. Of course, “necessary is not sufficient”, but we can’t start without open-source software that underpins the local and wider community. We have a long journey ahead to ensure equality, diversity, and inclusivity for all in education, and education is driven by assessment, so let’s keep that lever out in the open where we can see it and evolve it.

Project status:

• Proof of Concept and initial sense-check with colleages – done
• Phase one feature-set implementation (marking)- done
• Initial focus sessions with alpha testers – done
• Initial trials with beta testers – done
• Intended first use done (5000 pages processed so far in one day)
• Round two feature set implementation (reporting) done
• Round three feature set done (front cover pages, entering, moderation, cross-referencing of all pages)
• Many more pages processed by a team of two … need to count them!
• Now the default option in another School too

Risk management

This system is decoupled completely from student submissions, so we can get on with receiving scanned electronic papers then bring this system in behind that to ease the workflow for markers and administrators. There is no lock-in, because it is just pdf – you can go back to the default human-intensive solution at any time (not that you will want to!)

Potentially interested?

Drop timothy a line (admin AT gradex DOT io) if you want early sight of in-development materials and some chat. Code freely shared already – documentation stills need backfilling, but the initial pre-release version of the command-line version of the tool is on github . If you want to see details of the how things work under the hood, see the parsesvg library for how pages are designed in Inkscape.

®ight, what next?

• Post date 13 November 2020
• No Comments on ®ight, what next?

This is just a “quick” update as we approach the next exam diet (Semester One, Academic Year 2020/2021 Northern Hemisphere). Short story, we’re jumping ship to the original plan. Read on for why, but first a reminder to those involved that we did a good thing, for good reasons, and time and context have changed, not the reasons for doing what we did, when we did it. (In the same situation, I’d do the same thing again).

There were a lot of really good things about doing this project the way it was done, that were specific to the situation we were in (pandemic, no access to a particular commercial tool, and the short time frame requiring a certain kind of risk management – i.e. don’t hide data in a non-human format or store it in a fragile place, so we don’t put student or staff work at risk if the tool breaks unexpectedly). But it did take a dedicated team of three of us to process the exams – often I was coding and testing features until 3:00, 4:00am or later, then they were going straight into production with one of our team using them from 7:00am or so. That was fun, but hard work.

Looking back, I still can’t quite believe the quantity of features, or level of detail that things got to, with this project … perhaps with the lack of sleep I wasn’t forming memories too well … but if a thing needs fixing, it needs fixing properly, so you can sleep at night (even if that is just for a an hour or two before you are bug-fixing your latest release), and that means handling detail.

One of the least rewarding aspects of this project was that the risk management strategy forced us to upload and download a lot of redundant data (all the images in the PDF exam copies had to go with the document everywhere, even if the new data added by a marker was just a few hundreds of text characters), and to add to our pain, we had decided to use a typical corporate data store that was at times, how shall we say, vexing and tiresome. There were also the inevitable human behaviour challenges – ideally, users shouldn’t need to read instructions (it’s a nice ideal, at least), nor should they be able to mess things up when they are trying to be a good actor. But relying on an eco-system like PDF, across a lot of devices, with a lot of patchy implementations of the standard was probably the thing that the biggest issue for me. With PDF there is no way you can script in the level of interactive guidance that you can with javascript on the web, and you can’t control what software a user wants to use. And that can be a major issue ….

For example, when a user misses instructions not to do so (not really their fault, in my view), and uses the inbuilt software from a major premium brand of device supplier, which does not implement the spec correctly, and breaks your PDF parsing ability to the point it cannot actually understand the file (messing with the PDF catalogue is problematic for obvious reasons), then you have to get someone else to re-key the marking. At this point, you know you are at the limits of your ability to improve the user experience without moving away from PDF. When you are at the limit of the ecosystem like that, it’s not like more coding from you will help. Nor will tech testing, really, because how do you communicate all that to the users? It’s not like you can check the user agent and throw up some guidance on a better browser with the features you need, as you can do with the web.

So for this project, the issues of the ecosystem reliability, lack of control of user experience, and data overhead, mean it is now time to find a way to put on a web frontend and an online backend. That’s definitely possible but … the question is – would you really want to develop a whole new tool to do that when you can just buy one? Which is where we started – we had something we wanted to buy but couldn’t get it.

Well, more than half a year later, we’ve finally managed to get our mitts on the commercial tool we originally wanted, and others are now taking on the process of integrating it into our procedures. I’ve stepped back from that process because I want to get back to my first love, remote labs. Which I owe a whole shed load of long-overdue development time. Thanks to a great team of people at my current place, I now have a lab again for the first time in about two years to underpin that development work (There is more information on this project coming soon, to a different blog site which I’ll link here when it is up).

What does that mean for gradeX®? Well, astute Readers will have noticed the name is now registered. If you buy me a coffee post pandemic I’ll explain one of the other motivations for doing this, but mainly, it was a symbolic move to show the people that worked with me on this that what we did has laid the groundwork for some future goodness, which will remain true to the open-source, development-in-public approach of the project so far. For now, I think that future goodness needs to be something that adds to the tableau of marking codes out there, rather than duplicates (e.g. trying to compete with existing tools to do the same job). So, novel marking practices that are not adequately supported by existing tools are all potential candidates, with a possible aim being to support the piloting of a bunch of things related to authentic assessment and scale them up quickly if they are promising. It might be something to support some twist on evaluative judgment development, peer interaction, reflection or some other idea. I’m expecting those ideas to emerge from ongoing curriculum development efforts, and we’ll know when it is time to work on the software to support them. Meanwhile, if you find the unique combination of features in pdf.gradex to be useful, then remember, it is open-source, and remains available to use in the form that got us through >39,000 pages, although I won’t be available to hold your hand (unless you can make me an offer I can’t refuse, and chances are, you could get a commercial product for that sort of money). See the repo here https://github.com/timdrysdale/gradex-cli . Enjoy! Until the next update ….

Order that tab

• Post date 4 May 2020
• No Comments on Order that tab

So I’ve learned an awful lot more about PDF than I ever expected to. One of the traps is that tab order isn’t always respected by the viewer, but it is in Adobe Reader, so I’ve refined the design some more, to take advantage of that.

Feel free to drop me a line for a copy of the doc itself!

[email protected]

So how do I mark?

• Post date 20 April 2020
• No Comments on So how do I mark?

So you’ve got some exams to mark and wondering what the process is like? Here’s a quick video to give you an idea

https://www.youtube.com/watch?v=jIs0a4yU2tU

Comments – yes please!

• No Comments on Comments – yes please!

This post is actually about two different types of comments …

• feedback from a colleague
• how we can use/handle pop-up comments

The post title reflects me realising that I wanted to support pop-up comments in the pdf-handling bit of the gradex™ tool…

1. Feedback on the marking process

Overall, I think it’s cool. As long as we can get students to upload stuff that is readable I think it should be possible to mark at pretty much the same speed as normal. An Anonymous Colleague

Awesome, I am pleased with that. Avoiding time wasted with the tool is one of the main goals – use effort for marking, not marking up.

And here is one that should be suitable to follow through as a worked example for the web page. Page 1 – normal marked page, one comment and addition should be correct Page 2 – marked page but should have an adding up error (to show how this will be moderated) Page 3 – I’ve checked the box at the top to indicate that I can’t read it (assuming that is what the box is for) .. The Same Colleague

Here’s a look at their marked-up file (images-only )

This is a great example of how marking will get done – careful consideration, some half marks, a simulated adding mistake (it happens in real life from time to time, so we have processes to catch it) and a duff page getting flagged (that page wasn’t duff, but we’re using our imagination).

2. Handle those pesky virtual post-it comments

Seeing the use of pop-up comments in this marking demo, made me realise I wanted to handle them in the gradex tool. They reduce your cognitive load (you can park some parts of your thoughts on paper to focus on other bits). It’s unlikely that students will use comments on their scanned exam scripts, but they might. Markers definitely will. For marking in PDF, comments do produce some difficulties though:

• PDF readers don’t respect read-only settings
• Ghostscript doesn’t flatten them to the page image like the other annotations (see images above!)

That means I need to handle them myself, and flatten them in some way, so that they cannot be edited in a rogue non-compliant editor.

It just so happens that both comments in this marked document popped open to cover over parts of the marking area where there could be important information. So I can’t just flatten them where the document says to put them, because they would permanently obscure important information. I thought through a bunch of options

• stashing the comment text within the document, then duplicating the comment in a new pop-up
• putting comments in special part of the sidebar*
• putting comments in a special header or footer
• finding out which editors respect the readOnly flag and putting the rest on a “do not use” list.

These approaches all have wrinkles or gremlins. The least worst option is to stash comments in a reserved area so that nothing is overwritten. So I have made a space to investigate putting comments in the sidebar, once I have prefill text fields implemented in a general way. For now – I’m writing them to the bottom of the page. I’m also putting in a yellow numbered marker where the comment was located, because position is important context.

These two pages (the two with comments) don’t look that different with the comments on. There is a chance of conflicting with student work at some point …. hence the plan to shift it into the sidebar if possible. Meanwhile, even if it is not as pretty as the grids, at least we’ve captured and preserved the comments. Phew.

Postscript (sorry not sorry)

The more you dig into PDF, the more you start to have thoughts about its clunkiness compared to web markup – these are well articulated by others ( like this) . The TLDR is that we’d do things differently now we are more worried about security more than we are about performance – and avoid structures that permit infinite cycles.

You know it’s a serious when the CEO of a software house is dropping comments like this in their code (presumably from a code review):

But it’s not bad for a library that’s older than the average age of the students sitting the exams. And the structure diagrams are pretty – if you aren’t thinking about parsing it!

Something to try …

• Post date 16 April 2020
• No Comments on Something to try …

As an Academic doing marking with this system, you don’t need any software other than a PDF reader ( that can speak acroforms). Then you will just a file formatted something like this (preview below), which you then edit, save and return.

The design is flexible so it can be changed to suit feedback or different procedures in different unis.

I learned that a lot of PDF viewers have patchy implementations, so you get a nicer experience if you go with

• Adobe Reader
• Drawboard PDF (for pen display)
• (your suggestions here)

Things that DON’T work, or have quirks

• Microsoft Teams (you have to download to your desktop and use something else)
• Chrome (can edit but not save, known chrome bug)
• Edge (can edit, but does uses small letters so not as satisfying)

Here’s an original submission, with no pdf.gradex ™ treatment. You can delete the annotations. Oops!

If you fancy seeing what happens to your work next, send me your edited form (and let me know if I can post here ….)

The value of paper

• No Comments on The value of paper

Work were kind enough to give me a blog spot t o talk about this project:

That also provided the impetus to put up a quick demo release. I do my dev on linux, but it seems pretty standard across the people I’ve sent demos to (Canada, Scotland) that teaching offices are using Windows 10. You can get the demo code from github. If you want to experiment with your own layouts, then there is information on the editing process in the demo README .

Even though I know it is a good thing in projects like this, the curse of YAGNI is always in your mind when you are sinking time into automation processes with their usual little wrinkles that (hopefully) no-one else need ever be burdened with. It started to pay off today, already. I had some user feedback on the grid design, and within 10min of the call ending, I had the design revised and working. I’d split the question boxes to nudge freehand markers to spread their characters out so I could do OCR bounding box estimation better. Of course, I don’t use half marks, and had assumed no one else did either. Oops. I like it better this way, and we can handle the OCR by either (a) begging and pleading with markers to put a space between their handwritten characters, (b) not doing it, (c) giving in an using an external image service (some issues there). I estimate it would have been about an hour’s work to do the geometry change by writing coordinates into golang structs, and it would not have been done with such good alignment either (or the with the bordeline relief-smugness hybrid of knowing it wasn’t YAGNI). So thanks self-from-last-week.

An example process

• Post date 14 April 2020
• No Comments on An example process

Here’s an example of a page that has been marked by keyboard, and the moderator has found an adding error, which they’ve corrected. The checker re-keys the numbers as part of the checking procedure.

Here’s a couple of the steps shown on their own, showing the moderator has only to check the page if all is ok, but can handle minor issues and communicate them back to the office. If there needed to be additional stages, we’ve got some other coloured “query” and “resolve” sidebars waiting in the wings.

Performance

Since we’re going for robustness and safety, we’re rendering all the annotations at each step, exporting to an image file, reimporting that into a new page that has the marking forms added, then exporting that again (phew!) – the stuff in blue all happens in a single command.

The process in averages around 1.4sec/page/stage at our current quality setting of 90 JPEG quality at 175dpi.

The round trip to collect scripts from our exam halls easily comes in at over 45 minutes, or enough time to process about 2,000 pages.

Marking Flows & Ladders

• No Comments on Marking Flows & Ladders

Like all good things, you sometimes struggle to get past the names you liked at various points in the process, and so they stick even when the design aesthetic has moved on.

You can tell where the name “ladder” came from when you look at the earliest version shown to colleagues for their feedback (red for marker, green for moderator):

Then I started branching out a bit with the design “flair,” but still handling the box geometry calculations by hand. It was clear that novelty would be limited by how long it took to wield the ruler on screen to line things up and encode the coordinates, but things did start to look a little smarter:

Then I realised that these could be made a bit more self documenting in their design, if I could just put boxes where ever I wanted. It would also mean I could take feedback from colleagues without wincing about how long it would take to recalculate where the boxes go. It was like going back to my first interactive educational computer program (on Venn diagrams, written for a primary school science fair on an IBM XT). Some ~30 years later we can do better than that!

I set myself the challenge of being able to parse the SVG output from Inkscape and autogenerate the acroforms for this sample page with three different size boxes that don’t line up with each other. There’s a video of that in action here , and a picture below:

Then I got to work on some more expressive side bars, with an option to include moderation or not. Good user interfaces (even if asynchronous like this) should be self-documenting, ideally. So the vertical lines show that you might indicate the scan was not good, and not do any further marking. Or you might fill in some sub-totals so you can keep track, then tot them up, put them in the Q box, and then tick the ‘page marked’ box. That’s something we can track automagically to make sure every page gets seen.

After marking, our office might select this paper for moderation:

… or you might not .So we can make that more obvious by putting in a visually different sidebar, so we’re not wondering if the moderator left some green boxes blank by accident or not:

Either way it will be checked afterwards in the office. At this point, it is more likely the entries will be keyed in. Then we can extract the marks and make reports. This sort of approach is likely to be faster than the paper process, and can give you more confidence (e.g. run reports that make sure every page got checked).

The paper wasn’t moderated, we don’t want the blue bar hanging off in space, so we move it over to the left – that dynamic X-spacing is another reason for the X in pdf.gradex ™.

You can imagine further stages for resolving issues, and re-checking. What we do after that are topic(s) for a further post.

Example marking page

• Post date 13 April 2020
• No Comments on Example marking page

This is a proof-of-concept example hot off the press – the first step in marking is to append a sidebar with marking ladder for subtotals, and a marking flow diagram for the total marks for each question that will be formally captured. Or, if the image is unusable, you can flag it for further attention by your support staff, and they will be alerted when they process your marked scripts. If the image is good, you can TAB from box to box, entering marks, leaving your brainpower to focus on the marking task itself rather than how to get the marks onto the page. If you have a tablet or pen display, you can freehand annotate, and these will be safely flattened into images and preserved by the system without risk of erasing them at future steps.

You can download the actual example here . You can edit and save using Adobe Reader, reload and keep going later. Other editors have variable support. For example, you edit and save in Edge, but not Chrome (it lets you edit but not save ). If you’re using chrome, like me, feel free to try out the editing side in the browser, just download the form and use a better editor before doing any work you want to keep!

To support additional steps in the process – e.g. for moderating and checking – additional sidebars can be added with different content, designs, colours, names – examples to follow soon!

With multiple people involved in the marking process, they may have different preferred ways of working. So we can mix and match by flattening any annotations that are made on the script.

The script “freezing” is using ghostscript to render the original image in the PDF so that it bakes in any annotations that the student has made (e.g. erasing mistakes or adding extra work). This way, the marker cannot accidentally undo the electronic editing that the student did to their own work.

It’s even conceivable to mix in printing, marking and then scanning, although it will require some additional steps in the workflow to match up returning scripts with the metadata in the electronic version (QR codes are one possibility here – topic for a future post).