Reviews

Depending on when your Submission Flow was created you may see a different version of the reviews feature.

Browse to your Submission Flow and take a look at the menu on the left. You will see either Review Stages or Review Designer. Choose the matching tab below:

What are reviews and review stages?

In Lineup Ninja running a review process facilitates you gathering feedback about your received submissions. You might want to gather feedback from committee members, subject matter experts, or other members of your team.

You can run a review process at any time during the submissions flow and you can run multiple rounds of review. For example you might want to run a initial wide review process, then a follow up round where your team reviews the results of the first round.

How do I start a review process?

The review process is managed in Review Stages. To start

  1. Navigate to your Submission Flow
  2. Click on Review Stages

A Submission Flow starts life without a Review Stage, to add one just click the blue plus button next to ‘Review Stages’.

Give your review stage a name, such as ‘Reviews’ and if you wish you can configure a deadline. This deadline will be used in email templates and shown to the reviewers in their list of reviews.

Once the review is created you have a chance to also set the name of the Stage that is shown to the reviewer. This will default to the name of the Review Stage, but you can change it to something different in ‘Name shown to reviewer’

How do I run a blind review? / How can I anonymise submissions? / How can I hide some questions’ answers from reviewers?

  1. Navigate to your Submission Flow
  2. Click on Review Stages
  3. Click on the tab for the Review Stage you want to update

In the ‘Answer Visibility’ section you can toggle whether the answers for each question on your Form are shown to the submitter. If you are conducting a blind review you can toggle off fields like ‘First Name’, ‘Last Name’ etc..

Can reviewers see the feedback given by other reviewers?

Yes. Once you have added questions in ‘Review Questions’ you can optionally choose to share these with other reviewers, by toggling them on and off in ‘Answers Given By Other Reviewers’.

Answers become visible to other reviewers when a reviewer saves their review.

How do I show the reviewer answers given in one Stage with another Stage?

When configuring multiple stages the reviewer questions added in other stages become available in ‘Answers Given By Other Reviewers’.

For example, if you are running a two Stage review process, in the second Stage you may wish to share the answers given in the first Stage. To do this browse to the second Stage and toggle the ‘Answers Given By Other Reviewers’ for the questions in the first Stage.

Where do I configure review questions?

Review questions are configured in the Review Designer.

  1. Navigate to your Submission Flow
  2. Click on Review Stages
  3. Click on the tab for the Review Stage you want to update
  4. Scroll down to the section called “Review Questions”

How do I configure the emails that are sent to reviewers?

  1. Navigate to your Submission Flow
  2. Click on Review Stages
  3. Click on the tab for the Review Stage you want to update
  4. Scroll down to Emails

You can configure emails to be sent when one of three things happen

  • When reviewers are assigned to a reviewer in this stage
  • When you request reviews to be repeated in a stage
  • When you cancel incomplete reviews in a stage.

By default an email is sent to the reviewer, if you wish you can also add additional emails to send. These can be sent to your team or to specific e-mail addresses.

How do I send reminders and nudge reviewers to finish their reviews?

  1. Navigate to your Submission Flow
  2. Click on Review Stages
  3. Click on the tab for the Review Stage you want to update
  4. Scroll down to Emails

In ‘Scheduled Emails’ you can add emails that will be sent at a scheduled time. For example you may wish to use this to send an email one week, and then again one day, before the deadline for your review process. For each scheduled email one email will be sent to each reviewer who has incomplete reviews.

How can I test my review stage?

To test your review stage you can

  1. Add yourself as a reviewer for the event
  2. Assign a review for yourself
  3. Complete the review
  4. Browse to the the list of submissions and check the submission looks as you expect (i.e. correct score, recommendation etc)
  5. Browse into the submission and check your test answers look as you expect
  6. Browse to Review Stages, click the review stage and scroll to the bottom and click ‘Reset All Reviews’

How do I set the review deadline?

The review deadline is set when you assign the reviews (each review could, theoretically, have a unique deadline, rather than there being one single review deadline for all reviews.)

When you assign the new reviews, the dialogue that pops up will ask you what deadline you want to use for them: Review Deadlines

How do I run a blind review? / How can I anonymise submissions? / How can I hide some questions’ answers from reviewers?

Review questions are configured in the Review Designer.

  1. Navigate to your event, and then Call for Papers
  2. Click on the name of the relevant Call for Papers
  3. In the left-hand menu, click on “Review Designer”
  4. Scroll down to the section called “Answer Visibility”.
  5. For each question that appears in your submitter form, you can use the toggle switch to show or hide the answers from your reviewers.

How do I set automatic reminders for reviewers?

When you assign submissions to reviewers, the following dialogue box pops up. If you set a Review due date (near the bottom), it will automatically send the reviewers an email 72 & 24 hours before the due date: Review Deadlines

How do I nudge reviewers to finish their reviews?

Nudging reviews is easy in the Review Stages version of the product. If you are using Review Designer then, the easiest way would be to unassign the reviews and reassign them, setting the deadline again. This will trigger an email that they’ve been assigned, and configure the reminder emails to be sent at 72 and 24 hours prior to the due date. And it retains any work the reviewer has already done.

The deadline for the abstract reviewers has now passed but I need to reassign a few submissions to get reviewed ASAP. How do I reset the deadline to tomorrow?

The review deadline is set when you assign the reviews (each review could, theoretically, have a unique deadline, rather than there being one single review deadline for all reviews).

When you assign the new reviews, the dialogue that pops up will ask you what deadline you want to use for them.

How do I track the review progress of a submission?

Go to the submissions list and search for the submission. There are several ways to find it:

  1. When submissions have reviewers assigned to them, they move into the “Reviews in Progress” tab. Once all the reviews have been completed, they move into “Reviews Complete”. In Both cases, you can see reviewers’ scores and recommendations.
  2. You can search for the submission by its title, submitter name, submitter email by using the search tool in these columns’ headings.
  3. You can use the advanced search tool to build a complex query to find the submission you’re interested in.
  4. You can filter and sort various columns in the submissions list.

What type of review questions can I ask my reviewers?

Lineup Ninja has 3 special types of review questions that are used in particular ways:

  1. Score questions: you can use as many score questions as you like, and weight them differently if you wish.
  2. Recommendation questions: by default the recommendations are “accept”, “waitlist” or “decline”, but you can edit these to mean something else.
  3. Feedback questions that are shown to submitters. These questions allow you to capture feedback from reviewers and pass it to the submitter. Technically, this is an option that can be set on any review questions. So if, for example, you wanted submitters to see reviewers’ scores, you set this option on the score question. The reviewers’ names are annonymised in the feedback.

Apart from these special question types, like the Form designer, you can add your own questions of various types:

  1. Short and long text questions
  2. Numerical questions
  3. Multiple choice
  4. File upload
  5. Email and web addresses etc.

Can I show reviewers some questions based on their answers to previous questions?

Yes, you can use ‘Display rules’:

  1. Edit the question that you want to show (or hide) based on a previous answer
  2. Scroll down to the Display Rules section
  3. CLick on the blue button that says “Display Rule +” on it.
  4. Choose the question whose answer you want to determine whether this question will be shown
  5. Configure the rule that will determine whether this question will be shown or not

How do I preview what reviewers will see when I assign a review to them?

The best approach would be to assign a test submission to yourself to review: then you get to preview the whole reviewer experience process - email invitation, login flow etc.

See also: How to assign to individual reviewers or a group of reviewers

How do I track a single reviewer’s activity or progress?

You can see how many reviews are assigned, in-progress and completed for each reviewer in the reviewer list in Submission Flows.