Create & Manage Reviews
What are reviews and review stages?
In Lineup Ninja, running a review process facilitates gathering feedback on submissions and can be done at any time during the Submission Flow. You may want to gather feedback from committee members, subject matter experts, or other members of your team.
Within the review process, you can customise the form to ask reviewers to score submissions, recommend which ones to accept/decline and give feedback to submitters.
You can run multiple rounds, or ‘stages’, of review (e.g. you may want to run an initial wide review, then follow up round where your team reviews the results of the first round).
How do I start a review process?
A Submission Flow starts life without a Review Stage. To start one, follow these steps:
- Navigate to >
- Click next to ‘Stages’
- Fill out the Review Stage Name and optional deadline in the pop up. The deadline will be used in email templates and shows to the reviewers in their list of reviews (this can be edited at any time)
Once the review is created, you can edit the ‘Name shown to reviewer’ if you want a different name shown to the reviewer then the overall Stage Name.
What is ‘Answer Visibility’?
Answer Visibility allows you to decide which answers should be shown to the reviewer. This can be used to run a blind review or anonymising submissions by hiding the ‘Speaker’ answers.
Where do I configure review questions?
- Navigate into the and click on the menu item
- Click on the tab for the Review Stage you would like to update
- Scroll down to the section called ‘Review Questions‘
What type of review questions can I ask my reviewers?
Lineup Ninja has five special types of review questions that are used in particular ways:
- Score questions - you can use as many score questions as you like, and weight them differently if you wish
- Recommendation questions - by default the recommendations are “accept”, “waitlist” or “decline”, but you can edit these
- Feedback questions that are shown to submitters - these questions allow you to capture feedback from reviewers and pass it to the submitter (the reviewers names are anonymised in the feedback)
- Upload image
- Upload file
On all questions, you can tick or untick ‘Use For Feedback - Submitters will see the answers given by the reviewers’ to decide whether you want to Submitters to be able to see the answers given by reviewers.
Apart from these special question types, like the Form Designer, you can create a custom questions. You should only create a custom question if you can’t use one of the above options.
It is important to note here that you must select the correct question that matches the type of data you wish to collect. If you can’t see a question for the data you need to capture, consider creating a Custom Field or Label first, then adding a question that maps to it. Otherwise, if you create a custom question that doesn’t map to anything, the answers can’t be copied to your agenda and onto subsequent integrations.
How do I make a question show (or not) depending on the answer to a previous question?
In the question that you would like to be displayed (or not), set up a ‘Display Rule’.
This allows you to choose from the list of valid previous questions, then choose some qualifying condition (e.g. ‘is at least one of’), and then supply the answers that must be matched.
How do I add labels to reviews?
You can assign Labels to submissions to help group them. These labels can be optionally shared with your reviewers. To add labels, navigate to the Review Stage and scroll to ‘Label Visibility’.
How do I configure the emails that are sent to reviewers?
- Navigate to the Review Stage you wish to configure
- Scroll down to Emails
You can configure emails to be sent when one of three things happen
- When Reviews Are Assigned
- When Reviews Are Requested To Be Repeated
- When Reviews Are Cancelled
By default, an email is sent to the reviewer, if you wish you can also add additional emails to send. These can be sent to your team or to specific e-mail addresses.
You can also schedule emails. This can be useful to nudge reviewers to complete their work.
How can I test my review stage?
The best approach would be to assign a test submission to yourself to review. This way you get to preview the whole reviewer experience process - email invitation, login flow etc.
- Add yourself as a reviewer for the event
- Assign a review for yourself
- Complete the review
- Browse to the the list of submissions and check the submission looks as you expect (i.e. correct score, recommendation etc)
- Browse into the submission and check your test answers look as you expect
- Browse to Review Stages, click the review stage and scroll to the bottom and click ‘Reset All Reviews’
How do I copy review stages from one event to another?
To copy review stages, you will need to copy the Submission Flow it exists under.
- Navigate to in the event you wish to copy data to
- Click ‘Copy Another Event’
- Choose the event you wish to copy items from and select ‘Submissions Flows’
This will copy all Submissions Flows. You can delete the ones you do not need.
How do I copy review stages from within the same event?
To copy review stages, you will need to copy the Submission Flow it exists under.
- Navigate to
- Click next to the Submission Flow you wish to copy
- Name your new Submission Flow and click ‘Copy’
Want more information about copying data within Lineup Ninja? Follow this link.
How do I delete a Review Stage?
To delete a Review Stage, scroll to the bottom of the stage and click ‘Delete Review Stage’. This will delete the stage and ALL assigned reviews.
Use with caution: Unpublishing your call for papers will delete a LOT of data that you cannot retrieve!
How do I show the reviewer answers given in one Review Stage in another Review Stage?
When configuring multiple stages the reviewer questions added in other stages become available in ‘Answers Given By Other Reviewers’. For example, if you are running a two Stage review process, in the second Stage you may wish to share the answers given in the first Stage. To do this browse to the second Stage and toggle the ‘Answers Given By Other Reviewers’ for the questions in the first Stage.
Next: Assign Reviewers
Still can’t find what you’re looking for? Email support@lineupninja.com and one of the team will get back to you as soon as possible.