AUDIO RATING PROJECT

Contributor Guidelines

Content and Prosody Rating (1–5)

A practical guide for evaluating agent responses accurately and consistently, with the original embedded prosody audio examples retained from the reference file.

THE ONE PRINCIPLE THAT MATTERS MOSTRate the audio you hear, not the transcript you read. Audio is the ground truth. The transcript is only a visual aid and may contain errors.

1. Project Overview

You are evaluating conversations between a user and an agent. The agent may be speaking as a peer or helping the user complete a task. For every segment labelled Agent response, provide two independent ratings and a justification for each rating.

2. What You Are Rating

Dimension	What it measures	Key question
Content	The words and meaning: helpfulness, relevance, accuracy, completeness, organization, and tone of the words.	If you were the user, how satisfied would you be with what was actually said?
Prosody	The delivery: rhythm, melody, intonation, pacing, expressiveness, pronunciation, timing, and naturalness.	If you were the user, how satisfied would you be with how the response sounded?

Core rating rules

Rate only turns labelled Agent response. User turns provide context and are not rated.
Rate as if you were the user, while applying the rating anchors.
Rate Content and Prosody independently.
Do not penalize transcript errors. Base the decision on the audio.
Do not spend more than five minutes fact-checking a Content response.

3. Task Process Flow

Follow this sequence for every task. Listening to the full conversation is optional when it helps establish context.

Review context

Note the scenario description. Listen to the full conversation if it helps you judge the turns.

Listen to user turn

Understand what the user wants and what would satisfy the request.

Listen to agent response

Use the audio as ground truth. Use the transcript only as a visual aid.

Rate Content

Choose 1–5 based only on what was said.

Justify Content

Briefly explain the score using specific evidence and criteria.

Rate Prosody

Choose 1–5 based only on how it sounded.

Justify Prosody

Briefly explain delivery quality using specific audio evidence.

Repeat

Complete Steps 2–7 for every remaining agent turn, then submit.

If you cannot confidently complete the item: use Skip when the topic or accent is unfamiliar, or when the audio is slow to load without an error. Skip is item-level only: it sends the entire item to the next annotator for review. Never use Skip for one turn or one rating dimension. Use discard only when a defined discard rule applies.

4. Rating Content

Content is judged only on what was said, not how it was delivered.

Rating	Meaning	Anchor
1	Very dissatisfied	Fails outright: completely irrelevant, wrong, dismissive, ignores the request or correction, violates a basic principle, is rude or NSFW, or refuses a reasonable request.
2	Somewhat dissatisfied	Largely fails: mostly unhelpful or off-topic, has a material omission or error, ignores a constraint or correction, contradicts itself, or simply agrees without helping.
3	Neutral	Partially satisfies the request but has a clear, non-disqualifying gap, such as a missed component, minor inaccuracy, or clumsy handling of context or correction.
4	Somewhat satisfied	Helpful and on target, with one minor shortcoming such as a small omission or slightly unsuitable length or level of detail.
5	Very satisfied	Fully addresses the request accurately and relevantly, at the right depth and tone. Uses earlier context, honors constraints, stays consistent, applies corrections, and leaves no meaningful complaint.

Content decision checklist

Helpful and relevant to the user’s request
Addresses every component of a multi-part request
Factually correct
Asks a follow-up only when necessary
Appropriate tone in the words
Appropriate length and level of detail
Clear, organized, and not repetitive
Right comprehension level and word choice

What to Consider for Content

Judge only what was said. Use the points below as a guide when choosing the overall Content rating.

Helpfulness and relevance

Does the response directly help with the user’s request and stay focused on what the user asked?

Completeness

Does it address every part of the request, including all components of a multi-part question?

Factual accuracy

Is the information correct? Do not spend more than five minutes fact-checking a response.

Use of context and corrections

Does the agent use earlier conversation details, follow stated constraints, and apply the user’s corrections?

Follow-up questions

Does the agent ask a follow-up only when more information is genuinely needed to answer well?

Tone of the words

Are the actual words respectful, appropriate, and suitable for the situation, regardless of how they were spoken?

Length and level of detail

Is the response detailed enough to be useful without being unnecessarily long, repetitive, or incomplete?

Clarity and organization

Is the response easy to follow, logically ordered, and expressed using suitable word choice?

Audience fit

Is the explanation pitched at an appropriate comprehension level for the user?

5. Rating Prosody

Prosody is judged only on how the response was delivered, independent of the words.

Rating	Meaning	Anchor
1	Very dissatisfied	Severe failures: jarring artifacts, voice identity changes, or delivery so unnatural that it is distracting.
2	Somewhat dissatisfied	Multiple compounding issues: robotic or flat throughout, unnatural pacing, misplaced pauses, or audible distortion that remains intelligible.
3	Neutral	Clear issues alongside acceptable moments: robotic in parts, unnatural pauses, or shifts between natural speech and a reading style.
4	Somewhat satisfied	Mostly natural with minor imperfections, such as slightly overdone energy, an occasional unnatural pause, or a brief performed or scripted quality.
5	Very satisfied	Fully natural, expressive, and appropriate to the context. It could convincingly be a real person speaking in a real conversation.

Prosody decision checklist

Realistic, expressive, and natural rather than robotic
Natural disfluencies, breaths, and pauses
Correct emphasis on important words
Appropriate pace
Tone and emotion match the context
Appropriate intonation
Correct pronunciation and clear enunciation
Stable and appropriate volume
Natural timing
Consistent speaker identity

What to Consider for Prosody

Judge only how the response sounded, independent of the words. Use the points below as a guide when choosing the overall Prosody rating.

Naturalness

Does the voice sound like a real person speaking naturally rather than reading, acting, or sounding robotic?

Pacing and rhythm

Is the speed appropriate, with natural phrasing and enough time for the listener to follow?

Pauses, breaths, and disfluencies

Are pauses, breaths, fillers, and hesitations placed naturally rather than interrupting the flow?

Emphasis

Are important words stressed appropriately without sounding forced or overperformed?

Tone and contextual fit

Does the vocal tone and level of energy suit the user’s situation and the conversational context?

Intonation

Does the pitch movement sound natural, including appropriate rising intonation for questions?

Pronunciation and enunciation

Are words pronounced correctly and clearly enough to understand?

Volume and audio stability

Is the volume stable and appropriate, without distracting distortion or artifacts?

Timing and turn-taking

Does the response, interruption, or backchannel occur at a natural point without talking over the user inappropriately?

Speaker consistency

Does the voice remain consistent within the turn and across the conversation rather than sounding like different speakers?

6. Writing Justifications

Be specific and name the exact issue.
Reference a specific quote or moment and the relevant criterion.
Cover both strengths and weaknesses when both are present.
Keep it brief.
For ratings 1–3, always explain what was lacking or wrong.
Do not reuse identical justifications, except “Appropriate backchannel” when it genuinely applies.
For Prosody, discuss delivery only, not helpfulness or meaning.

Good Justification Examples

Rating	Example justification	Why it works
Content 2	“Helpful advice but slightly off-topic. The user asked about sleep, but the agent mentioned exercise.”	Identifies both what worked and the specific relevance issue.
Content 2	“The agent suggested ‘try deep breathing’ again after already giving the same advice. It did not acknowledge that the user said it had not worked.”	Uses conversation context and explains the repetition and missed acknowledgment.
Content 5	“Directly addressed the question with a clear, actionable suggestion.”	Connects the rating to relevance, clarity, and usefulness.
Prosody 2	“There is an unnatural pause after ‘I think,’ and ‘definitely’ has a noticeable pitch jump.”	Names the exact delivery issues and where they occur.
Prosody 3	“The pacing is clear, but the voice lacks warmth when the user is upset.”	Balances a strength with a context-related delivery issue.
Prosody 5	“The emphasis on ‘way’ sounds natural. The closing question has appropriate rising intonation, and the brief pause during the interruption feels human.”	Uses multiple observable Prosody criteria and specific moments.

Longer Content justifications are appropriate when needed. For example: “The response suggested parmesan cheese in a vegetarian recipe, even though parmesan often contains animal rennet,” or “The agent included the drive time; although the user did not ask for it, the information was likely helpful.”

Examples to Avoid

Rating	Weak justification	What is wrong
Content 3	“It was okay.”	Too vague. It does not explain what worked or what was missing.
Content 1	“Bad.”	Provides no evidence and does not identify whether the response was irrelevant, incorrect, unhelpful, or inappropriate.
Prosody 2	“Didn’t sound good.”	Does not identify the delivery issue, such as pacing, pitch, pauses, volume, or monotone delivery.
Prosody 4	“Fine.”	Does not explain why the response is a 4 rather than a 3 or 5.
Prosody 3	“The response was too short and should have been more reassuring.”	This discusses Content, not Prosody. Prosody justifications must focus only on delivery.

7. Skipping and Discarding

Action	Use when	What to do
Skip (item-level only)	You cannot confidently judge the item because the topic or accent is unfamiliar, or the audio is slow to load without an error.	Skip the entire item so the next annotator can review it. Do not use Skip for an individual turn or a single rating dimension.
Turn-level discard	One turn or one dimension is unrateable.	Discard only that rating and explain why.
Item-level discard	The whole file is unusable due to pervasive segmentation problems, unavailable or broken audio, inappropriate or violent content, or significant extraneous material.	Discard the whole task.

Not grounds for discard

Transcript errors
A naturally split agent response
Backchannels
Slightly wider audio boundaries
A non-fatal audio defect that still permits evaluation

8. FAQs and Edge Cases

Use these quick rules for common situations. Open a question to view the guidance.

Why is each speaker heard in only one ear?

This is expected. Rate the task normally.

What should I do if the audio is unclear?

Replay the audio a few times. If it is still unclear, rate only what you can confidently hear.

If the audio is completely silent, broken, or impossible to evaluate, use the applicable discard rule.

What should I do about glitches, artifacts, or distortion?

Do not discard for a minor audio defect. Consider the defect in the Prosody rating and identify it in the justification.

Discard only when the audio is unavailable or too damaged to evaluate.

How should I rate a repeated response?

Rate each occurrence independently. A repeated response may be less helpful or less natural in its new context.

What if one response is split across multiple turns?

Rate each labelled Agent response separately.

Do not penalize a turn when the answer continues naturally in the next segment. This is a segmentation issue, not an agent error.

Rate it down only when the response is genuinely incomplete, incoherent, or unnatural.

What if I disagree with the agent’s advice?

Use the rating criteria, not personal preference. Rate the response down when the advice is genuinely incorrect, irrelevant, or unhelpful.

What if Prosody is good in some parts and poor in others?

Give one overall Prosody rating for the full turn.

Mostly natural with one minor issue usually leans toward 4. Mostly poor with one good moment usually leans toward 2.

How should I rate “mm-hmm” or another backchannel?

Treat it as a valid Agent response.

For Content, decide whether the acknowledgment fits the context. It may rate highly when the user is still explaining, but poorly when the user asked a direct question and needed an answer.

For Prosody, rate naturalness, engagement, and timing.

When appropriate, “Appropriate backchannel” may be used as the Content justification.

What if I hear part of the other speaker’s turn?

Slightly wide audio boundaries are normal. If the labelled Agent response is audible and evaluable, rate only that response and do not discard.

Discard only when the labelled response is missing or the audio is too misaligned to identify and evaluate it.

What if the User and Agent labels appear to be switched?

Always rate the segment labelled Agent response. Do not relabel the speakers or rate a different segment.

9. Glossary

Use this glossary for the key rating, speech-quality, conversation, and task-action terms used throughout the guide.

Rating Terms

Term	Definition
Agent response	The labelled segment that receives Content and Prosody ratings.
User turn	The user’s speech. It provides context but is not rated.
Content	What the agent said: the meaning, helpfulness, relevance, accuracy, completeness, organization, and tone of the words.
Prosody	How the agent said it. Prosody includes rhythm, stress, pitch, loudness, pace, pausing, and speech melody. Good prosody sounds natural, expressive, and appropriate to the context.
Rating anchor	The description that defines what each score from 1 to 5 represents. Use the anchor to select the best overall rating.
Justification	A brief, specific explanation of why a rating was selected, supported by an exact issue, quote, or audible moment.

Speech Quality Terms

Term	Definition
Naturalness	How closely the speech resembles a genuine human conversation. Natural speech has appropriate timing, expression, prosody, and flow. Unnatural speech may sound robotic, rehearsed, acted, or strangely timed.
Intonation	The rise and fall of pitch across a sentence. Intonation helps communicate meaning, such as a rising pitch for a question and a falling pitch for a statement.
Pitch	How high or low the voice sounds. Sudden or misplaced pitch changes can make speech sound unnatural.
Pacing	The overall speaking speed and how that speed changes. Appropriate pacing gives the listener enough time to follow without sounding unusually slow or rushed.
Rhythm	The pattern of stressed and unstressed speech, including the timing between words and phrases.
Emphasis	Extra stress placed on an important word or phrase. Effective emphasis supports meaning without sounding forced or exaggerated.
Fluency	How smoothly and continuously a speaker talks without excessive pauses, repetition, stumbling, or restarts. Minor fillers can still sound natural.
Disfluency	An interruption in speech flow, such as “um,” “uh,” repetition, a false start, a self-correction, or a prolonged sound. Disfluencies can be natural when they fit the context and timing.
Pause	A brief period of silence within or between phrases. A well-placed pause supports meaning and breathing; a misplaced or overly long pause may sound unnatural.
Pronunciation	How accurately a word is spoken.
Enunciation	How clearly speech sounds are formed so the words are easy to understand.
Expressiveness	The amount and appropriateness of vocal variation, energy, and emotion. Expressiveness should fit the context rather than sound flat or overperformed.
Volume	The loudness of the voice. Volume should be stable, audible, and appropriate to the context.
Audio artifact	An unintended sound defect, such as a glitch, click, clipping, distortion, or abrupt audio change. If the response remains evaluable, include the defect in the Prosody rating rather than discarding automatically.
Speaker identity consistency	Whether the same voice remains recognizable within a turn and across the conversation. An unexplained voice change may indicate a serious Prosody issue.
Persona consistency	Whether the agent maintains a consistent character, tone, style, accent, voice quality, and energy unless the context justifies a change.

Conversation and Turn-Taking Terms

Term	Definition
Turn-taking	How speakers alternate in a conversation. Good turn-taking has natural starts and stops, minimal inappropriate overlap, and no awkward silence.
Timing	When a response, pause, interruption, or backchannel occurs. Natural timing makes the interaction feel conversational.
Barge-in	When one speaker begins talking while the other speaker is still speaking, intentionally interrupting the current turn.
Yielding	Stopping or pausing to give the other speaker the floor. An agent should normally yield when the user interrupts with a genuine new request.
Backchannel	A short signal such as “mm-hmm,” “uh-huh,” “yeah,” “right,” or “okay” that shows attention without taking the floor.
Topic change or topic switch	When the user introduces a new subject. The agent should acknowledge the change and respond to the new topic rather than continue the previous one.
Split response	One agent response divided across two or more labelled turns. Rate each labelled turn separately and do not penalize a natural continuation caused by segmentation.
Segmentation artifact	A boundary issue created when audio is divided into turns, such as a response being split or a small amount of adjacent-speaker audio being included.
Adjacent-speaker audio	A short portion of the other speaker heard at the beginning or end of a segment because the audio boundary is slightly wide. Rate the labelled response if it remains identifiable and evaluable.

Task Actions

Term	Definition
Skip	An item-level-only action used when the current annotator cannot confidently judge the item, such as an unfamiliar topic or accent, or when the audio is slow to load without an error. Skip sends the entire item to the next annotator for review.
Turn-level discard	Discarding only one unrateable turn or one rating dimension. Explain why that specific rating cannot be completed.
Item-level discard	Discarding the entire task because the overall item is unusable, such as pervasive segmentation failure, unavailable or broken audio, inappropriate or violent content, or significant unrelated material.
Audio ground truth	The rule that the recording, not the transcript, determines what was actually said and how it sounded. Do not penalize transcript errors.

APPENDIX

Original Prosody Rating Examples

These 11 examples, including their transcripts, ratings, justifications, and audio, are retained from the original reference file. Audio is embedded directly in this HTML, so no separate files are required.

5 Rating 5: Excellent Prosody

Prosody that sounds fully natural, expressive, and appropriate to the conversational context. You would not question that this is a real human speaking naturally.

Example A

Rating 5

User

Well, yeah. I mean, I've tried to fix a stuck zipper on my jacket before by moving the slider up and down, but it didn't always work.

Agent

If moving the slider does not help, it usually means that some fabric is stuck in the zipper or that the slider's not gripping the teeth tight enough. Sometimes you can pull the fabric away gently or squeeze the slider a little bit with pliers to fix it. If moving it up and down does not work, have you seen the zipper catch on fabric, or is it just hard to move even when nothing is in the way?

Response: (5) Very satisfied

Justification: The slowing down in intonation when saying “does not help” is very natural — it’s said in a slightly drawn-out tone which is appropriate for acknowledging the user’s frustration before pivoting to explain the cause.

Example B

Rating 5

User

You know me too well. Um, okay. How long do you think will I get at the gym?

Agent

Well, it'll be about eight minutes with typical traffic. So if you wrap your eleven o'clock on time, you should be ready to go and on a treadmill by twelve oh five.

Response: (5) Very satisfied

Justification: Natural breathing between phrases. Adds stress to “time” to emphasize the conditional, making the delivery expressive without sounding over-performed. Pacing matches the casual, helpful tone of the conversation.

4 Rating 4: Good Prosody

Prosody that is mostly natural but has minor imperfections: e.g. slightly overdone energy, an occasional unnatural pause, or a moment that sounds slightly “performed” rather than genuine.

Example A

Rating 4

User

Got it. Now, what types of flowers are best to plant for attracting bees and other pollinators?

Agent

Glad you asked, to attract bees, you might think about planting lavender, sunflowers and coneflowers because they have lots of nectar and pollen. Butterflies and hummingbirds are likely to go for bright and sweet smelling flowers like milkweed and salvia, which offer easy access to nectar. So what kind of space are you thinking of using for your pollinator friendly plants?

Response: (4) Somewhat satisfied

Justification: Nice energy and well-paced delivery of the list of flowers. However, the enthusiasm feels slightly overdone — there’s a bit too much effort in it to fully convince me it’s genuine rather than performed.

Example B

Rating 4

User

Ah. I see. Okay, well you mentioned that the thickness of the magma and the amount of gas affect how a volcano erupts, so could you tell me how do these things actually change what people see during an eruption.

Agent

I hear ya. Now, when magma is thick and sticky, it traps more gas, so eruptions are usually more explosive and throw out lots of ash and rocks. Now, if the magma's thin and runny, the gases escape more easily, which leads to slower and quieter flows of lava down the sides of the volcano.

Response: (4) Somewhat satisfied

Justification: The opening phrase “I hear ya” sounds a bit awkward and doesn’t quite fit this conversational flow. There’s also unnecessary emphasis on the word “volcano” that feels forced. Otherwise the pacing and intonation are good.

3 Rating 3: Mixed Prosody

Prosody that has clear issues alongside acceptable moments. May sound robotic in parts, have unnatural pauses, or shift between natural and “reading” styles within the same turn.

Example A

Rating 3

User

Hey, I have a question. What's the main difference between weather and climate?

Agent

Sure, so weather is the day-to-day conditions we see outside like rain and sunshine. And climate is the average weather over a long period like decades. And basically, weather tells us what to wear today and climate tells us what clothes have to have in our closet.

Response: (3) Neutral

Justification: The agent comes across as bored and reading off a sheet of paper — the pacing is unnatural and it does not pause at natural points in the sentence to help facilitate understanding. The intonation is flat throughout.

Example B

Rating 3

User

Alright, I'll keep that in mind. Can you tell me what are some ways to get past the fear of making mistakes when you're learning to speak a new language?

Agent

Sure, just reminding yourself that making mistakes is part of the learning process and it takes some pressure off since no one gets everything right the first time. And you might also try to focus on getting your message across and not worry so much about being perfect. Most people care more about understanding what you mean than being perfect grammar.

Response: (3) Neutral

Justification: Smooth and approachable with a casual tone for the first part. Then the delivery shifts to a “teacher reading off the page” style starting around “Most people care more about understanding what you mean than being perfect” — the intonation becomes flat and monotone.

Example C

Rating 3

User

Okay, so what happens if the hypothesis is wrong?

Agent

So you revise the hypothesis and test again, learning from mistakes.

Response: (3) Neutral

Justification: The pause before “learning” is way too long to sound natural — it breaks the sentence flow and sounds like the speaker lost their place.

2 Rating 2: Poor Prosody

Prosody with multiple compounding issues — robotic tone, flat delivery, unnatural pacing, misplaced pauses, or noticeable audio artifacts that are still intelligible.

Example A

Rating 2

User

So I was wondering, can you tell me what are some effective ways to promote environmental awareness.

Agent

Well, so education is a key part of promoting awareness, and there are events like cleanup drives, which can help people see the impact directly. Now, by sharing tips and stories through social media, you can reach a lot of people.

Response: (2) Somewhat dissatisfied

Justification: The voice sounds robotic and flat. Intonation sounds unnatural throughout. The pace is too slow, and there are too many pauses between words — many of them are placed inappropriately (e.g., pausing mid-phrase rather than at clause boundaries).

Example B

Rating 2

User

mhm Oh, fascinating Yeah, that would be great to connect with with other people from all over. So then, can I connect with these people outside of the courses?

Agent

Yes, many of the courses and platforms have social networking options for connection.

Response: (2) Somewhat dissatisfied

Justification: Odd pacing throughout — the delivery feels rushed and compressed, with no natural breathing room between phrases. The overall effect is flat and mechanical.

1 Rating 1: Bad Prosody

Severe prosody failures — jarring audio artifacts, voice inconsistency (sounds like different speakers), or delivery so unnatural it’s distracting.

Example A

Rating 1

User

mhm Hmm Well, you really broke that down for me. So yes, I can see how philosophies play a part in shaping the goals and methods of modern movements.

Agent

Well, it's interesting to see how social movements use those old philosophical ideas to push for change and balance personal and societal goals. When you consider this balance in social movements, is there a particular movement that stands out to you?

Response: (1) Very dissatisfied

Justification: After the word “old,” there is a jarring sound artifact on “philosophical” that interrupts the flow and naturalness of the conversation. It’s extremely distracting and off-putting.

Example B

Rating 1

User

Awesome, sounds perfect. Thanks so much for the help!

Agent

You've got this, Alex. Have a great night! you got it enjoy the movie

Response: (1) Very dissatisfied

Justification: The voice has three noticeably different variations within one short response — it does not sound like the same person speaking. Pacing and expressiveness are also very poor; the overall delivery sounds completely unnatural.

FINAL STEP

Ready for the Assessment

Once you have gone through the full guidelines and listened to the audio examples, confirm below. Your confirmation is shared with the project team so you can be moved to the assessment.