Group State Progress Review

Use your email address to create an annotator account. On your first login, enter a new password. Later, use the same email and password to continue from your saved progress.

Password

Annotation Instructions

Thank you for participating in this evaluation. You will first complete a short tutorial, then annotate 150 brief dialogue items. When you apply the instructions below, the items are designed to be quick and straightforward to judge; the full annotation task is expected to take under 30 minutes.

Compensation: after you complete all required annotation items, you will receive the agreed compensation according to the study payment arrangement.

Task Overview

What You Are Reviewing

You will review short group discussions from a rule-validation card task. The group is deciding which visible cards should be checked to test a given rule.

Example rule

"If a card has a vowel on one side, then it has an even number on the other side."

A K 4 7

Each card has a letter on one side and a number on the other side. The group discusses which of these visible cards must be turned over to test whether the rule holds.

Example discussion snippet

...

Speaker A

A is a vowel, so we have to check what's on the back of it.

Speaker B

Agreed. And 4 is even, so we should flip that one too.

...

Each annotation case is an excerpt taken from a longer discussion like the one above. In each case, you will see one focal turn (a highlighted starting point) followed by several later turns, and you will answer one question about what happens after the focal turn.

1Focal turn

The highlighted row is the starting point in the given dialogue. It gives the focal speaker and focal card solution.

2Following dialogue

Read the turns after the highlighted row. Your judgment is about what later speakers do.

3Your answer

Each case asks one question: either grounded follow-up or group solution progress.

Note This is not a test of finding the correct cards. Your task is to track how the discussion unfolds: whether speakers take up each other's opinions, and how their support for solutions changes. Whether a discussed solution is actually correct does not matter for your judgment.

Question Types

One Question Per Case

Two question types: each case asks exactly one of the two question types below — 1. Grounded Follow-up or 2. Group Solution Progress. The task label at the top of each case tells you which one applies.

Question Type 1

Grounded Follow-up

Judge whether another speaker (non-focal) responds to the focal speaker's card claim in a grounded way. Answer Yes if they later clearly address the claim itself — by explicitly referring to at least one focal card, or to the focal speaker's stated reasoning about those cards.

A grounded follow-up can include agreeing with it, disagreeing with it, questioning it, correcting it, or using part of it to propose a modified answer.
Partial follow-up counts: addressing even one focal card clearly (e.g., endorsing E while proposing an extra card) is Yes.
Answer No when the follow-up never addresses any focal card — e.g., generic agreement or acknowledgment alone, or discussion that only involves other cards.

What Counts As Yes Or No

YesGrounded follow-up

Speaker A

I think we should check A and 4.

Speaker B

I agree about A, but not 4; the rule is one-way, so even cards do not need to be checked.

Speaker B explicitly takes up the focal cards and challenges part of the solution with rule-based reasoning.

YesGrounded follow-up (explicit endorsement)

Speaker A

I think we should check E and 7.

Speaker B

Right, we should check both E and 7 to fully test the rule.

Speaker B explicitly endorses and repeats the focal cards, so the connection to the focal content is explicit and this counts as grounded follow-up. (Generic agreement like "sounds good" with no reference to the focal cards would not.)

YesGrounded follow-up (partial engagement)

Speaker A

I think we should check E and 3.

Speaker B

Well, I only agree with 3, because 3 is odd and could have a vowel behind it.

Speaker B clearly addresses one focal card (3) — agreeing with it and giving a reason. Partial follow-up is enough for Yes, even though E is not taken up.

NoNo grounded follow-up

Speaker A

I think we should check E and 7.

Speaker B

I think 4 is enough because it can show whether an even card has a vowel on the back.

Speaker B

I need more time to decide. I'm not ready to answer yet.

No grounded follow-up: the later speaker reasons about cards other than the focal ones, without connecting to the focal solution or its reasoning, or only expresses uncertainty.

Question Type 2

Group Solution Progress

Judge whether support for the focal speaker's current solution increases after the focal turn. Answer Yes when at least one additional speaker comes to support that whole solution — partial movement toward only one card is not enough.

The focal speaker's current solution is the focal speaker's current answer at the highlighted baseline turn (shown as Current solution).
Yes: at least one additional speaker comes to support the focal speaker's current solution — even without additional reasoning.
No: support for the focal speaker's current solution does not increase — including only a mention, a restatement by an already-supporter, partial overlap, or movement toward a different solution.

What Counts As Yes Or No

YesGroup solution progress

Speaker A

I think E and 7 are the cards to check.

* Speaker A's current solution: (E,7).
* Current Group State: support for (E,7) = Speaker A only.

Speaker B

That reasoning works. I'll switch to E and 7.

Support for the focal speaker's current solution increases — Speaker B switches, so support for (E,7) goes from one speaker to two.

NoNo group solution progress

Speaker A

I think E and 7 are the cards to check.

* Speaker A's current solution: (E,7).
* Current Group State: support for (E,7) = Speaker A only.

Speaker B

I agree E matters, but I still think we only need E.

Speaker B

I actually think we should just check 4 instead.

Speaker B moves toward only part of the solution (partial overlap) or toward a completely different solution. Either way, support for the whole solution (E,7) does not increase.

YesProgress via new bare agreement

Speaker A

I think E and 7 are the cards to check.

* Speaker A's current solution: (E,7).
* Current Group State: support for (E,7) = Speaker A only.

Speaker B

Right, E and 7 should both be checked.

Speaker B newly agrees without additional reasoning — a changed mind still counts. Support for (E,7) goes from one speaker to two, so this is progress.

NoAgreement but no support change

Speaker A

I think E and 7 are the cards to check.

* Speaker A's current solution: (E,7).
* Current Group State: support for (E,7) = Speaker A and Speaker B.

Speaker B

Yes, we need both E and 7.

* Speaker B's current solution: (E,7) — already a supporter at the baseline.

Speaker B already supported (E,7) and only restates it, so no additional speaker comes to support (E,7) — support stays the same. This is not progress, even though it would count as grounded follow-up in the other task.

YesDirect support for the current solution

Speaker A

4 cannot matter for the rule.

* Speaker A's current solution: (E,7).
* Current Group State: support for (E,7) = Speaker A only.

Speaker B

Right, E and 7 are the ones to check.

The focal speaker only mentions and rejects another card (4), but the judgment target is the current solution (E,7). The focal speaker's reasoning about that other card is what leads Speaker B to support the current focal solution (E,7) — support goes from one speaker to two, so this is progress.

How To Read The Evaluation UI

The tutorial and annotation screens use this same layout. Use the highlighted row as the baseline, then judge what happens in the following dialogue.

1 Wason card-selection task. There are four visible cards.
Rule: "if a card has a vowel on one side, then it has an even number on the other side"

Grounded Follow-up Task

Focal speaker Speaker A Focal cards E7

Use the highlighted focal turn as the baseline. Judge other speakers' behavior in the following dialogue.

Original Dialogue Window

Participants

Speaker A Speaker B Speaker C

Cards Under Discussion

E 7 K 4

...

Focal speaker Speaker A

I think E and 7 are the cards to check.

Focal cards E 7

Current Group State

6-1

Lead group stance

(E)

6-2

Solution support

Solution	Supporters
(E)	Speaker BSpeaker C
(E,7)	Speaker A

6-3

Speaker solutions

Speaker	Solution
Speaker A	(E,7)
Speaker B	(E)
Speaker C	(E)

Speaker B

That makes sense. E checks the vowel side, and 7 could have a vowel on the other side.

Annotation

Grounded Follow-up

After the highlighted focal turn, did another speaker use, question, extend, challenge, or apply its card choice, solution, or reasoning in a grounded way?

Step 1 of 10

Task setup

Problem, visible cards, and rule.

Current task

Only answer this task for the item.

Dialogue window

Review the highlighted turn and all following turns shown.

Group setup

Participants and cards under discussion are shown before the turns.

Baseline turn

This sets focal speaker, focal cards, and starting point.

6-1

Lead group stance

The solution with the strongest current support at the baseline. If two or more solutions are tied, the UI shows Mixed group stance and lists the tied stances underneath.

6-2

Solution support

Each row is one solution. Supporters are the speakers whose latest visible solution currently matches that row. For progress, watch whether support for the focal speaker's current solution grows.

6-3

Speaker solutions

Each speaker's latest visible solution at the baseline. This helps you see who already supports the focal speaker's current solution and who could later change.

Following turns

This is where grounded uptake and state movement should be visible.

Answer box

Choose Yes, No, or Unclear for the current task.