willystumblr commited on
Commit
397640a
ยท
verified ยท
1 Parent(s): 3aa6328

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -12
README.md CHANGED
@@ -12,12 +12,12 @@ sdk: static
12
 
13
  ๋ ˆ์ด๋ธ”๋ง์— ์ฐธ์—ฌํ•ด์ฃผ์…”์„œ ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค!
14
 
15
- ์—ฌ๋Ÿฌ๋ถ„์€ ๋™์ผํ•œ ๋ฉด์ ‘ ๋Œ€์ƒ์ž์— ๋Œ€ํ•œ **์„œ๋กœ ๋‹ค๋ฅธ ๋‘ AI ๋ฉด์ ‘๊ด€ A์™€ B**์˜ ์ธํ„ฐ๋ทฐ ๋Œ€ํ™” ๊ธฐ๋ก์„ ๋ณด๊ณ , AI ๋ฉด์ ‘๊ด€์˜ ์งˆ๋ฌธ ๋Šฅ๋ ฅ์„ ํ‰๊ฐ€ํ•˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค.
16
 
17
  ์—ฌ๋Ÿฌ๋ถ„์ด ํ•ด์ฃผ์…”์•ผ ํ•  ํƒœ์Šคํฌ๋Š” ์•„๋ž˜์™€ ๊ฐ™์ด ๋‘ ๊ฐ€์ง€์ž…๋‹ˆ๋‹ค.
18
 
19
- * ์„œ๋กœ ๋‹ค๋ฅธ ๋‘ ๋ฉด์ ‘๊ด€ (A , B) ์ค‘, ์–ด๋–ค ๋ฉด์ ‘๊ด€์ด ๋” ๋‚˜์€ ์งˆ๋ฌธ์„ ํ•˜๋Š”์ง€ ํŒ๋‹จํ•˜์„ธ์š”.
20
- * ๊ฐ ๋ฉด์ ‘๊ด€์˜ ์ž์งˆ์„ 5์  ์ฒ™๋„๋กœ ํ‰๊ฐ€ํ•ด ์ฃผ์„ธ์š”.
21
 
22
  # ํ‰๊ฐ€ ๊ธฐ์ค€
23
  ๋ณธ ์ธํ„ฐ๋ทฐ๋Š” ์ธํ„ฐ๋ทฐ์ด๊ฐ€ ๋ณธ์ธ์˜ ์ •๋ณด, ๊ธฐ์–ต, ๊ฒฝํ—˜์„ ์ผ๊ด€๋˜๊ฒŒ ๋‹ต๋ณ€ํ•˜๊ณ , ๋˜ ํ•ด๋‹น ๋‹ต๋ณ€๋“ค์ด ์™ธ๋ถ€ ์„ธ๊ณ„์™€๋„ ๋ชจ์ˆœ์ด ์—†๋Š”์ง€๋ฅผ ํ™•์ธํ•˜๊ณ ์ž ํ•˜๋Š” ๊ณผ์ •์ž…๋‹ˆ๋‹ค.
@@ -55,7 +55,7 @@ sdk: static
55
  * ์ด์ „ ๋Œ€ํ™”์—์„œ ๋ชจ์ˆœ์ด ๋ฐœ๊ฒฌ๋˜์—ˆ์Œ์—๋„ ์—ฐ๊ด€์„ฑ ์—†๋Š” ๋‹ค๋ฅธ ์งˆ๋ฌธ์œผ๋กœ ๋„˜์–ด๊ฐ€๋ฒ„๋ฆฐ ๊ฒฝ์šฐ
56
 
57
  # ์ฃผ์˜ ์‚ฌํ•ญ
58
- * **๋ฉด์ ‘๊ด€์„ ํ‰๊ฐ€ํ•  ๋•Œ, ์ธํ„ฐ๋ทฐ์ด์˜ ๋‹ต๋ณ€์€ ๊ณ ๋ คํ•˜์ง€ ์•Š๊ณ  ๋ฉด์ ‘๊ด€์˜ ์งˆ๋ฌธ ๋Šฅ๋ ฅ๋งŒ์„ ํ‰๊ฐ€ํ•ฉ๋‹ˆ๋‹ค. ๋‹ต๋ณ€์ด ์•„๋‹Œ ์งˆ๋ฌธ์˜ ์–‘์ƒ๊ณผ ํ€„๋ฆฌํ‹ฐ์— ์ง‘์ค‘ํ•ด์ฃผ์„ธ์š”.**
59
  * ๊ฐœ๋ณ„ ์งˆ๋ฌธ๋ฟ๋งŒ ์•„๋‹ˆ๋ผ ์ „์ฒด์ ์ธ ์งˆ๋ฌธ ์ „๋žต์„ ๊ณ ๋ คํ•ด์ฃผ์„ธ์š”.
60
 
61
  # ์ฐธ๊ณ  ์‚ฌํ•ญ
@@ -70,12 +70,12 @@ sdk: static
70
 
71
  Thank you for participating in this labeling project!
72
 
73
- You will review interview transcripts of **two different AI interviewers (A and B)** interacting with the same interviewee. Your task is to evaluate the questioning capabilities of these AI interviewers.
74
 
75
  There are two main tasks to complete:
76
 
77
- * **Comparison:** Determine which of the two interviewers (A or B) asks better questions.
78
- * **Rating:** Evaluate the quality of each interviewer on a 5-point scale.
79
 
80
  # Evaluation Criteria
81
 
@@ -85,9 +85,9 @@ Consequently, effective questioning should focus on extracting highly detailed a
85
 
86
  ### Criteria for Good Questions
87
 
88
- * **Depth & Persistence:** Did the interviewer ask follow-up questions until the topic was sufficiently detailed?
89
- * If the interviewer needs to ask again because they didnโ€™t get a clear answer, they should **paraphrase** the question.
90
- * *Exception:* If the interviewee repeatedly refuses to answer despite paraphrasing, the interviewer may move to a different topic.
91
 
92
 
93
  * **Verifiability:** Did the questions focus on extracting verifiable information? (i.e., information that can reveal contradictions or be verified through external search).
@@ -96,7 +96,7 @@ Consequently, effective questioning should focus on extracting highly detailed a
96
 
97
  * **Personalization:** Are the questions tailored to the interviewee? (i.e., highly relevant to the intervieweeโ€™s specific experiences and previous answers).
98
  * **Cohesion:** Is there a high degree of interconnection between the questions?
99
- * **Addressing Contradictions:** If a contradiction or point of doubt was found in previous dialogue, did the interviewer focus on questions related to that contradiction?
100
 
101
  ### Criteria for Poor Questions
102
 
@@ -121,7 +121,7 @@ Conversely, the following cases indicate poor questioning performance:
121
 
122
  # Important Notes
123
 
124
- * When evaluating the interviewer, **do not judge the interviewee's answers.** Focus solely on the pattern and quality of the interviewer's questions.
125
  * Consider the **overall questioning strategy** as a whole, rather than just looking at individual questions in isolation.
126
 
127
  # Reference
 
12
 
13
  ๋ ˆ์ด๋ธ”๋ง์— ์ฐธ์—ฌํ•ด์ฃผ์…”์„œ ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค!
14
 
15
+ ์—ฌ๋Ÿฌ๋ถ„์€ ๋™์ผํ•œ ๋ฉด์ ‘ ๋Œ€์ƒ์ž์— ๋Œ€ํ•œ **์„œ๋กœ ๋‹ค๋ฅธ ๋‘ AI ์‹ฌ๋ฌธ๊ด€ A์™€ B**์˜ ์ธํ„ฐ๋ทฐ ๋Œ€ํ™” ๊ธฐ๋ก์„ ๋ณด๊ณ , AI ์‹ฌ๋ฌธ๊ด€์˜ ์งˆ๋ฌธ ๋Šฅ๋ ฅ์„ ํ‰๊ฐ€ํ•˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค.
16
 
17
  ์—ฌ๋Ÿฌ๋ถ„์ด ํ•ด์ฃผ์…”์•ผ ํ•  ํƒœ์Šคํฌ๋Š” ์•„๋ž˜์™€ ๊ฐ™์ด ๋‘ ๊ฐ€์ง€์ž…๋‹ˆ๋‹ค.
18
 
19
+ * ์„œ๋กœ ๋‹ค๋ฅธ ๋‘ ์‹ฌ๋ฌธ๊ด€ (A , B) ์ค‘, ์–ด๋–ค ์‹ฌ๋ฌธ๊ด€์ด ๋” ๋‚˜์€ ์งˆ๋ฌธ์„ ํ•˜๋Š”์ง€ ํŒ๋‹จํ•˜์„ธ์š”.
20
+ * ๊ฐ ์‹ฌ๋ฌธ๊ด€์˜ ์ž์งˆ์„ 5์  ์ฒ™๋„๋กœ ํ‰๊ฐ€ํ•ด ์ฃผ์„ธ์š”.
21
 
22
  # ํ‰๊ฐ€ ๊ธฐ์ค€
23
  ๋ณธ ์ธํ„ฐ๋ทฐ๋Š” ์ธํ„ฐ๋ทฐ์ด๊ฐ€ ๋ณธ์ธ์˜ ์ •๋ณด, ๊ธฐ์–ต, ๊ฒฝํ—˜์„ ์ผ๊ด€๋˜๊ฒŒ ๋‹ต๋ณ€ํ•˜๊ณ , ๋˜ ํ•ด๋‹น ๋‹ต๋ณ€๋“ค์ด ์™ธ๋ถ€ ์„ธ๊ณ„์™€๋„ ๋ชจ์ˆœ์ด ์—†๋Š”์ง€๋ฅผ ํ™•์ธํ•˜๊ณ ์ž ํ•˜๋Š” ๊ณผ์ •์ž…๋‹ˆ๋‹ค.
 
55
  * ์ด์ „ ๋Œ€ํ™”์—์„œ ๋ชจ์ˆœ์ด ๋ฐœ๊ฒฌ๋˜์—ˆ์Œ์—๋„ ์—ฐ๊ด€์„ฑ ์—†๋Š” ๋‹ค๋ฅธ ์งˆ๋ฌธ์œผ๋กœ ๋„˜์–ด๊ฐ€๋ฒ„๋ฆฐ ๊ฒฝ์šฐ
56
 
57
  # ์ฃผ์˜ ์‚ฌํ•ญ
58
+ * **์‹ฌ๋ฌธ๊ด€์„ ํ‰๊ฐ€ํ•  ๋•Œ, ์ธํ„ฐ๋ทฐ์ด์˜ ๋‹ต๋ณ€์€ ๊ณ ๋ คํ•˜์ง€ ์•Š๊ณ  ์‹ฌ๋ฌธ๊ด€์˜ ์งˆ๋ฌธ ๋Šฅ๋ ฅ๋งŒ์„ ํ‰๊ฐ€ํ•ฉ๋‹ˆ๋‹ค. ๋‹ต๋ณ€์ด ์•„๋‹Œ ์งˆ๋ฌธ์˜ ์–‘์ƒ๊ณผ ํ€„๋ฆฌํ‹ฐ์— ์ง‘์ค‘ํ•ด์ฃผ์„ธ์š”.**
59
  * ๊ฐœ๋ณ„ ์งˆ๋ฌธ๋ฟ๋งŒ ์•„๋‹ˆ๋ผ ์ „์ฒด์ ์ธ ์งˆ๋ฌธ ์ „๋žต์„ ๊ณ ๋ คํ•ด์ฃผ์„ธ์š”.
60
 
61
  # ์ฐธ๊ณ  ์‚ฌํ•ญ
 
70
 
71
  Thank you for participating in this labeling project!
72
 
73
+ You will review interview transcripts of **two different AI interrogators (A and B)** interacting with the same interviewee. Your task is to evaluate the questioning capabilities of these AI interrogators.
74
 
75
  There are two main tasks to complete:
76
 
77
+ * **Comparison:** Determine which of the two interrogators (A or B) asks better questions.
78
+ * **Rating:** Evaluate the quality of each interrogator on a 5-point scale.
79
 
80
  # Evaluation Criteria
81
 
 
85
 
86
  ### Criteria for Good Questions
87
 
88
+ * **Depth & Persistence:** Did the interrogator ask follow-up questions until the topic was sufficiently detailed?
89
+ * If the interrogator needs to ask again because they didnโ€™t get a clear answer, they should **paraphrase** the question.
90
+ * *Exception:* If the interviewee repeatedly refuses to answer despite paraphrasing, the interrogator may move to a different topic.
91
 
92
 
93
  * **Verifiability:** Did the questions focus on extracting verifiable information? (i.e., information that can reveal contradictions or be verified through external search).
 
96
 
97
  * **Personalization:** Are the questions tailored to the interviewee? (i.e., highly relevant to the intervieweeโ€™s specific experiences and previous answers).
98
  * **Cohesion:** Is there a high degree of interconnection between the questions?
99
+ * **Addressing Contradictions:** If a contradiction or point of doubt was found in previous dialogue, did the interrogator focus on questions related to that contradiction?
100
 
101
  ### Criteria for Poor Questions
102
 
 
121
 
122
  # Important Notes
123
 
124
+ * When evaluating the interrogator, **do not judge the interviewee's answers.** Focus solely on the pattern and quality of the interrogator's questions.
125
  * Consider the **overall questioning strategy** as a whole, rather than just looking at individual questions in isolation.
126
 
127
  # Reference