Shared Task

We have announced our Shared Task AWARD WINNERS!

DialDoc Shared Task focuses on building goal-oriented information-seeking dialogue systems. In particular, the goal is to teach a dialogue system to identify the most relevant knowledge in the associated document for generating agent responses in natural language. It includes two subtasks for building goal-oriented document-grounded dialogue systems. The first subtask is to predict the grounding in the given document for next agent response; the second subtask is to generate agent response in natural language given the contexts.

Please join our Google Group for important notifications.

leaderboard

end user
agent
U1 :I heard about a disability benefit for clothing. What is it about?
U3: I am quite interested about such benefits.
U5: Absolutely.
U7: The latter just happened to me!
U9: True. 
A10: You have met the requirements. [5]&[6]
U11: How much do I get for such benefit?
A12: $888.88    [12]
A2: This is a disability compensation benefit known 
as an annual clothing allowance. [2]&[3]
A4: Would you like to find out if you are eligible?  [4]
A6: Any damages to your clothes that were caused by prosthetic 
or orthopedic device or your skin medicine? [7]
A8: Sorry to hear that. And Is it true that you need this device or skin
 medicine because of an injury or illness related to your military service? [8]

[1] Clothing allowance 

[2]You may be able to get money each year to help buy new clothes.  [3]This is a disability compensation benefit known as an annual clothing allowance.

[4]Can I get disability benefits from VA?

 [5]It depends on whether you meet both of the requirements listed below.  

[6]Both of these must be true:
  • [7]Your prosthetic or orthopedic device or your skin medicine causes damage  to your clothes, and
  • [8]You need this device or skin medicine because of an injury or illness related to your military service (called a service-connected condition) 

[9]What disability benefits can I get?
  • [10]A one-time payment, or
  • [11]A yearly payment

[12]Note: The current clothing allowance rate is $888.88.
Home
>
Disability Benefits
Eligibility
Clothing allowance
>
>
U13: Am I still eligible for a student loan if I get the disability benefit?
A14: Irrelevant request detected.
Figure 1: Sample conversation turns that are grounded in an associated document.
Data

Our training data is derived from Doc2Dial dataset. The dataset contains goal-oriented conversations between an end user and an assistive agent. Each turn is annotated with a dialogue scene, which includes role, dialogue act, and grounding in a document (or irrelevant to domain documents). The documents are from different domains, such as va and studentaid.

Please check out  data download  and  github repo .

Subtasks
  • Subtask1 - knowledge identification
  • Goal: identifying the grounding knowledge in form of document span for the next agent turn.
    Input: dialogue history, current user utterance and associated document.
    Output: a text span.
    Evaluation: Exact Match and F1 metrics.
  • Subtask2 - text generation
  • Goal: generating next agent response in natural language.
    Input: dialogue history and associated document.
    Output: agent utterance.
    Evaluation: sacrebleu and human evaluations.
Awards
For each subtask, we would like to reward top three participant teams with prizes $1000, $500 and $200 respectively. While we cannot recognize every team, we earnestly thank everyone who submitted to leaderboard !
The following are the award recipients:
Subtask 1
1st Prize ER
2nd Prize SCIR-DT
3rd Prize RWTH code
Subtask 2
1st Prize KU_NLP
2nd Prize RWTH code
3rd Prize CAiRE code

Call for Participation

In order to qualify for the competition prizes, each participant will be asked to make a paper submission and complete at least one of the two subtasks.
All accepted submissions will be presented at the workshop.

Submission Details

Please submit a paper that describes your models and systems to the technical system track. You can choose to describe all the methods for all participated tasks in one single paper submission.

The paper should be up to four pages including references. The format should conform to ACL submission information.

Review Process

All submissions will be peer-reviewed by at least two reviewers. The reviewing process will be double-blinded at the level of the reviewers. Authors are responsible for anonymizing the submissions

Important Dates

  • Leaderboard Submission Final Date: April 26th, 2021 (AoE)
  • Workshop Paper Due Date: May 1, 2021 (AoE)   → May 6, 2021 (AoE)
  • Notification of Acceptance: May 30, 2021 (AoE)   → June 1, 2021 (AoE)
  • Camera-ready papers due: June 7, 2021 (AoE)
  • Workshop Dates: August 5, 2021

Task Organizers

Song Feng (IBM Research)