CogAgent - Framework

System design and architecture

Dialogue Reader

Dialogue Readers are designed for preprocessing the raw user inputs along with the dialogue history. The dialogue can either be texts or images and they are loaded in a unified approach. We provide a number of classes inherited from \texttt{BaseReader} and users can choose which utterance reader to use depending on different task requirements.

Knowledge Retriever

Knowledge Retrievers are designed to modularize the knowledge incorporating process to build plug-and-play modules. Knowledge can be obtained by analyzing the user utterance or using it as a search query. We implement various knowledge Retrievers to acquire different kinds of knowledge, such as unstructured knowledge, structured knowledge and multimodal knowledge. They should be included in Processor class based on user need. Utterance readers and corresponding processors make up the whole data processing procedure.

Response Generator

Given the aforementioned user utterances and the retrieved knowledge, response generators are required to fuse the features and generate appropriate responses while satisfying the need of dialogue safety and being empathetic. Various models featuring different aforementioned aspects are available and they have a unified interface for training, evaluating and prediction.

Core Framework

Some utilities facilitating the training and evaluation process are available. Trainer class is responsible for single-GPU model training, distributed model training, loss recording and checkpoint saving. Evaluator class is designed to evaluate the given model using different provided metrics. Toolkit class offers the interaction surface with users given a trained model. Loss, Metric and Logger are convenient modules supporting the loss function calculating, evaluation metric computing and results logging. The data storage form is provided by Datable and Datableset.

Availabel datasets of CogAGENT

Dataset	Category	Reference
MultiWOZ 2.0	Fundamental	MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
MultiWOZ 2.1	Fundamental	MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Baselines
Chinese chitchat Dataset	Fundamental	Chinese chitchat
MOD	Multimodal	DSTC10-Track1
MMConvQA	Multimodal	MMCoQA: Conversational Question Answering over Text, Tables, and Images
OK-VQA	Multimodal	Ok-vqa: A visual question answering benchmark requiring external knowledge
VQAv2	Multimodal	Making the v in vqa matter: Elevating the role of image understanding in visual question answering
WAY	Multimodal	Where Are You? Localization from Embodied Dialog
Wizard of Wikipedia	Knowledgeable	Wizard of Wikipedia: Knowledge-Powered Conversational Agents
Holl-E	Knowledgeable	Towards Exploiting Background Knowledge for Building Conversation Systems
OpenDialKG	Knowledgeable	OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs
DIASAFETY	Controllable	On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark
EmpatheticDialogues	Controllable	Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset

Enhancer module details of CogAGENT

Modal	Category	Reference
SUMBT	Fundamental	SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking
SC-LSTM	Fundamental	Semantically conditioned lstm-based natural language generation for spoken dialogue systems
BERTNLU	Fundamental	ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems
MDRG	Fundamental	Towards end-to-end multi-domain dialogue modelling
UBAR	Fundamental	owards fully end-to-end task-oriented dialog system with gpt-
GPT2 for Chinese chitchat	Fundamental	Chinese chitchat
TransResNet-Ret	Multimodal	Image-Chat: Engaging Grounded Conversations
MMBERT	Multimodal	Selecting Stickers in Open-Domain Dialogue through Multitask Learning
MAE	Multimodal	MMCoQA: Conversational Question Answering over Text, Tables, and Images
PICa	Multimodal	An empirical study of gpt-3 for few-shot knowledge-based vqa
LingUNet	Multimodal	Where Are You? Localization from Embodied Dialog
DifffKS	Knowledgeable	Difference-aware Knowledge Selection for Knowledge-grounded Conversation Generation
KE-Blender	Knowledgeable	Knowledge Enhanced Fine-Tuning for Better Handling Unseen Entities in Dialogue Generation
NPH	Knowledgeable	Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding
BERTQA	Knowledgeable	Dense Passage Retrieval for Open-Domain Question Answering
KEMP	Controllable	OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs
RobertaClassifier	Controllable	On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark