Implementing Automatic Filtering with PyTorch and Transformers

Table of Links

Abstract & Introduction

Related Work

Datasets Examined

Manual Error Type Analysis and Taxonomies

Automatic Filtering for Potentially Relevant Dialogs

Statistical Analysis

Evaluation and Experiments

Discussion

Conclusion, Limitation, Acknowledgments, and References

A Integrated Error Taxonomy – Details

B Error-Indicating Sentences And Phrases

C Automatic Filtering – Implementation

D Automatic Filtering – Sentence-Level Analysis

E Task-Oriented Dialogs – Examples

F Effectiveness Of Automatic Filtering – A Detailed Analysis

G Inter-Annotator Agreement – Detailed Analysis

H Annotation Guidelines

I Hyperparameters and Baseline Experiments

J Human-Human Dialogs – Examples

C Automatic Filtering – Implementation

To implement the automatic filtering (Section 5) we use PyTorch (Paszke et al., 2019), the Transformers library (Wolf et al., 2020), and the pretrained allmpnet-base-v2 Sentence-Transformer[10]. It is based on MPNet (Song et al., 2020) and finetuned on a large corpus of sentence pairs from multiple tasks and domains, e.g., Yahoo Answers (Zhang et al., 2015) and Reddit Comments (Henderson et al., 2019), using a contrastive objective. It is a 12- layer Transformer model with a vocabulary size of 30,527 words that calculates the cosine similarity between two sentences in a 768-dimensional dense vector space.

Our compute infrastructure consists of one Tesla V100-SXM3 GPU (with 32 GB memory) and it took an average of 76 mins to run automatic filtering on one dataset.

[10] Model page in the HuggingFace Model Hub, last accessed