Development of an open IE benchmark

Abstract

Open Information Extraction (Open IE) is the task of extracting triples (subject, predicate, object) from plain text. Tools that perform these extractions are called Open IE extractors. While benchmarks to evaluate the performance of these tools exist, they provide aggregate metrics and do not have room for fine-grained analysis. This benchmark addresses this problem by assigning category labels to test sentences so that it is apparent on what type of sentence the extractor is failing.