Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Datasets

We explore the ability of GPT-4 to perform ad-hoc schema based informationextraction from scientific literature. We assess specifically whether it can,with a basic prompting approach, replicate two existing material sciencedatasets, given the manuscripts from which they were originally manuallyextracted. We employ materials scientists to perform a detailed manual erroranalysis to assess where the model struggles to faithfully extract the desiredinformation, and draw on their insights to suggest research directions toaddress this broadly important task.