Repeated Text in the Kurdish (Kurmanji) File of CC-100 and the Calibration of a MinHash-LSH Near-Duplicate Detector

Abstract

In post hoc analyses of CC-100’s Kurmanji file, extra paragraph copies hold 22.5% of the tokens, extra document copies 5.5% and near-duplicates at Jaccard 0.8 up to 7.3% (levels overlap); 6,004 paragraphs occur over twelve times, whereas one copy per monthly snapshot allows at most twelve. A MinHash-LSH detector finds, averaged over hash draws, 99.2% of the document pairs at that threshold, nearly as predicted. Its frozen calibration test passes only under a later-chosen validity reading; stricter later rules fail, as they often would even for a calibrated detector; the cost hypothesis fails. An exact search takes minutes.