A new approach for cleansing geographical dataset using Levenshtein distance, prior knowledge and contextual information.
Journal:
Studies in health technology and informatics
Published Date:
Jan 1, 2015
Abstract
Epidemiological studies are necessary to take public health decisions. Their relevance depends on the quality of data. Doctors in continuous care collect a big amount of data that can be used for epidemiological purpose, but spatial data may be dirty; based on city names, the localization is imprecise, even more if it is misspelled. The only way to identify a city without ambiguity is to use its identifier, which can be retrieved by cleansing geographical textual data. In France, cities are organized in administrative zones called departments and some city names are shared by several cities in several departments. The clear identification of the department and the city name allows to deduce the city unique identifier and to make some spatial analysis such as epidemiological studies. In this paper, we propose a method to cleanse such data, using several steps. After having standardized the text to cleanse, we use the Levenshtein distance to generate a first set of propositions. Finally, the propositions are filtered, by removing the less likely candidates, so that it remains only one, which becomes the chosen city. Tested on a dataset of 9818 entries, we obtained 89.1% of concordance, whereas the standard Levenshtein distance obtained 70.5%. This demonstrates that our method has better results.