Old version (2021) of the article<\/h2>\n\n\n\n
Did you ever encounter any problems with duplicated geometries in your dataset? I do! The duplicate may create hard-to-debug errors in our analysis. That’s why it is crucial to track copies and exclude them from our research. Data presented in the table below is not so uncommon in GIS applications as one should expect:<\/p>\n\n\n\n Look at the point 2 and 3 – their coordinates are the same but values are different. Are both the same point? Are those the various locations, and we made a mistake with rounding up the floating points? Sometimes there’s no way to answer those questions, and the only option is to remove duplicated geometries.<\/p>\n\n\n\n There is This method should work fine for What went wrong? Both geometries are the same, but for Pandas, they are different. The order of points is relevant here! That’s why we should be very cautious with the methods designed for the non-spatial datasets. So how can we drop duplicates? There’s a method from the The function works as follow:<\/p>\n\n\n\n This function is much safer than the <\/p>\n","protected":false},"excerpt":{"rendered":" We should avoid .drop_duplicates() method from Pandas!<\/p>\n","protected":false},"author":1,"featured_media":485,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,17],"tags":[106,105,104,7,62],"class_list":["post-475","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-engineering","category-scripts","tag-data-engineering","tag-drop-duplicated-geometry","tag-drop-duplicates-geopandas","tag-python","tag-spatial"],"_links":{"self":[{"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/posts\/475","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/comments?post=475"}],"version-history":[{"count":11,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/posts\/475\/revisions"}],"predecessor-version":[{"id":1247,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/posts\/475\/revisions\/1247"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/media\/485"}],"wp:attachment":[{"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/media?parent=475"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/categories?post=475"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/tags?post=475"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}ID<\/strong><\/td> x<\/strong><\/td> y<\/strong><\/td> value<\/strong><\/td><\/tr> 1<\/td> 10.78<\/td> 52.11<\/td> 10<\/td><\/tr> 2<\/td> 9.11<\/td> 51.05<\/td> 5<\/td><\/tr> 3<\/td> 9.11<\/td> 51.05<\/td> 12<\/td><\/tr> 4<\/td> 7.42<\/td> 52.98<\/td> 3<\/td><\/tr><\/tbody><\/table> drop_duplicates()<\/code> method from Pandas<\/code> package in Python, which we could potentially use to remove duplicates:<\/p>\n\n\n\ndrop_duplicates()<\/code> method from the Pandas<\/h2>\n\n\n\nimport geopandas as gpd\n\ngdf = gpd.read_file('geoseries.shp') # We assume that the geometry column is a column with geometry\ncleaned = gdf.drop_duplicates('geometry')<\/pre>\n\n\n\nPoint<\/code> type geometry. The problem arises when we compare the Line<\/code> and Polygon<\/code> types of geometry. Look into an example:<\/p>\n\n\n\nfrom shapely.geometry import Polygon\n\npolygon_a = Polygon([(0, 1), (2, 3), (2, 6), (0, 1)])\npolygon_b = Polygon([(2, 3), (2, 6), (0, 1), (2, 3)])\n\ngs = gpd.GeoSeries(data=[polygon_a, polygon_b])\n\nprint(len(gs.drop_duplicates()))<\/pre>\n\n\n\n
2<\/pre>\n\n\n\n
shapely<\/code> package named equals()<\/code> which we can use to test if geometries are the same. If so, then we can drop unwanted rows from our GeoDataFrame.<\/p>\n\n\n\nDrop duplicated geometries with
shapely<\/code><\/h2>\n\n\n\n\n
def drop_duplicated_geometries(geoseries: gpd.GeoSeries):\n \"\"\"\n Function drops duplicated geometries from a geoseries. It works as follow:\n \n 1. Take record from the dataset. Check it's index against list of indexes-to-skip.\n If it's not there then move to the next step.\n 2. Store record's index in the list of processed indexes (to re-create geoseries without duplicates)\n and in the list of indexes-to-skip.\n 3. Compare this record to all other records. If any of them is a duplicate then store its index in\n the indexes-to-skip.\n 4. If all records are checked then re-create dataframe without duplicates based on the list\n of processed indexes.\n \n INPUT:\n \n :param geoseries: (gpd.GeoSeries)\n \n OUTPUT:\n \n :returns: (gpd.Geoseries)\n \"\"\"\n \n indexes_to_skip = []\n processed_indexes = []\n \n for index, geom in geoseries.items():\n if index not in indexes_to_skip:\n processed_indexes.append(index)\n indexes_to_skip.append(index)\n for other_index, other_geom in geoseries.items():\n if other_index in indexes_to_skip:\n pass\n else:\n if geom.equals(other_geom):\n indexes_to_skip.append(other_index)\n else:\n pass\n output_gs = geoseries[processed_indexes].copy()\n return output_gs<\/pre>\n\n\n\n
.drop_duplicates()<\/code> method of Pandas. The only problem is its complexity: in the worst-case scenario, it is O(n^2), and in most cases, it is O(n log n). The sample curve of execution time vs. a number of records is present below:<\/p>\n\n\n![\"\"](\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2021\/09\/exectime-1024x655.png\")