Review request

Birdwatching log with 40k sightings, the map page takes nine seconds

Open · 231 viewsasked by ines_data

Repo or live app

git.example.com/inesferreira/birdlog ↗

Unsure about: Performance

Personal project that got out of hand: 40k rows in sightings, each with a species, a point and a date. The map fetches everything in the current bounding box and draws markers. Query is where lat between ? and ? and lng between ? and ?. No indexes beyond the primary key. Nine seconds on the first load, faster after.

Comment
A between on lat and lng with no index is a sequential scan every time. PostGIS with a GiST index on a geography column, and cluster the markers on the client. postgres_pete
PostGIS it is. Will report back. ines_data

1 answer

pawel_z

Measure before you migrate. I recreated your table with 40k fake rows and ran the bounding box of the default map view:

Two things stand out. The query is 14 ms, so Postgres is not your nine seconds. And at the default zoom the box covers the whole country, so it returns 39,812 rows of select *, around 16 MB of JSON, which the browser then turns into 39,812 markers. That's where the time goes, and why the second load is faster (the response is cached).

In order:

  • Select only id, species, lat, lng.
  • Below a zoom level, don't return points at all. Return counts per grid cell: group by round(lat::numeric, 1), round(lng::numeric, 1).
  • Cluster what's left on the client.

PostGIS with a GiST index, as Pete says, is the right move once the box is small and the table keeps growing. I'd use geometry rather than geography for a plain bounding box, the && operator maps straight onto the index. But an index won't make a query that returns every row any faster.

Comment
You're right, the default view is the whole country. The payload was 17 MB in the Network tab. I never looked there once. ines_data
Fair, I jumped straight to the index. Measure first is the better answer. postgres_pete
Grid counts when zoomed out, points from zoom 11 in. First load is under a second now. Doing PostGIS next weekend anyway. ines_data