Skip to content

Pagination and large queries

A server answers a query one page at a time, and each page but the last links to the next. iter_pages() yields one Page per response as it arrives, and collect() reads every page into one QueryResult. Both follow the next links to the end, and only a page without one ends the traversal.

A WaterData page asks for up to 10,000 features unless params set a limit, which WaterData caps at 50,000. FabricData pages ask for up to 1,000 and GeoConnex pages 500 by default. The last page, and any page of a small query, holds fewer.

from water_ogcapi import WaterData

DC = (-77.12, 38.80, -76.91, 38.99)

wd = WaterData()
for page in wd.iter_pages("monitoring-locations", bbox=DC, params={"limit": "200"}):
    print(page.feature_count, "features")

A page from iter_pages() holds the decoded document in body and its bytes in content. collect() keeps neither, unless keep_pages=True keeps the bytes.

Splitting a WaterData query

WaterData can split one query into pieces that run concurrently, as the USGS efficiency guidance recommends for long records and long site lists. split_datetime="year" or "month" splits a closed datetime interval at calendar boundaries in UTC, and sites_per_piece splits a site list. The transport runs up to max_per_origin pieces at once, 4 by default, and pages arrive in the order they finish:

daily = wd.collect(
    "daily",
    params={
        "monitoring_location_id": "USGS-01646500",
        "parameter_code": "00060",
        "statistic_id": "00003",
        "datetime": "2021-01-01/2025-12-31",
    },
    split_datetime="year",
)
print(len(daily.geojson["features"]), "days in", daily.pagination.page_count, "pages")

Each piece costs at least one request, so size pieces near one page: a year of daily values for one site is 365 features, well under one page.

Each page from iter_pages() carries a Checkpoint whose pending tuple lists every request still left. A loop that keeps only the latest page pays for one tuple at a time, and collect() keeps no page's checkpoint, so it pays only for copying them. A loop that keeps every page of a query split into N one-page pieces holds about N²/2 references to requests: the tuples alone take 15 MiB at 2,000 pieces and 381 MiB at 10,000. To keep such pages, drop each checkpoint you no longer need with dataclasses.replace(page, checkpoint=None).

Converting page by page

QueryResult.to_arrow() needs every feature in Python beside the table it builds. Page.to_arrow() converts one page, so a loop over iter_pages() holds a few pages at a time: the page it last yielded and the next one, or for a split query up to max_per_origin pages in flight plus the last yielded. Each page types its columns on its own, and promote_options="permissive" joins pages where a column is missing or null throughout. A column whose types cannot combine, such as an integer on one page and a string on another, still raises:

import pyarrow as pa

tables = [
    page.to_arrow()
    for page in wd.iter_pages(
        "daily",
        params={
            "monitoring_location_id": "USGS-01646500",
            "parameter_code": "00060",
            "statistic_id": "00003",
            "datetime": "2021-01-01/2025-12-31",
        },
        split_datetime="year",
    )
]
first = tables[0].schema.field("geometry").metadata
assert all(t.schema.field("geometry").metadata == first for t in tables), "CRS differs"
table = pa.concat_tables(tables, promote_options="permissive")
print(table.num_rows, "rows from", len(tables), "pages")

The join keeps the first table's geometry field metadata, which names the CRS, and raises nothing when another table's names a different one. Each page reads its CRS from its own Content-Crs header, so the example compares the metadata before joining. To keep memory flat, write each table to its own file, such as one Parquet file per page.

Resuming a stopped query

A query that stops on a failure can continue where it stopped. Each page from iter_pages(), except the last, carries a checkpoint of the requests still to send, and iter_pages(resume=checkpoint) sends only those, never one whose page the caller already holds. A split query's checkpoint also lists the pieces still in flight, so a piece whose page had arrived but was not yet yielded is sent again. A Checkpoint pickles, so a stopped process can save it and another resume from it:

import pickle

pages = wd.iter_pages("monitoring-locations", bbox=DC, params={"limit": "200"})
first = next(pages)
pages.close()
saved = pickle.dumps(first.checkpoint)

rest = list(wd.iter_pages("monitoring-locations", resume=pickle.loads(saved)))
print(first.feature_count + sum(page.feature_count for page in rest), "features in all")
wd.close()

collect() raises PartialResultsError when a page fails after others arrived. Its result holds the pages that arrived and the checkpoint, and collect(resume=result) returns the whole result. The Errors page shows both, and the cache, quota, and resume notebook walks through a failure.