ERA5 timeseries chunking

[windy-mark-2]

I am using the Microsoft Planetary Computer ERA5 collection to retrieve a very small spatial domain, such as a single grid point or a 3×3 grid, over several decades of hourly data. The current Zarr layout appears optimized for broader spatial access within individual monthly assets. For point-based time-series workloads, the client must open hundreds of monthly assets and read chunks containing substantially more spatial data than requested. As a result, retrieving one location over a long period can require many requests and take considerably longer than the small output size would suggest.

ECMWF now provides an alternative ERA5 data source with a layout chunked by grid node, making long time-series retrieval for one or a few locations much faster. Would Microsoft consider publishing an additional Planetary Computer representation optimized for this access pattern, with spatial chunks corresponding to individual grid nodes, or similarly small spatial units, and longer chunks along the time dimension?

This could complement rather than replace the existing layout. The current organization may remain preferable for regional or map-oriented analysis, while a point-optimized representation would support weather stations, renewable-energy sites, asset locations, and other applications that repeatedly retrieve small domains over long periods. It would also reduce unnecessary data transfer and the number of monthly asset and metadata requests required for these workloads.

See link:
https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels-timeseries?tab=analysis_ready_data

[Gustavo Hidalgo (ghidalgo3)]

We do not have plans to do this right now, but we may in the future. Could you share more details of your analysis so we can inventory use cases that would benefit from rechunking ERA5?

[windy-mark-2]

Hi Gustav,

I work in the energy industry providing long term wind resource and energy yield assessments for renewable energy assets. We use this data in the optimising of the design as well as combining with engineering and physics based model for converting the wind data to energy data.

For this, the primary focus is a limited geographic region, generally 100km, for a long time period (20+ years, at least). Accessing long time periods, in monthly chunks, creates many open requests that slows down the data access.

We look at longer time periods > 50 years when considering extreme weather events.

Its certainly not prohibitive in its current form; however providing the data in both spatial and temporal chunked zarr files would be helpful for these types of analysis'.

Mark