Could we run our own mesoscale model over Arabia?
Yes — and it is far cheaper than it looks. The interesting question is not whether the compute is affordable but which resolution actually buys a different forecast, and that answer is sharper than "higher is better". Here is the arithmetic, shown rather than asserted, so you can substitute your own numbers.
The short version
ICON-EU is not the answer, and cannot be. DWD's European nest covers 23.5°W–62.5°E, 29.5°N–70.5°N. Riyadh is at 24.7°N, Dubai 25.3°N, Muscat 23.6°N, Doha 25.3°N. Every capital of the Arabian Peninsula lies south of the ICON-EU domain. It reaches Türkiye, Cyprus, northern Iraq and northern Iran and stops.
ICON-LAM is. Same dynamical core, same free global boundary data we already download from DWD four times a day, and BSD-3 open source since January 2024 — you define your own domain, anywhere. That is the route.
Cost, for the whole Middle East at 6.5 km, four cycles a day, to T+48: about 12,000 core-hours a month — a quarter of one rented 64-core node, or roughly $120–360/month on spot cloud. Convection-permitting at 3 km over the same frame is about 67,000 core-hours a month, one and a half nodes, or $700–2,000.
On GPUs it is cheaper still, and the model choice stops being a choice. Derived from MeteoSwiss's operational GPU system, one A100 is worth 224 to 358 CPU cores on this workload — so the 3 km run above is 37 minutes on eight GPUs and about $300–600/month. But ICON is fully ported, physics included, and has carried a national weather service's operational duty since 2024; WRF's only production GPU path is a commercial licence, and the free community port accelerates the dynamical core only, which Amdahl's law caps near 1.7× end to end. If GPUs are on the table, the model is ICON.
Not on this box. The server this site runs on has 12 cores, no GPU, and already renders five other sites. A single 48-hour run at 6.5 km would take it 8 hours.
The arithmetic, and where it comes from
Everything below rests on two numbers, both traceable rather than guessed.
Work per grid point. NCAR publishes an operation count for its standard WRF CONUS 2.5 km benchmark: 411.7 GFLOP per timestep on a 1501 × 1201 × 35 grid. That is 63.1 million grid points, so each grid point costs about 6,530 floating-point operations per timestep — dynamics, physics and all. Non-hydrostatic cores of this generation are close enough to each other that the same figure works for ICON and MPAS to within the precision anyone should claim here.
Work a core can do. WRF and ICON are memory-bandwidth bound and sustain a small fraction of a modern core's peak. A defensible planning figure on current AMD EPYC or Intel Xeon silicon is 1.5 GFLOP/s sustained per physical core. Together those give:
per core-second
Sanity check, against a model whose runtime is public: NOAA's HRRR is 3 km over the continental US, 1799 × 1059 × 50 points, 20-second timestep. A 48-hour forecast is 8,640 timesteps, so 8.2 × 10¹¹ point-updates, so ≈ 990 core-hours — which is what HRRR is known to cost on a thousand-plus cores. The figures below are the same arithmetic applied to four candidate domains.
Four candidate domains
A · Whole Middle East, 6.5 km recommended first step
24–65°E, 11–43°N — the frame this site already publishes, so nothing about the maps, the explorer or the country links would change; only the data underneath them would get sharper. 4,070 × 3,560 km, 626 × 548 columns, 55 levels, 39-second timestep, T+48.
at 4 cycles a day
What it buys: better terrain almost everywhere — the Asir escarpment, the Hajar, the Zagros and the Yemeni highlands are all badly under-resolved at 13 km, and they are exactly where this region's rain falls. A better diurnal cycle over the peninsula. Sea breezes on the Gulf and Red Sea coasts that a 28 km model does not have the coastline to produce.
What it does not buy: thunderstorms. At 6.5 km convection is still parameterised, and parameterised convection over the Gulf in autumn is the single largest source of forecast error in this region.
B · Arabian Peninsula and the Gulf, 4 km
34–62°E, 12–34°N. 2,870 × 2,450 km, 718 × 613 columns, 55 levels, 24-second timestep, T+48.
The awkward middle. Four kilometres is inside the convective grey zone: too fine for a convection scheme to be well-posed, too coarse to resolve a storm. It is a real improvement on 13 km and a real compromise, and it is the option to take only if the budget will not stretch to C.
C · Gulf and Oman, convection-permitting at 2 km
45–60°E, 21–33°N — Shatt al-Arab to the Sea of Oman, the whole Gulf, the Hajar and the Emirati interior. 1,490 × 1,340 km, 744 × 668 columns, 60 levels, 12-second timestep, T+36.
at 2 cycles a day
This is the one that changes the forecast. Below about 3 km the convection scheme is switched off and storms are resolved, and in this region that is not a detail: the UAE's summer sea-breeze convection over the Hajar, the autumn Gulf convection that floods Dubai and Doha, and the outflow-driven haboobs that close Kuwaiti and Iraqi airports are all phenomena that a parameterised model produces in the wrong place, at the wrong time, or not at all.
D · Whole Middle East, convection-permitting at 3 km
The same frame as A, at storm-resolving resolution. 1,356 × 1,187 columns, 60 levels, 18-second timestep, T+48.
at 2 cycles a day
The destination. About one and a half dedicated 64-core nodes, or a spot-cloud bill smaller than one junior salary's worth of anything. It is worth stating plainly that this is not an exotic budget: a convection-permitting forecast for four hundred million people costs less per month than a mid-range car payment.
What about GPUs?
This changes the answer, and it changes it in a way that makes the choice of model one-sided rather than a matter of taste. It is also the part of this page worth reading closely, because "ICON runs on GPUs" is true in a much stronger sense than that phrase usually means.
WRF on GPUs: no free production path
NCAR has frozen WRF development, and GPU acceleration is not a priority for what remains. Stock WRF is a CPU code. There are exactly two ways around that and neither is what you would want:
- AceCAST (TempoQuest) — a commercial, closed-source GPU fork of WRF-ARW, refactored in CUDA and OpenACC and sold as a drop-in replacement. NVIDIA's own write-up puts it at roughly 9× a CPU node per GPU node, with 18 CPU nodes needed to match one GPU node. It works. It is a licence you have to buy, on a code you cannot inspect.
- wrf-gpu-port — a free community port of WRF 4.7.1 that injects OpenACC directives into the generated Fortran and compiles with NVHPC. It claims 6× on an RTX 5090 and has been run at 250 m. But read what it covers: the dynamical core only — small_step, big_step, diffusion, module_em — with no physics parameterisations and no advection on the GPU.
That last caveat is the whole story, and it is Amdahl's law. Physics is typically
40–60 % of a WRF timestep. If half the work gets 6× and the other half does not move,
the end-to-end speedup is 1 / (0.5 + 0.5/6) = 1.7×, not 6×. A
dynamical-core port is a real piece of engineering and an excellent starting point; it
is not yet a GPU forecast system.
ICON on GPUs: what is actually ported
This is the contrast that matters. ICON's port is not a dynamical core with the physics left behind — it is the whole forecast model, and it has been carrying a national weather service's operational duty since 2024. From the paper describing it, what runs on the GPU:
- The dynamical core, fully, with most loops parallelised along the
npromaand vertical dimensions. - Tracer transport, fully — including a purpose-built GPU implementation for the high-order schemes, which use indirect addressing that does not map to a GPU naively.
- Every physics parameterisation an NWP run needs: microphysics, turbulence, the soil scheme (on independent CUDA streams, since the tiles are independent), ecRad radiation, sub-grid orography, and convection.
- Diagnostics and the vertical interpolation to pressure and height levels, done on the GPU before anything is copied back for output.
- Latent-heat nudging of radar precipitation, ported specifically because it is called often enough to matter.
What stays on the CPU: initialisation, file I/O, and most of the data-assimilation observation operators — the forecast runs on the GPU and the observation operators do not, with the transfer between them costing 0.09 % of runtime. The LETKF analysis step is also CPU-side.
One caveat worth knowing before you plan around it: the paper states that the GPU code changes are in the open-source distribution with the exception of the data assimilation part. For a site downscaling a global run that is irrelevant. For anyone hoping to inherit a regional assimilation system, it is not.
How the speed was actually won
The optimisation history is published, and it is more interesting than a single ratio because it shows where the time goes on this class of code. From a working baseline of 125.6 s to 95.80 s — 23 % — in eight steps:
| Step | Gain | What it is |
|---|---|---|
| Compiler flags | 6.8 % | -O2 -Mstack_arrays; most of it from moving temporary arrays off the heap |
| Async execution | 1.5 % | OpenACC ASYNC so kernels and transfers overlap |
| Inlining | 1.8 % | mostly in the physics schemes |
| CUDA Graphs | 5.0 % | replays a captured kernel sequence instead of launching each one — 3× locally in the soil scheme, 3.5× in turbulence, because those are many tiny kernels and the launch overhead was the cost |
| Rank distribution | 1.0 % | which MPI rank lands on which GPU |
Compile-time nproma | 0.7 % | measured, but not used operationally |
| Mixed precision in the dycore | 8.0 % | single precision where scientific judgement says it is safe; validated against a double-precision run over a long period with no impact on the meteorological scores |
| ecRad in single precision | 0.9 % | following ECMWF's own use in the IFS; the real prize is the memory it frees |
Two things to take from that. The single largest win is mixed precision, not a clever kernel — and it was a meteorological decision validated meteorologically, not a numerical shortcut. The second largest is CUDA Graphs, which is pure launch-overhead removal: at kilometre scale on a fast GPU, a physics scheme's kernels finish faster than the CPU can queue them.
How they proved it still gives the right answer
This is the question a sceptic should ask about any GPU port, and ICON's answer is better than most.
It is not bit-reproducible, and the paper says so plainly. Fused multiply-add is applied aggressively on GPUs and the intrinsic maths functions differ between compilers and hardware, so a GPU run will not reproduce a CPU run bit for bit and never will. Anyone claiming otherwise for a code this size is not measuring carefully.
So instead of bit-matching they use probtest: build an ensemble on the CPU by perturbing selected input fields in their least significant digits, then require the GPU result to fall inside that spread. It is a statement that the GPU answer differs from the CPU answer by no more than the model differs from itself under rounding — which is the honest bar, and it runs in continuous integration on reduced domains.
On top of that, four summers (2021–2024) of verification against 159 Swiss surface stations, scoring precipitation ETS, 2 m temperature and 10 m wind MAE, and cloud cover against the COSMO model it replaced. ICON-CH1-EPS has beaten COSMO-1E on precipitation skill since summer 2023, and is slightly behind on cloud cover.
What a GPU is worth, in this site's own units
There are two defensible numbers, and it is worth having both because they bracket the answer.
The paper's own benchmark is 536.5 s on eight AMD EPYC 64-core sockets against 95.80 s on eight A100s — 5.6× socket to socket, or 512 CPU cores' worth of work in 8 GPUs, so one A100 ≈ 358 cores. That is a clean integration benchmark.
The operational configuration is the more conservative reading, because it includes everything a benchmark leaves out. ICON-CH1-EPS is 1,147,980 cells × 80 levels = 91.8 M column-levels, a 10-second timestep, and a 33-hour forecast — 11,880 timesteps, so 1.09 × 1012 column-level updates, completed in 2,642 s on 8 A100s:
per GPU-second
per A100
for efficiency
The range comes from the two CPU baselines: this page's own planning figure of 2.3 × 10⁵ updates per core-second (from the WRF operation count) gives 224, while the ICON paper's socket-to-socket comparison implies a CPU rate of 1.44 × 10⁵ and therefore 358. The truth is somewhere in there, and the CPU core-hour figures earlier on this page are at the optimistic end — which means the GPU advantage below is, if anything, understated.
The four domains again, on GPUs
GPU counts here are set by the strong-scaling floor, not by wall clock. The paper reports that ICON-CH1-EPS degrades past 12 GPUs because that puts fewer than 100,000 cells on each one and the hardware sits under-utilised — so these domains use the largest GPU count that stays above that line, capped at 8.
| Domain | Cells | GPUs | Wall clock | GPU-h / run | GPU-h / month | $/month |
|---|---|---|---|---|---|---|
| A · whole ME 6.5 km, 4×/day | 343,048 | 3 | 9 min | 0.45 | 54 | $55–110 |
| B · peninsula 4 km, 4×/day | 440,134 | 4 | 14 min | 0.94 | 113 | $115–225 |
| C · Gulf 2 km, 2×/day | 496,992 | 4 | 26 min | 1.73 | 104 | $105–210 |
| D · whole ME 3 km, 2×/day | 1,609,572 | 8 | 37 min | 4.99 | 299 | $300–600 |
At $1–2 per A100-hour. Domain D could use up to 16 GPUs before hitting the scaling floor, which would bring it to about 19 minutes.
Correction: memory is a tunable, not a wall
An earlier version of this page claimed the 3 km domain needed 800 GB of GPU memory and therefore ten A100s, from dividing MeteoSwiss's total GPU memory by their grid points. That was wrong, in the sense that it measured what they had rather than what ICON needs — they chose eight GPUs for time-to-solution, having previously run the same forecast on twelve.
What the paper actually describes is a memory knob. Radiation is the
memory hog, and ICON sub-blocks it: nproma_sub splits the radiation grid
into batches computed in sequence, so only one block is resident at a time. It exists
because the 16 GB P100s of the previous generation could not hold ICON's data
structures, and it is still how the model is tuned — the operational setting on
96 GB A100s is nproma_sub=6054. Trading it down costs parallelism inside
ecRad and buys memory.
So the right way to size the hardware is not a byte count but per-GPU density against a configuration known to fit. MeteoSwiss carries 11.5 M column-levels per 96 GB A100 in production. On that measure:
- Domains A, B and C sit at 6–7.5 M column-levels per GPU — comfortably below the proven density, on 80 GB cards.
- Domain D on 8 GPUs is 12.1 M per GPU — essentially MeteoSwiss's own operational load, but on cards with 17 % less memory. It is the one configuration here that should be sized on a rented node before anything is bought; ten GPUs brings it to a comfortable 9.7 M.
Is OpenACC a dead end?
A fair worry — the paper itself is candid that OpenACC costs maintainability, that
directives now sit on top of OpenMP and vendor-specific pragmas, and that
shared routines need an lacc flag threaded through them to know which
device they are on. It is also NVIDIA-only in practice: nvhpc is the only compiler
that fully supports the standard, AMD works only through the Cray compiler on HPE
hardware and has unresolved issues with some NWP components, and there is no Intel
GPU path at all.
But it is not a dead end, because it is being replaced from underneath rather than abandoned. Under the EXCLAIM project, ICON's dynamical core — 40–55 % of runtime depending on configuration — has been rewritten in GT4Py, a Python domain-specific language that compiles to whatever the hardware is. It is about 10 % faster than the Fortran+OpenACC version near optimal GPU configurations on GH200, and it is spliced into the existing model by a preprocessor that can substitute the generated stencils for the Fortran or run both side by side and compare them.
That is the reassuring shape for anyone deciding whether to build on this: you adopt ICON, and the model gets faster underneath you without the configuration you wrote changing.
The cost nobody quotes: boundary conditions
Every estimate on this page so far has been compute, and compute turns out to be the cheap part. The expensive part of running a limited-area model is feeding it.
ICON-LAM needs the driving model's full three-dimensional state on the lateral
boundary, refreshed every one to three hours through the forecast. DWD publishes exactly
what is required — icon_global_icosahedral_model-level_<cycle>_<fhr>_<level>_<VAR>.grib2.bz2,
on 120 model levels, for u v w t p qv qc qi — but in DWD's usual
arrangement of one file per variable per level per hour. Measured: about 3.6 MB a
file, so:
boundary time
boundary time
3-hourly boundaries
At two cycles a day that is 60 to 120 GB a day pulled from Offenbach — two orders of magnitude more than this site's entire current ICON traffic, and far more consequential than the $300–600 of GPU time. Any plan that quotes only the compute is quoting the small number.
But the same trick applies, and we already have the machine for it. `$lake`
has been pulling ICON from DWD every cycle for months and has 16 TB free on the disk it
does it with. Add the model-level fetch there — once, for the whole network rather than
once per site — and cut the Arabian subregion out with iconsub before
anything crosses the network. Our domain is about 3.5 % of the globe by area, so
what actually travels is on the order of 2 GB a cycle. It is exactly the pattern
this site already uses for the rendering data, applied to a bigger file.
The alternative — driving ICON-LAM from ECMWF's open IFS, which is byte-rangeable and already fetched here — is documented and much cheaper, but the open IFS carries 13 pressure levels rather than a model-level state, and you give up the same-core, same-physics property that makes ICON-into-ICON clean. Worth testing; not the first thing to build.
Getting it and building it
- Code: public releases at gitlab.dkrz.de/icon/icon-model under BSD-3C, GPU support included. Development branches are restricted to ICON partner institutions, so you get releases rather than the git history — which for building an operational forecast is fine.
- Toolchain: NVIDIA HPC SDK. The paper's operational build is nvhpc 23.3
with CUDA 11.8, configured with
--enable-gpuand compiled-acc=verystrict -gpu=cc80, plus a register cap on ecRad (-gpu=maxregcount:48after the single-precision conversion). A CUDA-aware MPI is wanted so halo exchanges move device to device. - The domain: a LAM grid file you generate, plus initial and lateral
boundary conditions remapped from a driving global run with
DWD's ICON Tools —
iconremapfor the horizontal interpolation andiconsubto cut the subregion out of the global icosahedral grid. ICON also wants a ring-shaped boundary grid holding only the points the lateral data live on, which is what makes the boundary I/O cheap. DWD's ICON tutorial carries a worked LAM preparation exercise. - The driving data: DWD's public ICON global run — which this site already has, on its own machine. See Data sources.
Which model
ICON-LAM — the recommendation
ICON has been BSD-3 open source since January 2024, and its limited-area mode is not a research branch: ICON-D2, DWD's operational 2.1 km forecast for Germany, is ICON-LAM. Three things make it the right choice here specifically:
- The driving run is one we already have a pipeline for. ICON-LAM wants its initial and lateral boundary conditions from ICON global, on the native icosahedral grid — same core, same physics, no interpolation across a model discontinuity. Correction to an earlier version of this page: those files are not already on our disk. What we hold is a rendering subset — surface fields and seven pressure levels, regridded — and boundary conditions need the full 3D state on model levels. That is a real new cost, and it is sized below.
- The domain is yours to draw. Unlike the fixed ICON-EU nest, a LAM domain is a grid file you generate. The Arabian domain that ICON-EU cannot reach is a configuration choice, not a request to Offenbach.
- ICON-ART adds online mineral dust to the same run. In this region that is not an optional extra; see below.
WRF — the pragmatic alternative
Better documented than anything else in the field, a vastly larger community, decades of Middle East literature to calibrate against (Saudi Arabia's NCM runs an operational WRF suite; the UAE's NCM runs WRF for cloud-seeding operations), and WRF-Chem with GOCART for dust. It is easier to hire for and easier to debug. It also needs its own boundary-condition pipeline from GFS or IFS, and it is somewhat heavier per grid point than ICON. If the constraint is people rather than compute, WRF is the right answer.
MPAS — elegant, and not yet
MPAS's variable-resolution global mesh is a genuinely better idea for this problem: refine to 3 km over Arabia, relax to 60 km over the Pacific, and there are no lateral boundaries at all — no nesting artefacts, no boundary-relaxation zone eating the edge of your domain, no dependence on someone else's file arriving on time. The catch is that you pay for the coarse part of the globe too, which roughly doubles the bill for a given refined area, and the operational tooling around it is thinner than ICON's or WRF's. Worth revisiting; not the first thing to build.
What the money does not buy
This is the part usually left out, and it decides whether the project is honest.
No data assimilation means downscaling, not forecasting
A cold start from global analysis inherits the global model's errors and adds 6–12 hours of spin-up while the model's own boundary layer and cloud field come into balance. What a limited-area run without assimilation gives you is a better-resolved version of ECMWF's forecast: sharper terrain, a real diurnal cycle, real sea breezes, real orographic rain. What it does not give you is a better synoptic forecast. If the global model has the trough in the wrong place, so will yours, in higher definition.
Regional 3D-Var or EnKF over this domain would need observations, and that returns us to the finding this whole site is built on: the regional observing network is not openly available. Assimilating METARs from Gulf airports and satellite radiances is possible and would help; assimilating the radar network that actually exists across Saudi Arabia, the UAE, Qatar and Oman is not, because none of it is published.
Verification will be thin
You can verify against the METAR and SYNOP stations that reach the global exchange — a few hundred across the region, concentrated at airports. That is enough to check temperature and wind and nothing like enough to check rainfall on the Asir escarpment. Plan to be honest about it rather than to claim skill you cannot demonstrate.
Dust needs its own decision
Mineral dust is the single most consequential aerosol in this region and the field most obviously missing from every free global model here. Running it online — WRF-Chem with GOCART, or ICON-ART — multiplies the cost by roughly two to four, which turns Domain C from $700 into $1,400–2,800 a month.
Before spending that: the WMO's Barcelona Dust Regional Centre already publishes a free ten-model dust ensemble and a multi-model median over exactly this domain — MONARCH, ICON-ART, SILAM, NASA GEOS, MetOffice-UM, MOCAGE, LOTOS-EUROS, WRF-Chem and more, on open THREDDS. Taking theirs is better than running your own tenth model of the same kind, and it is free. Spend the compute on convection instead, where nobody is publishing a free product for this region at all.
A build order
- Now. Render the free global models well — which is what this site does. ICON at 13 km is already the best free grid over Arabia and most readers have never seen it drawn properly for this region.
- Add the free regional products. Barcelona Dust for dust, Meteosat IODC at 45.5°E for the satellite dust view, station observations for verification. All free, all already covering the domain.
- Before the model: the boundary-condition pipeline. Model-level ICON global
onto `$lake`, subset with
iconsub, shipped as ~2 GB a cycle. This is the part that costs real bandwidth and the part to build first, because without it the GPUs have nothing to integrate. - Phase 1 model: ICON-LAM at 6.5 km over the site's frame, 4×/day to T+48. Nine minutes a cycle on three GPUs, at about half MeteoSwiss's proven per-GPU density — a machine you can buy rather than rent. Nothing on the site changes except that a fifth model tab appears, and it is ours.
- Phase 2: convection-permitting. Either 2 km over the Gulf and Oman on the same box with a fourth card (26 minutes a cycle), or 3 km over the whole frame on eight GPUs (37 minutes, and the one configuration here worth renting before buying). Take the first if the budget is capital and the second if it is operational.
- Only then consider assimilation, ensembles, or online dust — each of which is a larger project than the deterministic run itself.