2017 Transportation Forum
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation during Severe Weather Events by Fusing and Assimilating Diverse Data Sources Xu Liang Department of Civil and Environmental Engineering University of Pittsburgh
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Acknowledgement Felipe Hernรกndez and Daniel Luna (PITT) Yao Liang and Rui Wang (IUPUI) William Teng (NASA/ADNET) Jason Norville, James Long, Matthew Delp, Charles Martin, Vince Mazzocchi, and Nevin Myers (PennDOT) 2
zeroto60mph.com
nce.co.uk
topspeed.com
Hydrologic threats to the transportation infrastructure
nydailynews.com
3
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Real-time hydrologic disaster forecasting and response for severe weather events Automated process for forecasting hydrologic disasters and proposing response procedures:
Assimilate data
Fuse data
Run hydrologic model to compute critical variables
Assess the severity of the estimated events
Identify potentially affected areas
Retrieve real-time hydrologic data and weather forecasts
4
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
How to monitor all the threatened bridges at present? Pennsylvania’s scour-critical bridges Category A Category B Category C
3448 watersheds
5
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
PennDOT’s policy for monitoring threatened bridges
• Tr < 10 years: no bridges require inspection • 10 years ≤ Tr < 50 years: inspect cat. A bridges • 50 years ≤ Tr < 100 years: inspect cat. A and B bridges • 100 years ≤ Tr: inspect cat. A, B and C bridges
2
50 year return period
24-hour accumulated rainfall2 (mm)
Match bridges’ SCBI category with the return period of the storm:
10 year return period
100 year return period
NOAA’s Precipitation Frequency Data Server: http://dipper.nws.noaa.gov/hdsc/pfds/
6
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Current practice at PennDOT for determining post-storm bridge inspections
1
erh.noaa.gov, water.weather.gov
Gather precipitation data from the NWS for the previous 24/48 hours
7
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Current practice at PennDOT for determining post-storm bridge inspections
2 Compute the average precipitation on the watershed/drainage area corresponding to each bridge
8
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Current practice at PennDOT for determining post-storm bridge inspections
3 Manually determine the most severe duration of the storm from rain gauges nearby
9
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
4
Determine which region each watershed belongs to and estimate the return period with the corresponding IDF curve
Rain intensity
Current practice at PennDOT for determining post-storm bridge inspections 100 y
50 y Mean intensity
Rain return period (T)
25 y 10 y
Duration
Rain duration
10
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Current practice at PennDOT for determining post-storm bridge inspections
Compare return periods with category thresholds to determine which bridges require inspection
5 11
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Hydrologic Disaster Forecast and Response (HDFR) system • … automatically retrieve, process, and fuse data from heterogeneous/diverse data sources in near real-time • … automatically compute critical variables to access and forecast hydrologic disasters • … significantly reduce forecast uncertainties through state-of-the-art parameter calibration and data assimilation algorithms using modern hydrological models
• … achieve near real-time forecasts for hazards management and responses
12
Data
Fuser
Modeling
Assimilation
Severity
Components of the Hydrologic Disaster Forecast and Response (HDFR) system
13
Data
Fuser
Modeling
Assimilation
Severity
Data modules Status: 12 completed, 2 pending • NASA NLDAS-2 (gridded land observations) and TRMM (precipitation) • NOAA RFC (GRASS) multi-sensor precipitation • NOAA MADIS METAR, HYDRO: meteorological stations • NOAA NWS GFS (GRASS), NAM: global and continental forecasts • USGS hydrometeorology • NASA MODIS (GRASS): satellite snow cover • NOAA SNODAS (GRASS): snow cover and snow water equivalent • NASA LPRM: satellite soil moisture • NASA GPM (satellite precipitation) and SMAP (satellite soil moisture) • NASA SSW (multiple) 14
Data
Fuser
Modeling
Assimilation
Severity
NOAA RFC radar precipitation: Completed
1 hour accumulated rainfall (mm)
This data source includes quality-verified gauge-validated radar precipitation estimates •
Coverage: Most of the United States’ territory
•
4 km HRAP grid, hourly, 1 hour latency
•
Variables: precipitation only
15
Data
Fuser
Modeling
Assimilation
Severity
NWS Global Forecast System: Completed
Surface temperature 1/21/15 (k)
This data source includes gridded hydrometeorological forecasts computed using numerical models •
Coverage: Entire planet
•
0.25°, 0.5° and 1° grid, 3-hourly, 240-hour forecast every 6 hours
•
Variables: 358, including values for the precipitation, soil, planet surface, and several layers of the atmosphere
16
Data
Fuser
Modeling
Assimilation
Severity
USGS hydrometeorology: Completed
This data source includes measurements from USGS rain gauge and river monitoring stations •
Coverage: 1.5 million sites throughout the United States
•
Point data on select locations, temporal resolution and latency varies, typically 1 hour for both
•
Variables: primarily runoff, gage height, and precipitation
17
Data
Fuser
Modeling
Assimilation
Severity
NASA MODIS: Completed
This data source contains snow cover estimates based on the MODIS instrument on the Terra and Aqua satellites •
Coverage: Global
•
0.05° grid, daily, 1 day latency
•
Variables: snow cover and albedo
18
Data
Fuser
Modeling
Assimilation
Severity
NOAA SNODAS: Completed
Assimilates satellite, airborne, and ground observations into a model to provide combined estimates •
Coverage: Contiguous United States
•
30 arc-sec grid, daily, 1 day latency
•
8 variables: including snow water equivalent, depth, and temperature
19
Data
Fuser
Modeling
Assimilation
Severity
GPM: Completed
This data source contains precipitation estimates based on observations from a satellite constellation •
Coverage: Global (60°S - 60°N)
•
0.1° grid, 30 minute, 6 hour latency
•
Variables: precipitation
20
Data
Fuser
Modeling
Assimilation
Severity
Integration with GRASS GIS GRASS GIS
PyGrass
C Library (compiled)
Python Library
New Python module
21
Data
Fuser
Modeling
Assimilation
Severity
Multiscale Kalman Smoother based framework*
1
Initialization
// Compute total mean scaleSum ← 0 scaleCount ← 0 foreach scale in scales do if scale has observations then mean ← compute observations’ mean scaleSum ← scaleSum + mean end if end foreach totalMean ← scaleSum/scaleCount // Initialize values foreach scale in scales do assign values for P, Q, and R scale.fallibility ← get initial fallibility foreach cell in scale do cell.value ← totalMean end foreach end foreach
3
Downward sweep
2
Upward sweep
grid ← get grid with finest scale foreach cell in grid do fuse observations (Eq. 4, 6) end foreach grid ← get next coarser scale repeat newFall ← propagate fallibility (Eq. 1) foreach cell in scale do propagate from children (Eq. 2) fuse observations (Eq. 4, 6) end foreach grid.fallibility ← newFall grid ← get next coarser scale until scale doesn’t exist
scale ← get coarsest scale repeat foreach cell in scale do smooth from parent (Eq. 3) end foreach scale.fallibility ← update fallibility (Eq. 5) scale ← get next finer scale until scale doesn’t exist
*
Parada, L. M. & Liang, X. (2004), “Optimal multiscale Kalman filter for assimilation of near-surface soil moisture into land surface models,” Journal of Geophysical Research, AGU, 109(D24).
22
Data
Fuser
Modeling
Assimilation
Severity
MKS fuser improvements • 2 datasets with different spatial resolutions and different projections • Arbitrary target projection • Presence of “no-data” values (extensible to point measurements) • Spatiotemporal interpolation module
• Removal of the database dependency • Integration with GRASS 23
Data
Fuser
Modeling
Assimilation
Severity
Precipitation fusion NLDAS-2
NWS precipitation
24-hour accumulated rainfall 1/10/10 8:00 (mm)
24
Data
Fuser
Modeling
Assimilation
Severity
Components of the Hydrologic Disaster Forecast and Response (HDFR) system
25
Data
Fuser
Modeling
Assimilation
Severity
Juniata watershed â&#x20AC;&#x201C; storm 1 20 18
Return Period (years)
16 14 12 10 8 6 4 2 0 9/4/04
9/14/04 TQ Measured
TP - Watershed
9/24/04 TP - HDFR
10/4/04 TP - PennDOT
26
Data
Fuser
Modeling
Assimilation
Severity
West Branch Susquehanna watershed â&#x20AC;&#x201C; storm 1 35
Return Period (years)
30 25 20 15 10 5 0 9/4/04
9/14/04 TQ Measured
TP - Watershed
9/24/04 TP - HDFR
10/4/04 TP - PennDOT
27
Data
Fuser
Modeling
Assimilation
Severity
Juniata watershed â&#x20AC;&#x201C; storm 2 5.0 4.5
Return Period (years)
4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 2/26/11
3/8/11 TQ Measured
TP - Watershed
3/18/11 TP - HDFR
TP - PennDOT
3/28/11
28
Data
Fuser
Modeling
Assimilation
Severity
West Branch Susquehanna watershed â&#x20AC;&#x201C; storm 2 3.5
Return Period (years)
3.0 2.5 2.0 1.5 1.0 0.5
0.0 2/26/11
3/8/11
TQ Measured
TP - Watershed
3/18/11
TP - HDFR
3/28/11
TP - PennDOT
29
Data
Fuser
Modeling
Assimilation
Severity
Watersheds of scour-critical bridges Pennsylvaniaâ&#x20AC;&#x2122;s scour-critical bridges Category A
Category B Category C
3448 watersheds
30
Data
Fuser
Modeling
Assimilation
Severity
Distribution of watershed size 10000
Small-scale modeling
1000
Area (km2)
100
10
1
Large-scale modeling
0.1
0.01
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percentage of watersheds
31
Data
Fuser
Modeling
Assimilation
Severity
Distributed Hydrology Soil Vegetation Model Bug fixes: • Misplaced initial state on previous time step • Re-initialization of stream channel storage • Absence of water table calculations in the initial time step • … Lettenmaier, Dennis et al., “Distributed Hydrology Soil Vegetation Model,” University of Washington, Pacific Northwest National Laboratory, available online at: http://www.hydro.washington.edu/Lettenmaier/Models/DHSVM/
32
Data
Fuser
Modeling
Assimilation
Severity
Indiantown Run watershed
Calibration DHSVM model 100 24 1,472 4 21 33,455 1
m resolution variables/cell cells soil layers reaches state variables h time step
Sensitivity analysis: 7 parameters per soil type 3 parameters for forest cells Manningâ&#x20AC;&#x2122;s n for streams 22 months (+ 10 months spin-up) Multi-objective: NSE, MARE, bias Ensemble optimization: GA, CMA-ES, MetroACO, GD 33 2531 evaluations
Data
Fuser
Modeling
Assimilation
Severity
Calibrated model performance 100000
0
10
10000
20
1000
40
50
100
60
Precipitation (mm/h)
Discharge (l/s)
30
70
10
80
90
1
100
Measured
Modeled
Precipitation
34
Data
Fuser
Modeling
Assimilation
Severity
Distribution of watershed size 10000
Large/robust bridges
Small-scale modeling
1000
Area (km2)
100
10
High threat!
Low threat
1
Large-scale modeling
Small/vulnerable bridges
0.1
0.01
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percentage of watersheds
35
Data
Fuser
Modeling
Assimilation
Severity
DHSVM-GRASS integration Elevation
Stream definition
Stream width estimation
36
Data
Fuser
Modeling
Assimilation
Severity
OPTIMISTS (Optimized PareTo Inverse Modeling through Integrated Stochastic Search)
Root samples
Front 3
Observations
Simulation
Front 2
Error
Streamflow
Random samples
Assimilation time step
Front 1 (Pareto front)
Time
Initial distribution of state variables
Likelihood
Target distribution of state variables
Evolutionary optimization
HernĂĄndez, F. and Liang, X.: Hybridizing sequential and variational data assimilation for robust high-resolution hydrologic forecasting, Hydrol. Earth Syst. Sci. Discuss., (September), 1â&#x20AC;&#x201C;25, doi:10.5194/hess-2016-454, 2016.
37
Data
Fuser
Modeling
Assimilation
Severity
OPTIMISTS performance â&#x20AC;&#x201C; Indiantown Run (DHSVM) Scenario 2 Scenario 1 Scenario 2 Assimi. period Forecast period 0
Streamflow (l/s)
10000
1000
0
10000 10
10
20
20
30
30
1000 40
40
50
50
60
60
100 70
70
100
0.022 / -0.003 / 54.3% 0.301 / 0.394 / 41.3% 10
80
-0.428 / -0.733 / 58.3%
90
0.451 / 0.212 / 23.2%
100 10
Observations
Default model
Precipitation (mm/h)
Scenario 1 Assimi. period Forecast period
80 90 100
OPTIMISTS
Precipitation
38
Data
Fuser
Modeling
Assimilation
Severity
Components of the Hydrologic Disaster Forecast and Response (HDFR) system
39
Data
Fuser
Modeling
Assimilation
Severity
Storm severity module â&#x20AC;&#x201C; single duration
PennDOTâ&#x20AC;&#x2122;s IDF Maps
Precipitation [mm]
Return Period Time Series [years]
40
Data
Fuser
Modeling
Assimilation
Severity
Storm severity module â&#x20AC;&#x201C; multi-duration Duration 1
Maximum Return Period [years]
Duration 2
Associated Aggregation [Dt]
Time Stamp 41
Data
Fuser
Modeling
Assimilation
Severity
Severity curves for other variables Lackawanna River at Old Forge, PA 25000
20000
Pearson Type III
Streamflow (cfs)
Gumbel 2 years
15000
5 years
10 years 25 years 50 years
10000
100 years 200 years 500 years 1000 years
5000
0 1
Log-Normal
10
Duration (days)
100
42
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Screenshots of the HDFR under GRASS. Left: the HDFR module menu. Right: examples of data sets either downloaded using the data modules or computed using analysis modules
43
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Severity calculation using HDFR where precipitation information is downloaded, fused, and the return period is estimated. This precipitation event shows a return period of 45 years
44
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Severity calculation using HDFR with forecasted precipitation GFS NAM
MKS Forecast: 3:00-9:00 am, March 25th, 2017
Severity RETURN PERIOD
DURATION
TIME STAMP
45
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Severity calculation using HDFR with forecasted precipitation (video)
46
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Pending activities • SMAP data at the SSW; module completion • LPRM soil-moisture module development
• Testing and integration of data modules into GRASS • Testing of OPTIMISTS on high-resolution models
• Improvements to the fast flood severity estimation module
47
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Acknowledgement This work is supported in part by US DOT under the Award # OASRTRS-14H-PIT, and in part by PennDOT under the Award PITT WO #009 Disclaimer: The views, opinions, findings and conclusions reflected in this presentation are solely those of the authors and do not represent the official policy or position of the USDOT/OST-R, or any State or other entity. USDOT/OST-R does not endorse any third party products or services that may be included in this presentation or associated materials. 48
Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources
Thank You! Any Questions?
49