EMK:Improvements to Water Values
Aims
Water Value Series
Currently a run is restricted to the use of a single annual water value profile. This means sequences where the water values change significantly require multiple runs to model. I can see two ways to deal with this.
The first method is to produce annual water values starting at the end of a run, and then projecting these values backwards over the run. This method would produce a single dated water value sequence that is valid over the length of a run regardless of its length.
The second method would use annual water values always, but to allow them to be assessed starting at any date in the run. This method would produce a series of annual water value profiles. The run would then switch between profiles as it progressed.
The second method may allow more flexibility for user judgement on when profiles are likely to be reassessed. However, I have a preference for the first method as it is simpler, more widely understood from a theoretical viewpoint, and would not produce artefacts from switching from one profile to the next. This paper briefly describes water values calculated using the first method which is based around Stochastic Dynamic Programming (SDP) techniques.
Multi-Dimensional Water Values
The current system of controlling water release in EMarket operates each hydro system only with regard to its own stored energy. This method is clearly unrealistic in some situations, despite the fact that it manages total storage in the country fairly reasonably, with Waitaki dominating the control of marginal storage release. However, the other systems tend to be operated more loosely than would be expected in the real market as they are unable to anticipate system-wide shortage or excess.
With five major controlled reservoirs and two uncontrolled, the levels of which all potentially affect water values at any reservoir, a comprehensive multi-dimensional analysis of water value is quickly overloaded with complexity, inevitably resulting in excessive computation time for often marginal gain.
I propose a compromise consisting of a two dimensional profile for each reservoir, where two values determining water values are the local storage and the total of all other storage in the country. A water value profile could consist of, for each hydro system, $/MWh values for a set of storage values, e.g. 20 levels from empty to full for the local reservoir, 5 levels for the other national storage levels, giving 100 values in total. Water value at any level would be interpolated between these points
SDP can then be used to determine the two dimensional matrices for each hydro-system. It is still possible to use 1-in-N security factors to drive the cost of shortage, but this would probably need to be done by iterating the SDP and leading to excessive run times, so the design outlined in this paper uses another approach.
Retail/Hedge Commitment
Hydro operators will inevitably offset their generation earnings risks with wholesale purchase agreements and hedges. The effect of these pressures can be allowed for in water value modelling, but care needs to be taken that this is done in a balanced manner. In as far as retail and hedge commitment is required by the water value algorithm one average commitment power value (MW) per reservoir is all that is required for each weekly time step and scenario (A scenario is assumed to be one inflow sequence, but in could in theory extend to include multi-dimensional scenarios, for example, an inflow-demand scenario. But typically, runs and their respective WVs will be set up with approximately 80 historical inflow scenarios).
EMarket currently allows for the input of company portfolio data. This information could be used to generate commitment values, but there are two difficulties with this approach:
- extra care is required of the user to enter reasonable portfolio figures;
- where two or more reservoirs are owned by the same entity, the commitment should be shared – which would require a complication of the algorithm - or divided, which requires some assumptions about the reservoirs.
I don’t think co-optimising Manapouri and Waitaki (because they are both owned by Meridian) would impact greatly on the outcome, and to do so would add considerable complication to the algorithm.
Instead I suggest that a certain level of retail commitment is assumed, based on the total rated output of the hydro system, and then profiled using the average demand profile. While this may be far from accurate as an estimate of commitment, it is an easily generated and robust value that can be used at least to complement the response of hydro management to the provision of retail demand. The extent to which inaccuracies in this approach are unacceptably high will need to be tested.
Other Operating Considerations
Some reservoirs have strategic values that fall outside the profit maximisation objective of the normal water value relationships. Most importantly, low storage levels can be seen as undesirable without consideration of the lack of future revenue they represent. This is because a greater level of management, both operationally and politically, is required when security of supply is perceived to be threatened. Examples could include the analog of the Waitaki storage buffer currently used in EMarket, or perhaps the cost of risk aversion which acts to keep storage higher going into winter due to the risk of low winter inflows.
In order to include these considerations in water value calculation an additional marginal value of storage can be added. For example, if storage below 600GWh was considered undesirable to the extent that every GWh below 600 incurred a cost of $50 per week this can be expressed in the water values by adding a $50 * 0.168 = $8.4 cost per MWh to the water values after each weekly back projection (The figure of 0.168 is obtained by dividing 168 hours in a week by 1,000 MWh).
Another example is the mitigation of flood protection costs. This consideration may put a negative premium on high storage level, reducing their marginal value, possibly even below zero.
These ‘strategic’ costs may vary with the time of year as their nature can be a matter of perception and policy. In general the addition of costs such as these to the water value algorithm provides a flexible and fairly meaningful way of adapting the algorithm to mimic real life policies.
I propose that a number of additional marginal costs can be specified for each reservoir, expressed as a single $/MWh figure, a lower or higher limit to apply to the cost and a week from/week to range to apply the cost to.
Smaller Reservoirs
The current water value algorithm uses deterministic water values to evaluate smaller reservoirs. This will change with the new approach as these reservoirs need risk aware management to behave realistically. Accordingly, all reservoirs will be treated the same in the water value SDP algorithm.
Changing from the Current System
Matching Water Values to Simulation Runs
The current style of water valuation links contours to offer bands and uses this information to support two features. The first of these features is the ‘Link WVs to Offer Prices’ which responds to changes in the price of any generation tranche by adjusting the price of the related water value contour. This feature is apparently pretty much unused by modellers currently. It also suffers from problems when adjusted prices cause contours to swap.
The second feature is the ‘Adjust WVs by Location’ which also implements a price adjustment to contours, but does it by taking the location factor between the contour and its related offer into account. This feature is currently being made use of extensively and is essential given that water values are calculated on a ‘flat’ pricing grid whereas simulation runs use full nodal pricing. The water value adjusters, therefore, ensure that hydro offers adjust to match changing patterns of power flow across the grid.
Both contour adjustment features will have to be dropped when using a water value profile based on price alone, but use of the location factor adjustment suggests that some incorporation of location based pricing might be desirable.
It is possible that some form of locational pricing could to be incorporated into the water value generation, but this will potentially slow the model down significantly so an efficient scheme for applying the dispatch model will need to be found. A method with potential is to make location factor adjustments to water values which a re referenced to the demand-weighted average price (DWAP).
Performance
The current water valuation is very efficient with CPU usage and a new algorithm will inevitably be slower, but I hope to keep the time required to less than 20x the current time. As a highly fast and flexible modelling tool, we certainly do not want EMarket to suffer the same fate as the South American SDDP (which is used to a surprising extent in New Zealand), and given the small fraction of time that each multi-inflow run spends on calculating the current water values, even a 20x slowdown is not anticipated to degrade overall performance significantly.
Impact on Modellers
Input Parameters
I propose that when the new water value methodology is introduced the old method will be available for as long as is necessary for users to adapt their methods to the new method.
The old inputs that will be retired are:
- Link WVs to Offer Prices
- Major hydro Security factors
- Waitaki buffer, spill and reduction ratios
- Contour Separation (currently obsolete)
New Inputs are:
- Storage level additional costs/penalties, by Reservoir (replaces, for example, the Waitaki Low Buffer feature)
- Purchase commitment factor, either for all systems or for each system
- Market Power factor (used for the retail/hedge commitments), either for all systems or for each system, and possibly initially as hidden default market power factor
- Shortage Cost, possibly as a default system-wide shortage cost, say 100000 $/MWh.
The most important input parameters that do not translate from an old run one using the new water valuation calculation are those concerning spill and shortage buffers. These roughly correspond to the new storage level costs but no direct translation is possible.
I propose that the storage level costs, purchase commitment factors and market power factors be defined as part of the hydro system generator. A default value for purchase commitment and market power would be entered into all existing and new systems, with no storage level costs. This done, any existing run will be ready to be used to generate the new water values.
Water Value Profiles
The new water value profiles consist of a (W + 52) * N * M set of values for each hydro system, where W is the number of week in the run and N and M are the number of storage divisions for local and other storage. The profile is also given a start date. Unlike the previous profiles, the new profiles are fixed to a certain date, have an end date and have limited applicability beyond these dates. The profiles will be saved in the data directory as CSV files, allowing them to be processed or edited.
Viewing the water value profiles will need a somewhat different viewer to the one currently in use in EMarket (The current viewer makes use of the 1-1 relationship between offer bands and water value contours, a technique originally developed for use in ECNZ’s SPECTRA model and adapted into EMarket). There are no explicit price contours in the new format, but the viewer could show contours for a standard set of price values, e.g 10 dollar contours up to 100, 20 dollar contours up to 200, 50 up to 500 and so forth. The contours for any one value of ‘other storage’ can be view at a time.
Performance
As mentioned before, water valuation will definitely take longer under the new algorithm. I will aim to keep the total time for calculation to not much more than 10 minutes on a good machine. To put this in context, a dual core 3 GHz PC running Windows Vista calculates water values now in about 25 seconds and takes 1 hour 15 minutes to complete an all-inflows day-night run with reserves modelling turned on, but excluding generating reports. If the new water values take 10 times as long as the current water values then this adds 3.5 minutes to the run and 20 times longer adds 7.5 minutes.
There is a lot of uncertainty in my ability to include all the desired features of the algorithm described below and meet this performance constraint, so the design will necessarily be flexible for now.
Comparison to Current Water Values
The proposed new SDP-based water values will produce simulated behavior different to that observed in EMarket at present. There are three key expected benefits:
- More realistic management of individual reservoirs - reservoirs will be managed by the model taking the country-wide storage directly into consideration and so will not be subject to unanticipated shortage and surplus of supply.
This should result in less full and empty events in reservoirs than currently occurs, e.g. at present Taupo and Hawea tend to empty completely during a very dry winter. This in turn will result in corresponding changes in the output of the large marginal thermal stations, e.g. Huntly may tend to run harder earlier going into a dry winter. - Enhanced ability to model management policy – adjustments to hydro system policy based on purchase commitment and shortage or spill aversion should allow reservoirs to mimic some of these strategic objectives in a meaningful way.
- Better management of smaller reservoirs – use of stochastic water values for smaller reservoirs should result in more realistic management of storage.
Appendix 1: Description of the Proposed Water Valuation Algorithm
Initiation Stage
Determine Annual Cycle
At the end of the water valuation horizon a set of values needs to be produced to represent the ongoing value of storage. This can be done by taking a year of demand and offer values and using them to produce an iterated annual water value set. The year following the end of the defined run could be used (NOTE: this means realistic demand and offers should be defined over this period).
Collect Offers and Load
Do a dummy run, with no dispatch and running only those schedules marked to include in water valuation, to collect non-hydro offers and load for every four hourly period of the run and the year following the run.
Determine response curves for all reservoirs
This difficult step builds a profile of each hydro system that is not at the level of detail used in the main model cycle, but is sufficient for calculating water values. The proposed method will be a change from the previous algorithm, which has hard-coded the discretionary and must-run inflows for the purposes of generating two offers from each reservoir.
First a profile of generation by release and inflow will be generated. This is done by modelling release as pro-rata (by reservoir capacity) over all reservoirs in the system, adding inflows, again pro-rata (by average inflow) and determining how much of the resulting flows can be converted to generation. Once this is done a process can determine – for any given total inflow – a generation/release profile which can be used to produce offers.
From this point on only total inflow will be used to model inflows for each hydro system.
The profile will be generated using the hydrological constraints for the starting period, if hydrological constraints change significantly over the period of the run a new profile would be appropriate. For the mean time no allowance will be made for this sort of change, although this is a feature that could be added later.
Offers Pre-Processing
Generating hydro offers for every step in the valuation below would be a computationally onerous part of the procedure. It will be much more efficient to generate offers for every week and scenario, with the price of system offers expressed as a multiple of the water value, rather than absolute price. Once this is done hydro offers can be quickly generated from this set, given the current water value.
At the same time as the step above is taken, non-hydro offers will be collected for all periods in each week and scenario and combined into weekly offers. Load can also be collected for each weekly period.
The output of the pre-processing will be a weekly offers set, with water valuation multipliers for hydro offers and load for each node by week.
Valuation Stage
Annual Water Values
Each system is allocated an annual water value profile consisting of an N by M by 52 matrix, where N is a fixed number of storage levels for the system, M a fixed number of storage levels for the rest of the country. Water values are set to zero for all reservoirs initially.
Dispatch
For dispatch a linear programming (LP) problem will be generated for each dispatch with the usual objective (min offer price x dispatch). The constraints are the total energy balance and the DC link capacity constraint, using a LP allows other transmission constraints to be added as needed.
Dispatch will be done for a particular week and scenario given a set of system water values. This is done using the pre-processed offers and load set described above. A multiplier will be used to account for losses, this figure may be generated by running the LP twice if this does not impact too greatly on performance - otherwise a running average may be used. The output of the dispatch process is a set of generation figures for each hydro system, which can then be converted into storage displacements.
Back Projection
Each step of back projection consists of the following steps for a single entry in the Water Value Profile Matrix for a given system.
Get the current water value of the system from the current water value profile.
Calculate water value for all other systems from the current water value profile where their storage is set by the ‘rest of country’ storage level.
Dispatch for each scenario using these water values. Use the resulting generation figures to determine change in storage for the current system, and all other systems combined. Calculate the water value of the resulting position on the matrix representing this storage combination in the following week. Resulting storage projections that exceed full are assigned a water value of zero, those that are on or below empty are assigned the shortage cost.
The average water value figure generated above will be the back projected water value. The objective of solving the water value calculation is to ensure the back projected water value is equal to the water value at all storage points. Once these values are near enough to equal the water value is said to have converged at that point.
A single back projection step will be defined here as the calculation of all points in the water value profile for all systems for a single week. Convergence will be achieved in two different ways, one method for the annual water values (Recall that the annual water values are the water values relating to the year tacked onto the actual run period. These water values define the value of water at the end of the run period and define the starting values for the backward projection) and another for the run period water values.
Convergence of Annual Water Values
The first method will be applied to the annual water values and take advantage of the fact that these values need to be recalculated in an annual cycle. Each week will be recalculated a number of times as part of this cycle, so convergence of the water values can also be achieved during this iteration.
The annual water value calculation cycle will proceed as follows, starting with any reasonable seed water values:
Back project each week starting with the last week and proceeding to the first
After each back projection, set the water values for that week to be a combination of the existing and newly calculated water values, e.g. the midpoint of the two values.
Repeat the annual cycle above until water values have converged.
Convergence of Current Run Water Values
Once the annual water values have been generated, the water values for the weeks in the run can be produced by back projecting over the period of the run, starting at the last week of the run. Each week will be converged before proceeding to the week before. The convergence of a week’s water values will proceed as follows:
Initially set the week’s water value to the same as the following week and then repeat the following cycle until V converges:
- Back project the week
- Save current water value and the resulting back projected water value, call these Vi and Bi for now, where i is the iteration number.
- If in the first step of the cycle simply set the new water value to back projected water value. Otherwise retrieve the last water value, back projected water value pair (Vi-1, Bi-1) and combine with the current pair using the equation:
This is a version of the secant method for finding the solution. However, in addition to this projection a limit should be placed on how closely the new value can approach either Vi or Vi-1, for example one quarter of the difference between the two values. This is because the B = f(V) relationship, while it should be monotonic, may have a highly fluctuating gradient.
Retail/Hedge Commitment Adjustment
This can be done at offers pre-processing stage. The principle used is that a fixed proportion of the total load belongs to the reservoir. This proportion will represent an average power commitment that is a given proportion of the system’s total output. For example if the total power rating of the Waitaki system is 1600MW then an average retail/hedge power commitment of 1600 x 0.5 = 800MW could be assumed, representing a total annual commitment of 7000 GWh. If total annual load is 40000GWh then the Waitaki would be linked to about 0.17 of the load in any week.
Once a system is linked to a load figure, offers can be adjusted accordingly. Offering below the total load should be discounted and offers above given a premium. There is no simple answer to how this adjustment should be made, as it would usually depend on other offers in the market. The method outlined in the Company Optimisation document is also suitable for use here, however a ‘market power factor’ is required for any offer adjustment, and this value needs to be calculated from a full offer stack. The generation of a market power factor could occur in the offers pre-processing stage.
Appendix 2: Water Value Calculation - Objective Equations
The objective of the new water value calculation module in EMarket is to solve the equations below.
Indexes used are:
| w | week |
| t | non-hydro offer tranche |
| h | hydro system |
| q | inflow year |
| T | hydro system tranche |
| n | node |
| c | circuit |
| a | additional cost/benefit objective (The additional objective parameters are supplementary to the core water value equations (see Input Parameters section) |
Inputs
The following variables are treated as inputs to the water value module
| wstart, wend | starting and ending weeks of the run |
| Lwn | Load by week and node (MWh) |
| Qwt, Pwt | Weekly offer prices, quantity by week, tranche (MW, $/MWh) |
| Nt | Node of non-hydro tranche |
| Ihwq | Inflow by system, week, inflow year (Mm3) |
| Sh | System total storage quantity (Mm3) |
| Kh | Shortage cost by hydro system ($/MWh) |
 |
Node of hydro system |
| nfromc, ntoc | Node from and to for each circuit |
| Cc | Capacity of each circuit |
| Oh(I, R) | System generation as a function of inflow and release rate (MW) |
| Awa | Added/reduced value of water by additional objective ($/MWh/week). * The additional objective parameters are supplementary to the core water value equations (see Input Parameters section) |
| ha,sminasmaxa | Hydro system effected by objective and storage range over which the objective applies |
After pre-processing
The variables below are generated from the inputs before entering the solution cycle
| Pmaxhwq, Rminhwq | Maximum and minimum release by system, week, inflow year (Mm3/s) |
| Δh(I, G) | System storage rate of change as a function of inflow and generation (Mm3/s) |
 |
Weekly hydro offer quantity by system, week, hydro system tranche |
 |
Weekly hydro offer price multiplier by system, week, hydro system tranche |
| Hnc | Circuit power flow multiplier by node and circuit |
Water Value Equations
The equations below are solved by repeated iteration.
Water Values (Vhwso) are generated for each system and week at fixed levels of local storage – s and other storage – o. The water value at any given week, local storage and other storage is 0 when full, at shortage cost when empty and interpolated between the calculation points otherwise. If additional objectives exist and apply to the storage level they are added to the value.
1.
Where 
is the next local storage level under s, is the next local storage exceeding s; likewise with the o values and other storage and
Back Projection
All water values satisfy the formula:
2.
The change in storage for a given hydro system, week storage levels and scenario (∆hwsoq)is calculated by dispatching with
- The load for the week
- Non-hydro offers for the week
- Hydro offer quantities for the week and scenario
- Hydro offer prices for the week and scenario given the water value at the week and storage
- transmission constraints
The dispatch equation is expressed using these additional variables:
 |
Hydro system offer price by system, hydro tranche ($/MWh) |
 |
Hydro system generation by system, hydro tranche (MW) |
 |
Non-hydro generation by tranche (MW) |
The hydro offer prices are generated using the current water values and the tranche price multipliers:
The dispatch is a linear minimisation problem:
3.
When dispatch is solved the storage charges are calculated:
4.
The storage changes are then entered into formula 2 to complete the recursive relationship.