EMK Wiki - User contributions [en]

Special:Badtitle/NS106:Using the Redispatch Feature

2013-02-08T02:03:25Z

Sisep: /* Purpose */

{{#hidens:}}
=== Purpose ===
To isolate a single Trading Period of interest and examine it in detail, without having to amend or create '''''[[EMO:The Forecast Definition Window|Forecast Definitions]]''''', by altering any factor affecting the dispatch calculation and immediately re-dispatching the period to observe the outcome.

NOTE: The Re-dispatch feature only operates when the '''''Presentation Interface''''' is in '''''Scenario Data Source''''' mode.

'''''Dispatch Factors''''' that can be altered:
*Nodal Demand
*Unit Offers Quantities and Price (energy and Reserves) for all five bands
*Circuit Capacities
*Equation Constraint Limits


'''>> To Re-dispatch a single period:'''
#Place the Presentation Interface into Scenario Data Source mode
#Select the Scenario you wish to examine (or create a new one using the Create button in the Main Window Data Source Manager)
#Use the Information Grids in the Main Window Display Panel to select the factors you wish to alter before re-disptach
#Click the '''Re-dispatch button''' on the Scenario Functions Panel
#The Presentation Interface (Map and Information Grids) will now display an updated view of the period as a result of the re-dispatch with the new factor values

NOTE: You can also use separate Windows created from the Main Window to alter dispatch factor values. These windows are also updated with new information resulting from the Re-dispatch.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Scenarios|Back]]

Special:Badtitle/NS106:Scenarios and the Main Window

2013-02-08T02:02:33Z

Sisep: /* Information Grid Changes */

{{#hidens:}}
On loading a Scenario two changes take place in the '''''[[EMO:The_Main_Window|Main Window]]''''':
#A Scenario Panel appears at the top of the '''''[[EMO:The Control Panel|Main Window Control Panel]]'''''
#The Information Grids in the Display area become editable (i.e. you can change the values of certain fields for re-dispatch)

=== The Scenario Panel ===
{|class="wikitable"
! Feature !! Purpose
|-
|
:''Scenario Name Field''
|
:Displays the Scenario name
|-
|
:''Scenario Version Field''
|
:Displays the Scenario version
|-
|
:''Redispatch Button''
|
:Causes the Scenario to be re-dispatched using data updated in the Information Grids
|-
|
:''Analyse Button''
|
:Places the Main Window into Marginal Analysis Mode
|-
|
:''Save Button''
|
:Saves a new version of the current Scenario (including any changes made through the Information Grids). Note that version numbers are integers allocated sequentially
|}

=== Information Grid Changes ===
Columns containing editable data (for re-dispatch) are shaded pale yellow in the '''''[[EMO:The Display Area Information Grids|Main Window Information Grids]]'''''.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Scenarios|Back]]

Special:Badtitle/NS106:Scenarios and the Main Window

2013-02-08T02:02:18Z

Sisep: /* Information Grid Changes */

{{#hidens:}}
On loading a Scenario two changes take place in the '''''[[EMO:The_Main_Window|Main Window]]''''':
#A Scenario Panel appears at the top of the '''''[[EMO:The Control Panel|Main Window Control Panel]]'''''
#The Information Grids in the Display area become editable (i.e. you can change the values of certain fields for re-dispatch)

=== The Scenario Panel ===
{|class="wikitable"
! Feature !! Purpose
|-
|
:''Scenario Name Field''
|
:Displays the Scenario name
|-
|
:''Scenario Version Field''
|
:Displays the Scenario version
|-
|
:''Redispatch Button''
|
:Causes the Scenario to be re-dispatched using data updated in the Information Grids
|-
|
:''Analyse Button''
|
:Places the Main Window into Marginal Analysis Mode
|-
|
:''Save Button''
|
:Saves a new version of the current Scenario (including any changes made through the Information Grids). Note that version numbers are integers allocated sequentially
|}

=== Information Grid Changes ===
Columns containing editable data (for re-dispatch) are shaded pale yellow in the '''''[[EMO:EMO:The Display Area Information Grids|Main Window Information Grids]]'''''.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Scenarios|Back]]

Special:Badtitle/NS106:Scenarios and the Main Window

2013-02-08T02:01:54Z

Sisep: /* Information Grid Changes */

{{#hidens:}}
On loading a Scenario two changes take place in the '''''[[EMO:The_Main_Window|Main Window]]''''':
#A Scenario Panel appears at the top of the '''''[[EMO:The Control Panel|Main Window Control Panel]]'''''
#The Information Grids in the Display area become editable (i.e. you can change the values of certain fields for re-dispatch)

=== The Scenario Panel ===
{|class="wikitable"
! Feature !! Purpose
|-
|
:''Scenario Name Field''
|
:Displays the Scenario name
|-
|
:''Scenario Version Field''
|
:Displays the Scenario version
|-
|
:''Redispatch Button''
|
:Causes the Scenario to be re-dispatched using data updated in the Information Grids
|-
|
:''Analyse Button''
|
:Places the Main Window into Marginal Analysis Mode
|-
|
:''Save Button''
|
:Saves a new version of the current Scenario (including any changes made through the Information Grids). Note that version numbers are integers allocated sequentially
|}

=== Information Grid Changes ===
Columns containing editable data (for re-dispatch) are shaded pale yellow in the '''''[[EMO:Main Window Information Grids|Main Window Information Grids]]'''''.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Scenarios|Back]]

Special:Badtitle/NS106:Scenarios and the Main Window

2013-02-08T02:01:41Z

Sisep:

{{#hidens:}}
On loading a Scenario two changes take place in the '''''[[EMO:The_Main_Window|Main Window]]''''':
#A Scenario Panel appears at the top of the '''''[[EMO:The Control Panel|Main Window Control Panel]]'''''
#The Information Grids in the Display area become editable (i.e. you can change the values of certain fields for re-dispatch)

=== The Scenario Panel ===
{|class="wikitable"
! Feature !! Purpose
|-
|
:''Scenario Name Field''
|
:Displays the Scenario name
|-
|
:''Scenario Version Field''
|
:Displays the Scenario version
|-
|
:''Redispatch Button''
|
:Causes the Scenario to be re-dispatched using data updated in the Information Grids
|-
|
:''Analyse Button''
|
:Places the Main Window into Marginal Analysis Mode
|-
|
:''Save Button''
|
:Saves a new version of the current Scenario (including any changes made through the Information Grids). Note that version numbers are integers allocated sequentially
|}

=== Information Grid Changes ===
Columns containing editable data (for re-dispatch) are shaded pale yellow in the '''''[[EMO:The Main Window Information Grids|Main Window Information Grids]]'''''.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Scenarios|Back]]

Special:Badtitle/NS106:Forecasting and the Main Window

2013-02-08T01:49:34Z

Sisep:

{{#hidens:}}
When viewing Forecast results a Forecast Functions Panel appears at the top of the '''''[[EMO:The Control Panel|Main Window Control Panel]].'''''

=== Features ===
{|class="wikitable"
! Feature !! Purpose
|-
|
:''Name Field''
|
:Displays the name of the Forecast Definition used to produce the results
|-
|
:''From Field''
|
:'From' Date and Trading Period (first period of Effective Date Range)
|-
|
:''To Field''
|
:'To' Date and Trading Period (last period of Effective Date Range)
|-
|
:''Edit Button''
|
:Opens the '''''[[EMO:The Forecast Definition Window|Forecast Definition Window]]''''' for the Forecast
|-
|
:''Go to Start Button''
|
:Makes the first Forecast Trading Period appear as the current Trading Period in the Presentation Interface
|-
|
:''Summary Button''
|
:Places the Presentation Interface into Summary mode
|-
|
:''Date and Time Selector''
|
:Selects the '''''Current Data Set''''' from the Forecast
|}

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Forecasting|Back]]

Special:Badtitle/NS106:Input Data

2013-02-08T01:29:05Z

Sisep: /* How EMarketOffer processes Transmission System data */

{{#hidens:}}
The input data types processed by ''EMarketOffer:''
*'''''Generation Data'''''
*'''''Price data
*'''''Offers data
*'''''Demand data
*'''''Transmission data
*'''''Static (user-defined) data'''''

==Generation Data==
Generation data is used:
*As information-only in the'''''Units Information Grid''''' for the '''''Market Data Source'''''
*To calculate Powerflows for the Market Data Source
*To calculate Powerflows for the Powerflow Case Data Source

=== Loading Generation Data ===
Historical Generation data can be loaded from an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]'''''). Historical or user-created Generation data can be loaded into '''''Powerflow Case Definitions''''' from file as a '''''generation profile'''''.

==Price Data==
Historical Price data is used as information-only in the '''''Units Information Grid''''' for the '''''Market Data Source'''''

=== Loading Historical Price Data ===
Historical Price data can only be loaded into ''EMarketOffer'' through direct connection to an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]''''').

== Offers Data ==
Historical Offers data is used:
*As information-only in the '''''[[EMO:Units Information Grid|Units Information Grid]]''''' for the '''''Market Data Source'''''
*As input into the seven-day '''''Offers Profile''''' in '''''forecast Definitions'''''
*As input into '''''Scenario Data Sources''''' based on historical Trading Periods (when available)

----
NOTE: ''EMarketOffer'' does NOT use Offers to calculate Powerflows for historical Trading Periods.
----

===Loading Historical Offers Data===
Historical Offers can be loaded either through:
*'''''direct connection to an online database'''''
*from pre-formatted '''''Offers files'''''
*Offers profile files (Forecast Definitions only)

Pre-formatted Offers files must use the same formatting used by the COMIT service for providing Offers data and be named in the following manner:

:'''Offersyyyymmdd.csv

Where:
:''''Offers'''' is the text 'Offers'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

You can download Offers files directly from COMIT and use them without further editing (once renamed).

----
NOTE: Date format is important - the date in the file name is used to allocate offers data to specific week days. One Offers files cannot contain data for periods from two different calendar days.
----

Offers files are processed row by row without regard to overall integrity. This allows you to edit the raw Offers files prior to loading to:
*Alter Offer Prices and Quantities
*Restrict the number of offered periods to less than a full day
*Restrict the number of offering Generation Units to less than those actually offering on the day

The formatting for the Offers files can be found in '''''[[EMO:Offers File Format|Offers File Format]].'''''

Offers profile files can also be used. An Offers profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition. The format of this file can be found in '''''[[EMO:Offers Profile File Format|Offers Profile File Format]].'''''

===Using Offers with Forecast Definitions===
''EMarketOffer's'' Forecast Definitions are based on a weekly Offers Profile, organised by day of week (as opposed to date). Offers for no more than seven consecutive days can be loaded into a Forecast Definition.

The following points should be noted:
*Where a '''''Forecast Effective Date Range''''' extends beyond seven days, Offers from the loaded profile will be re-used (by corresponding day of the week) from day eight onward
*Where Offers for fewer than seven days are loaded into a Forecast Definition , any periods in the Effective Date Range not covered by the loaded Offers will fail to dispatch
*New Offers can be loaded over the top of existing Offers. examples of how this might be useful include:
**producing modified historical offers by loading a week profile from the database and then loading an edited and/or partial Offers file containing different data
**Modifying a single generator's offering behaviour by loading an edited file containing Offers for only the Generation Unit and specific Trading Periods of interest leaving all previously loaded Offers (for this or any other generator) unchanged.
*Offers files are loaded consecutively by date from earliest to latest. Loading more than seven-days of Offers data from file will cause over-writing of the weekly Offer profile. for example if ten days of offers are loaded from file the first three days will be lost.

== Demand Data ==
Historical Demand data is used:
*As input into calculating the Powerflows in the '''''[[EMO:Data_Sources#Market_Mode|Market Data Source]]'''''
*As input into the seven-day Demand profile for '''''[[EMO:The Forecast Definition Window|Forecast Definitions]]''''' and '''''[[EMO:Powerflow_Analysis|Powerflow Cases]]'''''

===Which Demand data to use with ''EMarketOffer?''===
For best results ''EMarketOffer's'' database connection should be configured to use reconciled SPD Daily Demand data (available to subscribed COMIT users) as this most closely matches the historical Offers used in final dispatch. However, ''EMarketOffer'' can be configured to work with SCADA or other Demand data. Where SCADA data is used the net final Load at each Market Node should also be used.

===Loading Historical Demand Data===
Historical Demand can be loaded into ''EMarketOffer'' either through:
*'''''direct connection to an online database''''',
*from pre-formatted '''''Demand files'''''
*from demand profile files

===Pre-formatted files===
Pre-formatted Demand files must comply with ''EMarketOffer's'' Demand file format (see '''''[[EMO:Demand File Format|Demand File Format]]''''') and named in the following manner:

:'''Loadyyyymmdd.csv'''

Where:
:''''Load'''' is the text 'Load'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

----
NOTE: The date format is important. ''EMarketOffer'' uses the date in the file name to allocate the Demand data to specific days. One demand file cannot contain data for periods from more than one calendar day.
----

''EMarketOffer'' processes the Demand files row by row and does not check overall integrity. This allows you to edit the raw files prior to loading into ''EMarketOffer'' to:
*Alter Demand Quantities
*Restrict the number of Demand periods to less than a full day
*Restrict the number of Demand Nodes to less than those actually drawing down power on the day

The formatting for the Demand files can be found in '''''[[EMO:Demand File Format|Demand File Formats]]'''''.

===Demand Profile Files===
A Demand profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition or Powerflow Case. The format of this file can be found in '''''[[EMO:Demand Profile File Format|Demand Profile File Formats]]'''''.

===Using Demand with Forecast Definitions and Powerflow Cases===
''EMarketOffer's'' Forecast and Powerflow Case Definitions are based on a weekly Demand profile, by day of week. Demand figures for no more than seven consecutive days can be loaded into a Definition.

The following points should be noted:
*Where a Forecast or Powerflow Case '''''Effective Date Range''''' extends beyond seven days ''EMarketOffer'' will re-use the loaded Demand by day of the week from the profile for all weeks in the Effective Date Range.
*Where Demand for fewer than seven days is loaded into a Definition, any periods in the Effective Date Range not covered by the loaded Demand will fail to dispatch
*New Demand data can be loaded over the top of existing data. For example it is possible to load a full week profile from the database and then load a partial Demand file that has been edited to contain Demand for a single Node for a specific number of periods on a specific day. In this case only those periods for the selected Node will be overwritten with new data, all other Demand figures will remain unchanged.
*Demand files are loaded consecutively by date from earliest to latest. '''''Loading more than seven days of data from file''''' will cause over-writing of the weekly Demand profile. For example if ten days of Demand figures are loaded from file the first three days will be lost.
*When '''''loading Demand data from database''''' it is possible to specify a date range covering several weeks of Demand data. In this case an averaged Demand profile is created by day-of-the-week and Trading Period. It is not possible to create averaged Demand profiles by loading from file.

== Transmission System Data ==
Transmission System data supplied by the System Operator through the EM6 and COMIT information services is used to:
*Calculate Powerflows for the '''''[[EMO:Data_Sources#Market_Mode|Market Data Source]]'''''
*Define the Transmission System configuration in Forecast and '''''[[EMO:The Powerflow Case Definition Window|Powerflow Case Definitions]]'''''

''EMarketOffer'' requires the following Transmission System data:
*Network configuration (default values for base grid)
*Circuit electrical properties
*Circuit Re-ratings
*Group Constraint descriptions

By preference ''EMarketOffer'' automatically loads Transmission System data from any files found in the '''''EM6/COMIT sub-folders''''' on start-up (unless the '''Use Grid File Inputs box''' in the '''''Source Data Administration Window''''' is checked). A full 'map' of the historical and expected network is created, allowing the exact configuration of the Transmission System to be determined for any Trading Period covered by the file data. Where Transmission System data must be obtained from an online database, the 'map' is built up period by period in response to requests made through the Presentation Interface.

===Base Transmission System===
The Branch_Data file (obtainable as a downloaded from the EM6 service) defines the properties of all Circuits (including Summer and Winter ratings) and how they are connected. Both historical and expected changes to these properties are stored in the file. Files downloaded from EM6 should be placed in the ''Grid\EM6\BaseGrid'' sub-folder named according the the following convention (the same as used by EM6):

:'''Branch_Data_ddmmyyyy.csv'''

Where:
*''''Branch_Data'''' is the text 'Branch_Data'
*''''ddmmyyyy'''' is the date the file was downloaded from EM6

On start-up ''EMarketOffer'' uses the Branch_Data file with the latest date. Where no Branch_Data file exists, ''EMarketOffer'' attempts to access data through a database connection. Branch_Data files will always be used preferentially to source Transmission System data regardless of whether a current database connection for the data has been defined or not.

''EMarketOffer'' will fail to operate if no Transmission System data can be found from any source at startup.

----
NOTE: You will need to download new branch data files from EM6 from time to time. These should be named according to the above convention and placed in the ''EM6\BaseGrid'' sub-folder. It is not necessary to remove previous Branch_Data files as only data from the file with the latest date will be used.
----

'''WARNING!''' Files provided by the system operator no longer accurately reflect historical system changes prior to 2002. If you wish to model the pre-2002 system you should use Transpower branch data files that were created close to the time you are interested in analysing. ''This will require removing Branch_Date files with later create dates from the EM6\BaseGrid sub-folder.''

===Security Constraints===
Files, in CSV format, of historical and expected Security Constraints are provided through both the EM6 and COMIT free to air services. The following table summarises the files that can be processed by ''EMarketOffer:''

{|class="wikitable"
! Source !! Data Content !! EMO Location !! EMO File Name !! Comment
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Reratings
|
:branch_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (circuit re-ratings) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Equations
|
:constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (equations) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Constraints
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:COMIT
|
:Security Constraints
|
:Input\Grid\COMIT
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:Ellsoft
|
:Security Constraints
|
:Input\Grid\Ellsoft
|
:ellsoft_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains an Ellsoft generated file containing all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|}

'''Warning!''' Ellsoft provides Constraint files to cover the historical period from the beginning of the market to a date subsequent to the mass circuit renaming exercise conducted by Transpower on the 8th December 2007. The file has been constructed using historical TPIX and recent EM6 data files and for the period 1 Jan 2007 to 31 May 2008 has used the maximum capacities from arcflow data to construct realistic branch re-ratings. This file is intended to overcome inconsistencies in constraint source data. It is provided without guarantee of accuracy; you are advised to independently confirm any data found in this file.

===How ''EMarketOffer'' processes Transmission System data===
When constructing its internal Transmission System database, ''EMarketOffer'' first analyses the base grid and creates a period-by-period 'map' of the system which includes the tracking of all changes in rating, electrical properties and overall configuration. Secondly the Branch re-ratings are applied 'over the top' of the initial map. Finally the Group Constraints are examined and a parallel map of these Constraints is created.

Two rules are observed when loading Constraint data from file:

#Data is loaded from folders in the following preference order:
##Input\Grid\Ellsoft
##Input\Grid\EM6\Constraints
##Input\Grid\COMIT
##Input\Grid\EM6\Equations and Input\Grid\EM6\Reratings
#Where files in different folders contain data for overlapping time periods, data will be taken from the file with the latest data, and from the first date in that file (see Figure 1 below)

[[File:Transmission system data.PNG|500px|thumb|centre|Figure 1 - Constraints File Handling Example]]

----
TIP: It is recommended that you keep your constraint data files from the various sources in a separate library and transfer only those files you wish to use to the ''EMarketOffer'' input data folders before launching the program.
----

In most cases ''EMarketOffer'' will be configured to use either EM6/COMIT files or online database connections (combination of there two options are supported). If you are using the EM6/COMIT file option, changing to a database link will require considerable configuration (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a networked database]]'''''), however if you are using the database option simply place EM6/COMIT files in the appropriate input date folders to override the database connection for the associated type of data.

== Static (User-Defined) Data ==
''EMarketOffer'' uses static data to calculate or organise the dynamic data produced by the Run (see the Circuit Item example below). Static data tends to remain unchanged between the Trading Periods contained in a Run,

{|class="wikitable"
! Item !! Description !! Static Data !! Dynamic Data
|-
! Circuit
|
:A branch (Circuit) conducting power on the NZ Transmission System
|
*Name
*From Node
*To Node
*Contributes to HVDC (Yes/No)
*Resistance
*Reactance
*Capacity (Summer/Winter)
|
*Powerflow
*Price Separation
*Losses
*Loading
|}

Modelled Items are defined by static data. Some Items are global, i.e. common for all Definitions, and their associated static data must be pre-defined (e.g. '''''Companies''''' and '''''Regions''''') Most Modelled Items, and their static data, are created for each new Definition (e.g. Generators or Nodes).

----
NOTE: To avoid time-consuming effort re-creating data in Definitions, Modelled Items, with their static data descriptions, can be imported into a Definition using ''EMarketOffer's'', '''''data loading functions''''', once loaded these Items can be modified, added to or removed with the Definition editor.
----

Functionality to maintain global static data can be accessed through the '''''[[EMO:The Edit Menu|Edit Menu]]''''' in the Main Window.

Static data in Definitions can be accessed through the '''''[[EMO:The Forecast Definition Window|Forecast Definition Window]]''''' or the '''''[[EMO:The Powerflow Case Definition Window|Powerflow Case Definition Window]]'''''.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Data Management|Back]]

Special:Badtitle/NS106:Input Data

2013-02-08T01:26:44Z

Sisep: /* Transmission System Data */

{{#hidens:}}
The input data types processed by ''EMarketOffer:''
*'''''Generation Data'''''
*'''''Price data
*'''''Offers data
*'''''Demand data
*'''''Transmission data
*'''''Static (user-defined) data'''''

==Generation Data==
Generation data is used:
*As information-only in the'''''Units Information Grid''''' for the '''''Market Data Source'''''
*To calculate Powerflows for the Market Data Source
*To calculate Powerflows for the Powerflow Case Data Source

=== Loading Generation Data ===
Historical Generation data can be loaded from an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]'''''). Historical or user-created Generation data can be loaded into '''''Powerflow Case Definitions''''' from file as a '''''generation profile'''''.

==Price Data==
Historical Price data is used as information-only in the '''''Units Information Grid''''' for the '''''Market Data Source'''''

=== Loading Historical Price Data ===
Historical Price data can only be loaded into ''EMarketOffer'' through direct connection to an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]''''').

== Offers Data ==
Historical Offers data is used:
*As information-only in the '''''[[EMO:Units Information Grid|Units Information Grid]]''''' for the '''''Market Data Source'''''
*As input into the seven-day '''''Offers Profile''''' in '''''forecast Definitions'''''
*As input into '''''Scenario Data Sources''''' based on historical Trading Periods (when available)

----
NOTE: ''EMarketOffer'' does NOT use Offers to calculate Powerflows for historical Trading Periods.
----

===Loading Historical Offers Data===
Historical Offers can be loaded either through:
*'''''direct connection to an online database'''''
*from pre-formatted '''''Offers files'''''
*Offers profile files (Forecast Definitions only)

Pre-formatted Offers files must use the same formatting used by the COMIT service for providing Offers data and be named in the following manner:

:'''Offersyyyymmdd.csv

Where:
:''''Offers'''' is the text 'Offers'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

You can download Offers files directly from COMIT and use them without further editing (once renamed).

----
NOTE: Date format is important - the date in the file name is used to allocate offers data to specific week days. One Offers files cannot contain data for periods from two different calendar days.
----

Offers files are processed row by row without regard to overall integrity. This allows you to edit the raw Offers files prior to loading to:
*Alter Offer Prices and Quantities
*Restrict the number of offered periods to less than a full day
*Restrict the number of offering Generation Units to less than those actually offering on the day

The formatting for the Offers files can be found in '''''[[EMO:Offers File Format|Offers File Format]].'''''

Offers profile files can also be used. An Offers profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition. The format of this file can be found in '''''[[EMO:Offers Profile File Format|Offers Profile File Format]].'''''

===Using Offers with Forecast Definitions===
''EMarketOffer's'' Forecast Definitions are based on a weekly Offers Profile, organised by day of week (as opposed to date). Offers for no more than seven consecutive days can be loaded into a Forecast Definition.

The following points should be noted:
*Where a '''''Forecast Effective Date Range''''' extends beyond seven days, Offers from the loaded profile will be re-used (by corresponding day of the week) from day eight onward
*Where Offers for fewer than seven days are loaded into a Forecast Definition , any periods in the Effective Date Range not covered by the loaded Offers will fail to dispatch
*New Offers can be loaded over the top of existing Offers. examples of how this might be useful include:
**producing modified historical offers by loading a week profile from the database and then loading an edited and/or partial Offers file containing different data
**Modifying a single generator's offering behaviour by loading an edited file containing Offers for only the Generation Unit and specific Trading Periods of interest leaving all previously loaded Offers (for this or any other generator) unchanged.
*Offers files are loaded consecutively by date from earliest to latest. Loading more than seven-days of Offers data from file will cause over-writing of the weekly Offer profile. for example if ten days of offers are loaded from file the first three days will be lost.

== Demand Data ==
Historical Demand data is used:
*As input into calculating the Powerflows in the '''''[[EMO:Data_Sources#Market_Mode|Market Data Source]]'''''
*As input into the seven-day Demand profile for '''''[[EMO:The Forecast Definition Window|Forecast Definitions]]''''' and '''''[[EMO:Powerflow_Analysis|Powerflow Cases]]'''''

===Which Demand data to use with ''EMarketOffer?''===
For best results ''EMarketOffer's'' database connection should be configured to use reconciled SPD Daily Demand data (available to subscribed COMIT users) as this most closely matches the historical Offers used in final dispatch. However, ''EMarketOffer'' can be configured to work with SCADA or other Demand data. Where SCADA data is used the net final Load at each Market Node should also be used.

===Loading Historical Demand Data===
Historical Demand can be loaded into ''EMarketOffer'' either through:
*'''''direct connection to an online database''''',
*from pre-formatted '''''Demand files'''''
*from demand profile files

===Pre-formatted files===
Pre-formatted Demand files must comply with ''EMarketOffer's'' Demand file format (see '''''[[EMO:Demand File Format|Demand File Format]]''''') and named in the following manner:

:'''Loadyyyymmdd.csv'''

Where:
:''''Load'''' is the text 'Load'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

----
NOTE: The date format is important. ''EMarketOffer'' uses the date in the file name to allocate the Demand data to specific days. One demand file cannot contain data for periods from more than one calendar day.
----

''EMarketOffer'' processes the Demand files row by row and does not check overall integrity. This allows you to edit the raw files prior to loading into ''EMarketOffer'' to:
*Alter Demand Quantities
*Restrict the number of Demand periods to less than a full day
*Restrict the number of Demand Nodes to less than those actually drawing down power on the day

The formatting for the Demand files can be found in '''''[[EMO:Demand File Format|Demand File Formats]]'''''.

===Demand Profile Files===
A Demand profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition or Powerflow Case. The format of this file can be found in '''''[[EMO:Demand Profile File Format|Demand Profile File Formats]]'''''.

===Using Demand with Forecast Definitions and Powerflow Cases===
''EMarketOffer's'' Forecast and Powerflow Case Definitions are based on a weekly Demand profile, by day of week. Demand figures for no more than seven consecutive days can be loaded into a Definition.

The following points should be noted:
*Where a Forecast or Powerflow Case '''''Effective Date Range''''' extends beyond seven days ''EMarketOffer'' will re-use the loaded Demand by day of the week from the profile for all weeks in the Effective Date Range.
*Where Demand for fewer than seven days is loaded into a Definition, any periods in the Effective Date Range not covered by the loaded Demand will fail to dispatch
*New Demand data can be loaded over the top of existing data. For example it is possible to load a full week profile from the database and then load a partial Demand file that has been edited to contain Demand for a single Node for a specific number of periods on a specific day. In this case only those periods for the selected Node will be overwritten with new data, all other Demand figures will remain unchanged.
*Demand files are loaded consecutively by date from earliest to latest. '''''Loading more than seven days of data from file''''' will cause over-writing of the weekly Demand profile. For example if ten days of Demand figures are loaded from file the first three days will be lost.
*When '''''loading Demand data from database''''' it is possible to specify a date range covering several weeks of Demand data. In this case an averaged Demand profile is created by day-of-the-week and Trading Period. It is not possible to create averaged Demand profiles by loading from file.

== Transmission System Data ==
Transmission System data supplied by the System Operator through the EM6 and COMIT information services is used to:
*Calculate Powerflows for the '''''[[EMO:Data_Sources#Market_Mode|Market Data Source]]'''''
*Define the Transmission System configuration in Forecast and '''''[[EMO:The Powerflow Case Definition Window|Powerflow Case Definitions]]'''''

''EMarketOffer'' requires the following Transmission System data:
*Network configuration (default values for base grid)
*Circuit electrical properties
*Circuit Re-ratings
*Group Constraint descriptions

By preference ''EMarketOffer'' automatically loads Transmission System data from any files found in the '''''EM6/COMIT sub-folders''''' on start-up (unless the '''Use Grid File Inputs box''' in the '''''Source Data Administration Window''''' is checked). A full 'map' of the historical and expected network is created, allowing the exact configuration of the Transmission System to be determined for any Trading Period covered by the file data. Where Transmission System data must be obtained from an online database, the 'map' is built up period by period in response to requests made through the Presentation Interface.

===Base Transmission System===
The Branch_Data file (obtainable as a downloaded from the EM6 service) defines the properties of all Circuits (including Summer and Winter ratings) and how they are connected. Both historical and expected changes to these properties are stored in the file. Files downloaded from EM6 should be placed in the ''Grid\EM6\BaseGrid'' sub-folder named according the the following convention (the same as used by EM6):

:'''Branch_Data_ddmmyyyy.csv'''

Where:
*''''Branch_Data'''' is the text 'Branch_Data'
*''''ddmmyyyy'''' is the date the file was downloaded from EM6

On start-up ''EMarketOffer'' uses the Branch_Data file with the latest date. Where no Branch_Data file exists, ''EMarketOffer'' attempts to access data through a database connection. Branch_Data files will always be used preferentially to source Transmission System data regardless of whether a current database connection for the data has been defined or not.

''EMarketOffer'' will fail to operate if no Transmission System data can be found from any source at startup.

----
NOTE: You will need to download new branch data files from EM6 from time to time. These should be named according to the above convention and placed in the ''EM6\BaseGrid'' sub-folder. It is not necessary to remove previous Branch_Data files as only data from the file with the latest date will be used.
----

'''WARNING!''' Files provided by the system operator no longer accurately reflect historical system changes prior to 2002. If you wish to model the pre-2002 system you should use Transpower branch data files that were created close to the time you are interested in analysing. ''This will require removing Branch_Date files with later create dates from the EM6\BaseGrid sub-folder.''

===Security Constraints===
Files, in CSV format, of historical and expected Security Constraints are provided through both the EM6 and COMIT free to air services. The following table summarises the files that can be processed by ''EMarketOffer:''

{|class="wikitable"
! Source !! Data Content !! EMO Location !! EMO File Name !! Comment
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Reratings
|
:branch_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (circuit re-ratings) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Equations
|
:constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (equations) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Constraints
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:COMIT
|
:Security Constraints
|
:Input\Grid\COMIT
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:Ellsoft
|
:Security Constraints
|
:Input\Grid\Ellsoft
|
:ellsoft_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains an Ellsoft generated file containing all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|}

'''Warning!''' Ellsoft provides Constraint files to cover the historical period from the beginning of the market to a date subsequent to the mass circuit renaming exercise conducted by Transpower on the 8th December 2007. The file has been constructed using historical TPIX and recent EM6 data files and for the period 1 Jan 2007 to 31 May 2008 has used the maximum capacities from arcflow data to construct realistic branch re-ratings. This file is intended to overcome inconsistencies in constraint source data. It is provided without guarantee of accuracy; you are advised to independently confirm any data found in this file.

===How ''EMarketOffer'' processes Transmission System data===
When constructing its internal Transmission System database, ''EMarketOffer'' first analyses the base grid and creates a period-by-period 'map' of the system which includes the tracking of all changes in rating, electrical properties and overall configuration. Secondly the Branch re-ratings are applied 'over the top' of the initial map. Finally the Group Constraints are examined and a parallel map of these Constraints is created.

Two rules are observed when loading Constraint data from file:

#Data is loaded from folders in the following preference order:
##Input\Grid\Ellsoft
##Input\Grid\EM6\Constraints
##Input\Grid\COMIT
##Input\Grid\EM6\Equations and Input\Grid\EM6\Reratings
#Where files in different folders contain data for overlapping time periods, data will be taken from the file with the latest data, and from the first date in that file (see Figure 1 below)

[[File:Transmission system data.PNG|500px|thumb|centre|Figure 1 - Constraints File Handling Example]]

----
TIP: It is recommended that you keep your constraint data files from the various sources in a separate library and transfer only those files you wish to use to the ''EMarketOffer'' input data folders before launching the program.
----

In most cases ''EMarketOffer'' will be configured to use either EM6/COMIT files or online database connections (combination of there two options are supported). If you are using the EM6/COMIT file option, changing to a database link will require considerable configuration (see '''''Connecting to a networked database'''''), however if you are using the database option simply place EM6/COMIT files in the appropriate input date folders to override the database connection for the associated type of data.

== Static (User-Defined) Data ==
''EMarketOffer'' uses static data to calculate or organise the dynamic data produced by the Run (see the Circuit Item example below). Static data tends to remain unchanged between the Trading Periods contained in a Run,

{|class="wikitable"
! Item !! Description !! Static Data !! Dynamic Data
|-
! Circuit
|
:A branch (Circuit) conducting power on the NZ Transmission System
|
*Name
*From Node
*To Node
*Contributes to HVDC (Yes/No)
*Resistance
*Reactance
*Capacity (Summer/Winter)
|
*Powerflow
*Price Separation
*Losses
*Loading
|}

Modelled Items are defined by static data. Some Items are global, i.e. common for all Definitions, and their associated static data must be pre-defined (e.g. '''''Companies''''' and '''''Regions''''') Most Modelled Items, and their static data, are created for each new Definition (e.g. Generators or Nodes).

----
NOTE: To avoid time-consuming effort re-creating data in Definitions, Modelled Items, with their static data descriptions, can be imported into a Definition using ''EMarketOffer's'', '''''data loading functions''''', once loaded these Items can be modified, added to or removed with the Definition editor.
----

Functionality to maintain global static data can be accessed through the '''''[[EMO:The Edit Menu|Edit Menu]]''''' in the Main Window.

Static data in Definitions can be accessed through the '''''[[EMO:The Forecast Definition Window|Forecast Definition Window]]''''' or the '''''[[EMO:The Powerflow Case Definition Window|Powerflow Case Definition Window]]'''''.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Data Management|Back]]

Special:Badtitle/NS106:Input Data

2013-02-08T01:25:09Z

Sisep: /* Demand Data */

{{#hidens:}}
The input data types processed by ''EMarketOffer:''
*'''''Generation Data'''''
*'''''Price data
*'''''Offers data
*'''''Demand data
*'''''Transmission data
*'''''Static (user-defined) data'''''

==Generation Data==
Generation data is used:
*As information-only in the'''''Units Information Grid''''' for the '''''Market Data Source'''''
*To calculate Powerflows for the Market Data Source
*To calculate Powerflows for the Powerflow Case Data Source

=== Loading Generation Data ===
Historical Generation data can be loaded from an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]'''''). Historical or user-created Generation data can be loaded into '''''Powerflow Case Definitions''''' from file as a '''''generation profile'''''.

==Price Data==
Historical Price data is used as information-only in the '''''Units Information Grid''''' for the '''''Market Data Source'''''

=== Loading Historical Price Data ===
Historical Price data can only be loaded into ''EMarketOffer'' through direct connection to an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]''''').

== Offers Data ==
Historical Offers data is used:
*As information-only in the '''''[[EMO:Units Information Grid|Units Information Grid]]''''' for the '''''Market Data Source'''''
*As input into the seven-day '''''Offers Profile''''' in '''''forecast Definitions'''''
*As input into '''''Scenario Data Sources''''' based on historical Trading Periods (when available)

----
NOTE: ''EMarketOffer'' does NOT use Offers to calculate Powerflows for historical Trading Periods.
----

===Loading Historical Offers Data===
Historical Offers can be loaded either through:
*'''''direct connection to an online database'''''
*from pre-formatted '''''Offers files'''''
*Offers profile files (Forecast Definitions only)

Pre-formatted Offers files must use the same formatting used by the COMIT service for providing Offers data and be named in the following manner:

:'''Offersyyyymmdd.csv

Where:
:''''Offers'''' is the text 'Offers'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

You can download Offers files directly from COMIT and use them without further editing (once renamed).

----
NOTE: Date format is important - the date in the file name is used to allocate offers data to specific week days. One Offers files cannot contain data for periods from two different calendar days.
----

Offers files are processed row by row without regard to overall integrity. This allows you to edit the raw Offers files prior to loading to:
*Alter Offer Prices and Quantities
*Restrict the number of offered periods to less than a full day
*Restrict the number of offering Generation Units to less than those actually offering on the day

The formatting for the Offers files can be found in '''''[[EMO:Offers File Format|Offers File Format]].'''''

Offers profile files can also be used. An Offers profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition. The format of this file can be found in '''''[[EMO:Offers Profile File Format|Offers Profile File Format]].'''''

===Using Offers with Forecast Definitions===
''EMarketOffer's'' Forecast Definitions are based on a weekly Offers Profile, organised by day of week (as opposed to date). Offers for no more than seven consecutive days can be loaded into a Forecast Definition.

The following points should be noted:
*Where a '''''Forecast Effective Date Range''''' extends beyond seven days, Offers from the loaded profile will be re-used (by corresponding day of the week) from day eight onward
*Where Offers for fewer than seven days are loaded into a Forecast Definition , any periods in the Effective Date Range not covered by the loaded Offers will fail to dispatch
*New Offers can be loaded over the top of existing Offers. examples of how this might be useful include:
**producing modified historical offers by loading a week profile from the database and then loading an edited and/or partial Offers file containing different data
**Modifying a single generator's offering behaviour by loading an edited file containing Offers for only the Generation Unit and specific Trading Periods of interest leaving all previously loaded Offers (for this or any other generator) unchanged.
*Offers files are loaded consecutively by date from earliest to latest. Loading more than seven-days of Offers data from file will cause over-writing of the weekly Offer profile. for example if ten days of offers are loaded from file the first three days will be lost.

== Demand Data ==
Historical Demand data is used:
*As input into calculating the Powerflows in the '''''[[EMO:Data_Sources#Market_Mode|Market Data Source]]'''''
*As input into the seven-day Demand profile for '''''[[EMO:The Forecast Definition Window|Forecast Definitions]]''''' and '''''[[EMO:Powerflow_Analysis|Powerflow Cases]]'''''

===Which Demand data to use with ''EMarketOffer?''===
For best results ''EMarketOffer's'' database connection should be configured to use reconciled SPD Daily Demand data (available to subscribed COMIT users) as this most closely matches the historical Offers used in final dispatch. However, ''EMarketOffer'' can be configured to work with SCADA or other Demand data. Where SCADA data is used the net final Load at each Market Node should also be used.

===Loading Historical Demand Data===
Historical Demand can be loaded into ''EMarketOffer'' either through:
*'''''direct connection to an online database''''',
*from pre-formatted '''''Demand files'''''
*from demand profile files

===Pre-formatted files===
Pre-formatted Demand files must comply with ''EMarketOffer's'' Demand file format (see '''''[[EMO:Demand File Format|Demand File Format]]''''') and named in the following manner:

:'''Loadyyyymmdd.csv'''

Where:
:''''Load'''' is the text 'Load'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

----
NOTE: The date format is important. ''EMarketOffer'' uses the date in the file name to allocate the Demand data to specific days. One demand file cannot contain data for periods from more than one calendar day.
----

''EMarketOffer'' processes the Demand files row by row and does not check overall integrity. This allows you to edit the raw files prior to loading into ''EMarketOffer'' to:
*Alter Demand Quantities
*Restrict the number of Demand periods to less than a full day
*Restrict the number of Demand Nodes to less than those actually drawing down power on the day

The formatting for the Demand files can be found in '''''[[EMO:Demand File Format|Demand File Formats]]'''''.

===Demand Profile Files===
A Demand profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition or Powerflow Case. The format of this file can be found in '''''[[EMO:Demand Profile File Format|Demand Profile File Formats]]'''''.

===Using Demand with Forecast Definitions and Powerflow Cases===
''EMarketOffer's'' Forecast and Powerflow Case Definitions are based on a weekly Demand profile, by day of week. Demand figures for no more than seven consecutive days can be loaded into a Definition.

The following points should be noted:
*Where a Forecast or Powerflow Case '''''Effective Date Range''''' extends beyond seven days ''EMarketOffer'' will re-use the loaded Demand by day of the week from the profile for all weeks in the Effective Date Range.
*Where Demand for fewer than seven days is loaded into a Definition, any periods in the Effective Date Range not covered by the loaded Demand will fail to dispatch
*New Demand data can be loaded over the top of existing data. For example it is possible to load a full week profile from the database and then load a partial Demand file that has been edited to contain Demand for a single Node for a specific number of periods on a specific day. In this case only those periods for the selected Node will be overwritten with new data, all other Demand figures will remain unchanged.
*Demand files are loaded consecutively by date from earliest to latest. '''''Loading more than seven days of data from file''''' will cause over-writing of the weekly Demand profile. For example if ten days of Demand figures are loaded from file the first three days will be lost.
*When '''''loading Demand data from database''''' it is possible to specify a date range covering several weeks of Demand data. In this case an averaged Demand profile is created by day-of-the-week and Trading Period. It is not possible to create averaged Demand profiles by loading from file.

== Transmission System Data ==
Transmission System data supplied by the System Operator through the EM6 and COMIT information services is used to:
*Calculate Powerflows for the '''''Market Data Source'''''
*Define the Transmission System configuration in Forecast and '''''Powerflow Case Definitions'''''

''EMarketOffer'' requires the following Transmission System data:
*Network configuration (default values for base grid)
*Circuit electrical properties
*Circuit Re-ratings
*Group Constraint descriptions

By preference ''EMarketOffer'' automatically loads Transmission System data from any files found in the '''''EM6/COMIT sub-folders''''' on start-up (unless the '''Use Grid File Inputs box''' in the '''''Source Data Administration Window''''' is checked). A full 'map' of the historical and expected network is created, allowing the exact configuration of the Transmission System to be determined for any Trading Period covered by the file data. Where Transmission System data must be obtained from an online database, the 'map' is built up period by period in response to requests made through the Presentation Interface.

===Base Transmission System===
The Branch_Data file (obtainable as a downloaded from the EM6 service) defines the properties of all Circuits (including Summer and Winter ratings) and how they are connected. Both historical and expected changes to these properties are stored in the file. Files downloaded from EM6 should be placed in the ''Grid\EM6\BaseGrid'' sub-folder named according the the following convention (the same as used by EM6):

:'''Branch_Data_ddmmyyyy.csv'''

Where:
*''''Branch_Data'''' is the text 'Branch_Data'
*''''ddmmyyyy'''' is the date the file was downloaded from EM6

On start-up ''EMarketOffer'' uses the Branch_Data file with the latest date. Where no Branch_Data file exists, ''EMarketOffer'' attempts to access data through a database connection. Branch_Data files will always be used preferentially to source Transmission System data regardless of whether a current database connection for the data has been defined or not.

''EMarketOffer'' will fail to operate if no Transmission System data can be found from any source at startup.

----
NOTE: You will need to download new branch data files from EM6 from time to time. These should be named according to the above convention and placed in the ''EM6\BaseGrid'' sub-folder. It is not necessary to remove previous Branch_Data files as only data from the file with the latest date will be used.
----

'''WARNING!''' Files provided by the system operator no longer accurately reflect historical system changes prior to 2002. If you wish to model the pre-2002 system you should use Transpower branch data files that were created close to the time you are interested in analysing. ''This will require removing Branch_Date files with later create dates from the EM6\BaseGrid sub-folder.''

===Security Constraints===
Files, in CSV format, of historical and expected Security Constraints are provided through both the EM6 and COMIT free to air services. The following table summarises the files that can be processed by ''EMarketOffer:''

{|class="wikitable"
! Source !! Data Content !! EMO Location !! EMO File Name !! Comment
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Reratings
|
:branch_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (circuit re-ratings) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Equations
|
:constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (equations) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Constraints
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:COMIT
|
:Security Constraints
|
:Input\Grid\COMIT
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:Ellsoft
|
:Security Constraints
|
:Input\Grid\Ellsoft
|
:ellsoft_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains an Ellsoft generated file containing all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|}

'''Warning!''' Ellsoft provides Constraint files to cover the historical period from the beginning of the market to a date subsequent to the mass circuit renaming exercise conducted by Transpower on the 8th December 2007. The file has been constructed using historical TPIX and recent EM6 data files and for the period 1 Jan 2007 to 31 May 2008 has used the maximum capacities from arcflow data to construct realistic branch re-ratings. This file is intended to overcome inconsistencies in constraint source data. It is provided without guarantee of accuracy; you are advised to independently confirm any data found in this file.

===How ''EMarketOffer'' processes Transmission System data===
When constructing its internal Transmission System database, ''EMarketOffer'' first analyses the base grid and creates a period-by-period 'map' of the system which includes the tracking of all changes in rating, electrical properties and overall configuration. Secondly the Branch re-ratings are applied 'over the top' of the initial map. Finally the Group Constraints are examined and a parallel map of these Constraints is created.

Two rules are observed when loading Constraint data from file:

#Data is loaded from folders in the following preference order:
##Input\Grid\Ellsoft
##Input\Grid\EM6\Constraints
##Input\Grid\COMIT
##Input\Grid\EM6\Equations and Input\Grid\EM6\Reratings
#Where files in different folders contain data for overlapping time periods, data will be taken from the file with the latest data, and from the first date in that file (see Figure 1 below)

[[File:Transmission system data.PNG|500px|thumb|centre|Figure 1 - Constraints File Handling Example]]

----
TIP: It is recommended that you keep your constraint data files from the various sources in a separate library and transfer only those files you wish to use to the ''EMarketOffer'' input data folders before launching the program.
----

In most cases ''EMarketOffer'' will be configured to use either EM6/COMIT files or online database connections (combination of there two options are supported). If you are using the EM6/COMIT file option, changing to a database link will require considerable configuration (see '''''Connecting to a networked database'''''), however if you are using the database option simply place EM6/COMIT files in the appropriate input date folders to override the database connection for the associated type of data.

== Static (User-Defined) Data ==
''EMarketOffer'' uses static data to calculate or organise the dynamic data produced by the Run (see the Circuit Item example below). Static data tends to remain unchanged between the Trading Periods contained in a Run,

{|class="wikitable"
! Item !! Description !! Static Data !! Dynamic Data
|-
! Circuit
|
:A branch (Circuit) conducting power on the NZ Transmission System
|
*Name
*From Node
*To Node
*Contributes to HVDC (Yes/No)
*Resistance
*Reactance
*Capacity (Summer/Winter)
|
*Powerflow
*Price Separation
*Losses
*Loading
|}

Modelled Items are defined by static data. Some Items are global, i.e. common for all Definitions, and their associated static data must be pre-defined (e.g. '''''Companies''''' and '''''Regions''''') Most Modelled Items, and their static data, are created for each new Definition (e.g. Generators or Nodes).

----
NOTE: To avoid time-consuming effort re-creating data in Definitions, Modelled Items, with their static data descriptions, can be imported into a Definition using ''EMarketOffer's'', '''''data loading functions''''', once loaded these Items can be modified, added to or removed with the Definition editor.
----

Functionality to maintain global static data can be accessed through the '''''[[EMO:The Edit Menu|Edit Menu]]''''' in the Main Window.

Static data in Definitions can be accessed through the '''''[[EMO:The Forecast Definition Window|Forecast Definition Window]]''''' or the '''''[[EMO:The Powerflow Case Definition Window|Powerflow Case Definition Window]]'''''.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Data Management|Back]]

Special:Badtitle/NS106:Input Data

2013-02-08T01:23:43Z

Sisep: /* Demand Data */

{{#hidens:}}
The input data types processed by ''EMarketOffer:''
*'''''Generation Data'''''
*'''''Price data
*'''''Offers data
*'''''Demand data
*'''''Transmission data
*'''''Static (user-defined) data'''''

==Generation Data==
Generation data is used:
*As information-only in the'''''Units Information Grid''''' for the '''''Market Data Source'''''
*To calculate Powerflows for the Market Data Source
*To calculate Powerflows for the Powerflow Case Data Source

=== Loading Generation Data ===
Historical Generation data can be loaded from an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]'''''). Historical or user-created Generation data can be loaded into '''''Powerflow Case Definitions''''' from file as a '''''generation profile'''''.

==Price Data==
Historical Price data is used as information-only in the '''''Units Information Grid''''' for the '''''Market Data Source'''''

=== Loading Historical Price Data ===
Historical Price data can only be loaded into ''EMarketOffer'' through direct connection to an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]''''').

== Offers Data ==
Historical Offers data is used:
*As information-only in the '''''[[EMO:Units Information Grid|Units Information Grid]]''''' for the '''''Market Data Source'''''
*As input into the seven-day '''''Offers Profile''''' in '''''forecast Definitions'''''
*As input into '''''Scenario Data Sources''''' based on historical Trading Periods (when available)

----
NOTE: ''EMarketOffer'' does NOT use Offers to calculate Powerflows for historical Trading Periods.
----

===Loading Historical Offers Data===
Historical Offers can be loaded either through:
*'''''direct connection to an online database'''''
*from pre-formatted '''''Offers files'''''
*Offers profile files (Forecast Definitions only)

Pre-formatted Offers files must use the same formatting used by the COMIT service for providing Offers data and be named in the following manner:

:'''Offersyyyymmdd.csv

Where:
:''''Offers'''' is the text 'Offers'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

You can download Offers files directly from COMIT and use them without further editing (once renamed).

----
NOTE: Date format is important - the date in the file name is used to allocate offers data to specific week days. One Offers files cannot contain data for periods from two different calendar days.
----

Offers files are processed row by row without regard to overall integrity. This allows you to edit the raw Offers files prior to loading to:
*Alter Offer Prices and Quantities
*Restrict the number of offered periods to less than a full day
*Restrict the number of offering Generation Units to less than those actually offering on the day

The formatting for the Offers files can be found in '''''[[EMO:Offers File Format|Offers File Format]].'''''

Offers profile files can also be used. An Offers profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition. The format of this file can be found in '''''[[EMO:Offers Profile File Format|Offers Profile File Format]].'''''

===Using Offers with Forecast Definitions===
''EMarketOffer's'' Forecast Definitions are based on a weekly Offers Profile, organised by day of week (as opposed to date). Offers for no more than seven consecutive days can be loaded into a Forecast Definition.

The following points should be noted:
*Where a '''''Forecast Effective Date Range''''' extends beyond seven days, Offers from the loaded profile will be re-used (by corresponding day of the week) from day eight onward
*Where Offers for fewer than seven days are loaded into a Forecast Definition , any periods in the Effective Date Range not covered by the loaded Offers will fail to dispatch
*New Offers can be loaded over the top of existing Offers. examples of how this might be useful include:
**producing modified historical offers by loading a week profile from the database and then loading an edited and/or partial Offers file containing different data
**Modifying a single generator's offering behaviour by loading an edited file containing Offers for only the Generation Unit and specific Trading Periods of interest leaving all previously loaded Offers (for this or any other generator) unchanged.
*Offers files are loaded consecutively by date from earliest to latest. Loading more than seven-days of Offers data from file will cause over-writing of the weekly Offer profile. for example if ten days of offers are loaded from file the first three days will be lost.

== Demand Data ==
Historical Demand data is used:
*As input into calculating the Powerflows in the '''''[[EMO:Data_Sources#Market_Mode|Market Data Source]]'''''
*As input into the seven-day Demand profile for '''''Forecast Definitions''''' and '''''Powerflow Cases'''''

===Which Demand data to use with ''EMarketOffer?''===
For best results ''EMarketOffer's'' database connection should be configured to use reconciled SPD Daily Demand data (available to subscribed COMIT users) as this most closely matches the historical Offers used in final dispatch. However, ''EMarketOffer'' can be configured to work with SCADA or other Demand data. Where SCADA data is used the net final Load at each Market Node should also be used.

===Loading Historical Demand Data===
Historical Demand can be loaded into ''EMarketOffer'' either through:
*'''''direct connection to an online database''''',
*from pre-formatted '''''Demand files'''''
*from demand profile files

===Pre-formatted files===
Pre-formatted Demand files must comply with ''EMarketOffer's'' Demand file format (see '''''[[EMO:Demand File Format|Demand File Format]]''''') and named in the following manner:

:'''Loadyyyymmdd.csv'''

Where:
:''''Load'''' is the text 'Load'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

----
NOTE: The date format is important. ''EMarketOffer'' uses the date in the file name to allocate the Demand data to specific days. One demand file cannot contain data for periods from more than one calendar day.
----

''EMarketOffer'' processes the Demand files row by row and does not check overall integrity. This allows you to edit the raw files prior to loading into ''EMarketOffer'' to:
*Alter Demand Quantities
*Restrict the number of Demand periods to less than a full day
*Restrict the number of Demand Nodes to less than those actually drawing down power on the day

The formatting for the Demand files can be found in '''''[[EMO:Demand File Format|Demand File Formats]]'''''.

===Demand Profile Files===
A Demand profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition or Powerflow Case. The format of this file can be found in '''''[[EMO:Demand Profile File Format|Demand Profile File Formats]]'''''.

===Using Demand with Forecast Definitions and Powerflow Cases===
''EMarketOffer's'' Forecast and Powerflow Case Definitions are based on a weekly Demand profile, by day of week. Demand figures for no more than seven consecutive days can be loaded into a Definition.

The following points should be noted:
*Where a Forecast or Powerflow Case '''''Effective Date Range''''' extends beyond seven days ''EMarketOffer'' will re-use the loaded Demand by day of the week from the profile for all weeks in the Effective Date Range.
*Where Demand for fewer than seven days is loaded into a Definition, any periods in the Effective Date Range not covered by the loaded Demand will fail to dispatch
*New Demand data can be loaded over the top of existing data. For example it is possible to load a full week profile from the database and then load a partial Demand file that has been edited to contain Demand for a single Node for a specific number of periods on a specific day. In this case only those periods for the selected Node will be overwritten with new data, all other Demand figures will remain unchanged.
*Demand files are loaded consecutively by date from earliest to latest. '''''Loading more than seven days of data from file''''' will cause over-writing of the weekly Demand profile. For example if ten days of Demand figures are loaded from file the first three days will be lost.
*When '''''loading Demand data from database''''' it is possible to specify a date range covering several weeks of Demand data. In this case an averaged Demand profile is created by day-of-the-week and Trading Period. It is not possible to create averaged Demand profiles by loading from file.

== Transmission System Data ==
Transmission System data supplied by the System Operator through the EM6 and COMIT information services is used to:
*Calculate Powerflows for the '''''Market Data Source'''''
*Define the Transmission System configuration in Forecast and '''''Powerflow Case Definitions'''''

''EMarketOffer'' requires the following Transmission System data:
*Network configuration (default values for base grid)
*Circuit electrical properties
*Circuit Re-ratings
*Group Constraint descriptions

By preference ''EMarketOffer'' automatically loads Transmission System data from any files found in the '''''EM6/COMIT sub-folders''''' on start-up (unless the '''Use Grid File Inputs box''' in the '''''Source Data Administration Window''''' is checked). A full 'map' of the historical and expected network is created, allowing the exact configuration of the Transmission System to be determined for any Trading Period covered by the file data. Where Transmission System data must be obtained from an online database, the 'map' is built up period by period in response to requests made through the Presentation Interface.

===Base Transmission System===
The Branch_Data file (obtainable as a downloaded from the EM6 service) defines the properties of all Circuits (including Summer and Winter ratings) and how they are connected. Both historical and expected changes to these properties are stored in the file. Files downloaded from EM6 should be placed in the ''Grid\EM6\BaseGrid'' sub-folder named according the the following convention (the same as used by EM6):

:'''Branch_Data_ddmmyyyy.csv'''

Where:
*''''Branch_Data'''' is the text 'Branch_Data'
*''''ddmmyyyy'''' is the date the file was downloaded from EM6

On start-up ''EMarketOffer'' uses the Branch_Data file with the latest date. Where no Branch_Data file exists, ''EMarketOffer'' attempts to access data through a database connection. Branch_Data files will always be used preferentially to source Transmission System data regardless of whether a current database connection for the data has been defined or not.

''EMarketOffer'' will fail to operate if no Transmission System data can be found from any source at startup.

----
NOTE: You will need to download new branch data files from EM6 from time to time. These should be named according to the above convention and placed in the ''EM6\BaseGrid'' sub-folder. It is not necessary to remove previous Branch_Data files as only data from the file with the latest date will be used.
----

'''WARNING!''' Files provided by the system operator no longer accurately reflect historical system changes prior to 2002. If you wish to model the pre-2002 system you should use Transpower branch data files that were created close to the time you are interested in analysing. ''This will require removing Branch_Date files with later create dates from the EM6\BaseGrid sub-folder.''

===Security Constraints===
Files, in CSV format, of historical and expected Security Constraints are provided through both the EM6 and COMIT free to air services. The following table summarises the files that can be processed by ''EMarketOffer:''

{|class="wikitable"
! Source !! Data Content !! EMO Location !! EMO File Name !! Comment
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Reratings
|
:branch_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (circuit re-ratings) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Equations
|
:constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (equations) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Constraints
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:COMIT
|
:Security Constraints
|
:Input\Grid\COMIT
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:Ellsoft
|
:Security Constraints
|
:Input\Grid\Ellsoft
|
:ellsoft_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains an Ellsoft generated file containing all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|}

'''Warning!''' Ellsoft provides Constraint files to cover the historical period from the beginning of the market to a date subsequent to the mass circuit renaming exercise conducted by Transpower on the 8th December 2007. The file has been constructed using historical TPIX and recent EM6 data files and for the period 1 Jan 2007 to 31 May 2008 has used the maximum capacities from arcflow data to construct realistic branch re-ratings. This file is intended to overcome inconsistencies in constraint source data. It is provided without guarantee of accuracy; you are advised to independently confirm any data found in this file.

===How ''EMarketOffer'' processes Transmission System data===
When constructing its internal Transmission System database, ''EMarketOffer'' first analyses the base grid and creates a period-by-period 'map' of the system which includes the tracking of all changes in rating, electrical properties and overall configuration. Secondly the Branch re-ratings are applied 'over the top' of the initial map. Finally the Group Constraints are examined and a parallel map of these Constraints is created.

Two rules are observed when loading Constraint data from file:

#Data is loaded from folders in the following preference order:
##Input\Grid\Ellsoft
##Input\Grid\EM6\Constraints
##Input\Grid\COMIT
##Input\Grid\EM6\Equations and Input\Grid\EM6\Reratings
#Where files in different folders contain data for overlapping time periods, data will be taken from the file with the latest data, and from the first date in that file (see Figure 1 below)

[[File:Transmission system data.PNG|500px|thumb|centre|Figure 1 - Constraints File Handling Example]]

----
TIP: It is recommended that you keep your constraint data files from the various sources in a separate library and transfer only those files you wish to use to the ''EMarketOffer'' input data folders before launching the program.
----

In most cases ''EMarketOffer'' will be configured to use either EM6/COMIT files or online database connections (combination of there two options are supported). If you are using the EM6/COMIT file option, changing to a database link will require considerable configuration (see '''''Connecting to a networked database'''''), however if you are using the database option simply place EM6/COMIT files in the appropriate input date folders to override the database connection for the associated type of data.

== Static (User-Defined) Data ==
''EMarketOffer'' uses static data to calculate or organise the dynamic data produced by the Run (see the Circuit Item example below). Static data tends to remain unchanged between the Trading Periods contained in a Run,

{|class="wikitable"
! Item !! Description !! Static Data !! Dynamic Data
|-
! Circuit
|
:A branch (Circuit) conducting power on the NZ Transmission System
|
*Name
*From Node
*To Node
*Contributes to HVDC (Yes/No)
*Resistance
*Reactance
*Capacity (Summer/Winter)
|
*Powerflow
*Price Separation
*Losses
*Loading
|}

Modelled Items are defined by static data. Some Items are global, i.e. common for all Definitions, and their associated static data must be pre-defined (e.g. '''''Companies''''' and '''''Regions''''') Most Modelled Items, and their static data, are created for each new Definition (e.g. Generators or Nodes).

----
NOTE: To avoid time-consuming effort re-creating data in Definitions, Modelled Items, with their static data descriptions, can be imported into a Definition using ''EMarketOffer's'', '''''data loading functions''''', once loaded these Items can be modified, added to or removed with the Definition editor.
----

Functionality to maintain global static data can be accessed through the '''''[[EMO:The Edit Menu|Edit Menu]]''''' in the Main Window.

Static data in Definitions can be accessed through the '''''[[EMO:The Forecast Definition Window|Forecast Definition Window]]''''' or the '''''[[EMO:The Powerflow Case Definition Window|Powerflow Case Definition Window]]'''''.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Data Management|Back]]

Special:Badtitle/NS106:Input Data

2013-02-08T01:22:19Z

Sisep: /* Offers Data */

{{#hidens:}}
The input data types processed by ''EMarketOffer:''
*'''''Generation Data'''''
*'''''Price data
*'''''Offers data
*'''''Demand data
*'''''Transmission data
*'''''Static (user-defined) data'''''

==Generation Data==
Generation data is used:
*As information-only in the'''''Units Information Grid''''' for the '''''Market Data Source'''''
*To calculate Powerflows for the Market Data Source
*To calculate Powerflows for the Powerflow Case Data Source

=== Loading Generation Data ===
Historical Generation data can be loaded from an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]'''''). Historical or user-created Generation data can be loaded into '''''Powerflow Case Definitions''''' from file as a '''''generation profile'''''.

==Price Data==
Historical Price data is used as information-only in the '''''Units Information Grid''''' for the '''''Market Data Source'''''

=== Loading Historical Price Data ===
Historical Price data can only be loaded into ''EMarketOffer'' through direct connection to an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]''''').

== Offers Data ==
Historical Offers data is used:
*As information-only in the '''''[[EMO:Units Information Grid|Units Information Grid]]''''' for the '''''Market Data Source'''''
*As input into the seven-day '''''Offers Profile''''' in '''''forecast Definitions'''''
*As input into '''''Scenario Data Sources''''' based on historical Trading Periods (when available)

----
NOTE: ''EMarketOffer'' does NOT use Offers to calculate Powerflows for historical Trading Periods.
----

===Loading Historical Offers Data===
Historical Offers can be loaded either through:
*'''''direct connection to an online database'''''
*from pre-formatted '''''Offers files'''''
*Offers profile files (Forecast Definitions only)

Pre-formatted Offers files must use the same formatting used by the COMIT service for providing Offers data and be named in the following manner:

:'''Offersyyyymmdd.csv

Where:
:''''Offers'''' is the text 'Offers'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

You can download Offers files directly from COMIT and use them without further editing (once renamed).

----
NOTE: Date format is important - the date in the file name is used to allocate offers data to specific week days. One Offers files cannot contain data for periods from two different calendar days.
----

Offers files are processed row by row without regard to overall integrity. This allows you to edit the raw Offers files prior to loading to:
*Alter Offer Prices and Quantities
*Restrict the number of offered periods to less than a full day
*Restrict the number of offering Generation Units to less than those actually offering on the day

The formatting for the Offers files can be found in '''''[[EMO:Offers File Format|Offers File Format]].'''''

Offers profile files can also be used. An Offers profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition. The format of this file can be found in '''''[[EMO:Offers Profile File Format|Offers Profile File Format]].'''''

===Using Offers with Forecast Definitions===
''EMarketOffer's'' Forecast Definitions are based on a weekly Offers Profile, organised by day of week (as opposed to date). Offers for no more than seven consecutive days can be loaded into a Forecast Definition.

The following points should be noted:
*Where a '''''Forecast Effective Date Range''''' extends beyond seven days, Offers from the loaded profile will be re-used (by corresponding day of the week) from day eight onward
*Where Offers for fewer than seven days are loaded into a Forecast Definition , any periods in the Effective Date Range not covered by the loaded Offers will fail to dispatch
*New Offers can be loaded over the top of existing Offers. examples of how this might be useful include:
**producing modified historical offers by loading a week profile from the database and then loading an edited and/or partial Offers file containing different data
**Modifying a single generator's offering behaviour by loading an edited file containing Offers for only the Generation Unit and specific Trading Periods of interest leaving all previously loaded Offers (for this or any other generator) unchanged.
*Offers files are loaded consecutively by date from earliest to latest. Loading more than seven-days of Offers data from file will cause over-writing of the weekly Offer profile. for example if ten days of offers are loaded from file the first three days will be lost.

== Demand Data ==
Historical Demand data is used:
*As input into calculating the Powerflows in the '''''Market Data Source'''''
*As input into the seven-day Demand profile for '''''Forecast Definitions''''' and '''''Powerflow Cases'''''

===Which Demand data to use with ''EMarketOffer?''===
For best results ''EMarketOffer's'' database connection should be configured to use reconciled SPD Daily Demand data (available to subscribed COMIT users) as this most closely matches the historical Offers used in final dispatch. However, ''EMarketOffer'' can be configured to work with SCADA or other Demand data. Where SCADA data is used the net final Load at each Market Node should also be used.

===Loading Historical Demand Data===
Historical Demand can be loaded into ''EMarketOffer'' either through:
*'''''direct connection to an online database''''',
*from pre-formatted '''''Demand files'''''
*from demand profile files

===Pre-formatted files===
Pre-formatted Demand files must comply with ''EMarketOffer's'' Demand file format (see '''''[[EMO:Demand File Format|Demand File Format]]''''') and named in the following manner:

:'''Loadyyyymmdd.csv'''

Where:
:''''Load'''' is the text 'Load'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

----
NOTE: The date format is important. ''EMarketOffer'' uses the date in the file name to allocate the Demand data to specific days. One demand file cannot contain data for periods from more than one calendar day.
----

''EMarketOffer'' processes the Demand files row by row and does not check overall integrity. This allows you to edit the raw files prior to loading into ''EMarketOffer'' to:
*Alter Demand Quantities
*Restrict the number of Demand periods to less than a full day
*Restrict the number of Demand Nodes to less than those actually drawing down power on the day

The formatting for the Demand files can be found in '''''[[EMO:Demand File Format|Demand File Formats]]'''''.

===Demand Profile Files===
A Demand profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition or Powerflow Case. The format of this file can be found in '''''[[EMO:Demand Profile File Format|Demand Profile File Formats]]'''''.

===Using Demand with Forecast Definitions and Powerflow Cases===
''EMarketOffer's'' Forecast and Powerflow Case Definitions are based on a weekly Demand profile, by day of week. Demand figures for no more than seven consecutive days can be loaded into a Definition.

The following points should be noted:
*Where a Forecast or Powerflow Case '''''Effective Date Range''''' extends beyond seven days ''EMarketOffer'' will re-use the loaded Demand by day of the week from the profile for all weeks in the Effective Date Range.
*Where Demand for fewer than seven days is loaded into a Definition, any periods in the Effective Date Range not covered by the loaded Demand will fail to dispatch
*New Demand data can be loaded over the top of existing data. For example it is possible to load a full week profile from the database and then load a partial Demand file that has been edited to contain Demand for a single Node for a specific number of periods on a specific day. In this case only those periods for the selected Node will be overwritten with new data, all other Demand figures will remain unchanged.
*Demand files are loaded consecutively by date from earliest to latest. '''''Loading more than seven days of data from file''''' will cause over-writing of the weekly Demand profile. For example if ten days of Demand figures are loaded from file the first three days will be lost.
*When '''''loading Demand data from database''''' it is possible to specify a date range covering several weeks of Demand data. In this case an averaged Demand profile is created by day-of-the-week and Trading Period. It is not possible to create averaged Demand profiles by loading from file.

== Transmission System Data ==
Transmission System data supplied by the System Operator through the EM6 and COMIT information services is used to:
*Calculate Powerflows for the '''''Market Data Source'''''
*Define the Transmission System configuration in Forecast and '''''Powerflow Case Definitions'''''

''EMarketOffer'' requires the following Transmission System data:
*Network configuration (default values for base grid)
*Circuit electrical properties
*Circuit Re-ratings
*Group Constraint descriptions

By preference ''EMarketOffer'' automatically loads Transmission System data from any files found in the '''''EM6/COMIT sub-folders''''' on start-up (unless the '''Use Grid File Inputs box''' in the '''''Source Data Administration Window''''' is checked). A full 'map' of the historical and expected network is created, allowing the exact configuration of the Transmission System to be determined for any Trading Period covered by the file data. Where Transmission System data must be obtained from an online database, the 'map' is built up period by period in response to requests made through the Presentation Interface.

===Base Transmission System===
The Branch_Data file (obtainable as a downloaded from the EM6 service) defines the properties of all Circuits (including Summer and Winter ratings) and how they are connected. Both historical and expected changes to these properties are stored in the file. Files downloaded from EM6 should be placed in the ''Grid\EM6\BaseGrid'' sub-folder named according the the following convention (the same as used by EM6):

:'''Branch_Data_ddmmyyyy.csv'''

Where:
*''''Branch_Data'''' is the text 'Branch_Data'
*''''ddmmyyyy'''' is the date the file was downloaded from EM6

On start-up ''EMarketOffer'' uses the Branch_Data file with the latest date. Where no Branch_Data file exists, ''EMarketOffer'' attempts to access data through a database connection. Branch_Data files will always be used preferentially to source Transmission System data regardless of whether a current database connection for the data has been defined or not.

''EMarketOffer'' will fail to operate if no Transmission System data can be found from any source at startup.

----
NOTE: You will need to download new branch data files from EM6 from time to time. These should be named according to the above convention and placed in the ''EM6\BaseGrid'' sub-folder. It is not necessary to remove previous Branch_Data files as only data from the file with the latest date will be used.
----

'''WARNING!''' Files provided by the system operator no longer accurately reflect historical system changes prior to 2002. If you wish to model the pre-2002 system you should use Transpower branch data files that were created close to the time you are interested in analysing. ''This will require removing Branch_Date files with later create dates from the EM6\BaseGrid sub-folder.''

===Security Constraints===
Files, in CSV format, of historical and expected Security Constraints are provided through both the EM6 and COMIT free to air services. The following table summarises the files that can be processed by ''EMarketOffer:''

{|class="wikitable"
! Source !! Data Content !! EMO Location !! EMO File Name !! Comment
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Reratings
|
:branch_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (circuit re-ratings) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Equations
|
:constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (equations) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Constraints
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:COMIT
|
:Security Constraints
|
:Input\Grid\COMIT
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:Ellsoft
|
:Security Constraints
|
:Input\Grid\Ellsoft
|
:ellsoft_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains an Ellsoft generated file containing all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|}

'''Warning!''' Ellsoft provides Constraint files to cover the historical period from the beginning of the market to a date subsequent to the mass circuit renaming exercise conducted by Transpower on the 8th December 2007. The file has been constructed using historical TPIX and recent EM6 data files and for the period 1 Jan 2007 to 31 May 2008 has used the maximum capacities from arcflow data to construct realistic branch re-ratings. This file is intended to overcome inconsistencies in constraint source data. It is provided without guarantee of accuracy; you are advised to independently confirm any data found in this file.

===How ''EMarketOffer'' processes Transmission System data===
When constructing its internal Transmission System database, ''EMarketOffer'' first analyses the base grid and creates a period-by-period 'map' of the system which includes the tracking of all changes in rating, electrical properties and overall configuration. Secondly the Branch re-ratings are applied 'over the top' of the initial map. Finally the Group Constraints are examined and a parallel map of these Constraints is created.

Two rules are observed when loading Constraint data from file:

#Data is loaded from folders in the following preference order:
##Input\Grid\Ellsoft
##Input\Grid\EM6\Constraints
##Input\Grid\COMIT
##Input\Grid\EM6\Equations and Input\Grid\EM6\Reratings
#Where files in different folders contain data for overlapping time periods, data will be taken from the file with the latest data, and from the first date in that file (see Figure 1 below)

[[File:Transmission system data.PNG|500px|thumb|centre|Figure 1 - Constraints File Handling Example]]

----
TIP: It is recommended that you keep your constraint data files from the various sources in a separate library and transfer only those files you wish to use to the ''EMarketOffer'' input data folders before launching the program.
----

In most cases ''EMarketOffer'' will be configured to use either EM6/COMIT files or online database connections (combination of there two options are supported). If you are using the EM6/COMIT file option, changing to a database link will require considerable configuration (see '''''Connecting to a networked database'''''), however if you are using the database option simply place EM6/COMIT files in the appropriate input date folders to override the database connection for the associated type of data.

== Static (User-Defined) Data ==
''EMarketOffer'' uses static data to calculate or organise the dynamic data produced by the Run (see the Circuit Item example below). Static data tends to remain unchanged between the Trading Periods contained in a Run,

{|class="wikitable"
! Item !! Description !! Static Data !! Dynamic Data
|-
! Circuit
|
:A branch (Circuit) conducting power on the NZ Transmission System
|
*Name
*From Node
*To Node
*Contributes to HVDC (Yes/No)
*Resistance
*Reactance
*Capacity (Summer/Winter)
|
*Powerflow
*Price Separation
*Losses
*Loading
|}

Modelled Items are defined by static data. Some Items are global, i.e. common for all Definitions, and their associated static data must be pre-defined (e.g. '''''Companies''''' and '''''Regions''''') Most Modelled Items, and their static data, are created for each new Definition (e.g. Generators or Nodes).

----
NOTE: To avoid time-consuming effort re-creating data in Definitions, Modelled Items, with their static data descriptions, can be imported into a Definition using ''EMarketOffer's'', '''''data loading functions''''', once loaded these Items can be modified, added to or removed with the Definition editor.
----

Functionality to maintain global static data can be accessed through the '''''[[EMO:The Edit Menu|Edit Menu]]''''' in the Main Window.

Static data in Definitions can be accessed through the '''''[[EMO:The Forecast Definition Window|Forecast Definition Window]]''''' or the '''''[[EMO:The Powerflow Case Definition Window|Powerflow Case Definition Window]]'''''.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Data Management|Back]]

Special:Badtitle/NS106:Input Data

2013-02-08T01:21:53Z

Sisep: /* Offers Data */

{{#hidens:}}
The input data types processed by ''EMarketOffer:''
*'''''Generation Data'''''
*'''''Price data
*'''''Offers data
*'''''Demand data
*'''''Transmission data
*'''''Static (user-defined) data'''''

==Generation Data==
Generation data is used:
*As information-only in the'''''Units Information Grid''''' for the '''''Market Data Source'''''
*To calculate Powerflows for the Market Data Source
*To calculate Powerflows for the Powerflow Case Data Source

=== Loading Generation Data ===
Historical Generation data can be loaded from an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]'''''). Historical or user-created Generation data can be loaded into '''''Powerflow Case Definitions''''' from file as a '''''generation profile'''''.

==Price Data==
Historical Price data is used as information-only in the '''''Units Information Grid''''' for the '''''Market Data Source'''''

=== Loading Historical Price Data ===
Historical Price data can only be loaded into ''EMarketOffer'' through direct connection to an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]''''').

== Offers Data ==
Historical Offers data is used:
*As information-only in the '''''[[Units Information Grid|Units Information Grid]]''''' for the '''''Market Data Source'''''
*As input into the seven-day '''''Offers Profile''''' in '''''forecast Definitions'''''
*As input into '''''Scenario Data Sources''''' based on historical Trading Periods (when available)

----
NOTE: ''EMarketOffer'' does NOT use Offers to calculate Powerflows for historical Trading Periods.
----

===Loading Historical Offers Data===
Historical Offers can be loaded either through:
*'''''direct connection to an online database'''''
*from pre-formatted '''''Offers files'''''
*Offers profile files (Forecast Definitions only)

Pre-formatted Offers files must use the same formatting used by the COMIT service for providing Offers data and be named in the following manner:

:'''Offersyyyymmdd.csv

Where:
:''''Offers'''' is the text 'Offers'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

You can download Offers files directly from COMIT and use them without further editing (once renamed).

----
NOTE: Date format is important - the date in the file name is used to allocate offers data to specific week days. One Offers files cannot contain data for periods from two different calendar days.
----

Offers files are processed row by row without regard to overall integrity. This allows you to edit the raw Offers files prior to loading to:
*Alter Offer Prices and Quantities
*Restrict the number of offered periods to less than a full day
*Restrict the number of offering Generation Units to less than those actually offering on the day

The formatting for the Offers files can be found in '''''[[EMO:Offers File Format|Offers File Format]].'''''

Offers profile files can also be used. An Offers profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition. The format of this file can be found in '''''[[EMO:Offers Profile File Format|Offers Profile File Format]].'''''

===Using Offers with Forecast Definitions===
''EMarketOffer's'' Forecast Definitions are based on a weekly Offers Profile, organised by day of week (as opposed to date). Offers for no more than seven consecutive days can be loaded into a Forecast Definition.

The following points should be noted:
*Where a '''''Forecast Effective Date Range''''' extends beyond seven days, Offers from the loaded profile will be re-used (by corresponding day of the week) from day eight onward
*Where Offers for fewer than seven days are loaded into a Forecast Definition , any periods in the Effective Date Range not covered by the loaded Offers will fail to dispatch
*New Offers can be loaded over the top of existing Offers. examples of how this might be useful include:
**producing modified historical offers by loading a week profile from the database and then loading an edited and/or partial Offers file containing different data
**Modifying a single generator's offering behaviour by loading an edited file containing Offers for only the Generation Unit and specific Trading Periods of interest leaving all previously loaded Offers (for this or any other generator) unchanged.
*Offers files are loaded consecutively by date from earliest to latest. Loading more than seven-days of Offers data from file will cause over-writing of the weekly Offer profile. for example if ten days of offers are loaded from file the first three days will be lost.

== Demand Data ==
Historical Demand data is used:
*As input into calculating the Powerflows in the '''''Market Data Source'''''
*As input into the seven-day Demand profile for '''''Forecast Definitions''''' and '''''Powerflow Cases'''''

===Which Demand data to use with ''EMarketOffer?''===
For best results ''EMarketOffer's'' database connection should be configured to use reconciled SPD Daily Demand data (available to subscribed COMIT users) as this most closely matches the historical Offers used in final dispatch. However, ''EMarketOffer'' can be configured to work with SCADA or other Demand data. Where SCADA data is used the net final Load at each Market Node should also be used.

===Loading Historical Demand Data===
Historical Demand can be loaded into ''EMarketOffer'' either through:
*'''''direct connection to an online database''''',
*from pre-formatted '''''Demand files'''''
*from demand profile files

===Pre-formatted files===
Pre-formatted Demand files must comply with ''EMarketOffer's'' Demand file format (see '''''[[EMO:Demand File Format|Demand File Format]]''''') and named in the following manner:

:'''Loadyyyymmdd.csv'''

Where:
:''''Load'''' is the text 'Load'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

----
NOTE: The date format is important. ''EMarketOffer'' uses the date in the file name to allocate the Demand data to specific days. One demand file cannot contain data for periods from more than one calendar day.
----

''EMarketOffer'' processes the Demand files row by row and does not check overall integrity. This allows you to edit the raw files prior to loading into ''EMarketOffer'' to:
*Alter Demand Quantities
*Restrict the number of Demand periods to less than a full day
*Restrict the number of Demand Nodes to less than those actually drawing down power on the day

The formatting for the Demand files can be found in '''''[[EMO:Demand File Format|Demand File Formats]]'''''.

===Demand Profile Files===
A Demand profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition or Powerflow Case. The format of this file can be found in '''''[[EMO:Demand Profile File Format|Demand Profile File Formats]]'''''.

===Using Demand with Forecast Definitions and Powerflow Cases===
''EMarketOffer's'' Forecast and Powerflow Case Definitions are based on a weekly Demand profile, by day of week. Demand figures for no more than seven consecutive days can be loaded into a Definition.

The following points should be noted:
*Where a Forecast or Powerflow Case '''''Effective Date Range''''' extends beyond seven days ''EMarketOffer'' will re-use the loaded Demand by day of the week from the profile for all weeks in the Effective Date Range.
*Where Demand for fewer than seven days is loaded into a Definition, any periods in the Effective Date Range not covered by the loaded Demand will fail to dispatch
*New Demand data can be loaded over the top of existing data. For example it is possible to load a full week profile from the database and then load a partial Demand file that has been edited to contain Demand for a single Node for a specific number of periods on a specific day. In this case only those periods for the selected Node will be overwritten with new data, all other Demand figures will remain unchanged.
*Demand files are loaded consecutively by date from earliest to latest. '''''Loading more than seven days of data from file''''' will cause over-writing of the weekly Demand profile. For example if ten days of Demand figures are loaded from file the first three days will be lost.
*When '''''loading Demand data from database''''' it is possible to specify a date range covering several weeks of Demand data. In this case an averaged Demand profile is created by day-of-the-week and Trading Period. It is not possible to create averaged Demand profiles by loading from file.

== Transmission System Data ==
Transmission System data supplied by the System Operator through the EM6 and COMIT information services is used to:
*Calculate Powerflows for the '''''Market Data Source'''''
*Define the Transmission System configuration in Forecast and '''''Powerflow Case Definitions'''''

''EMarketOffer'' requires the following Transmission System data:
*Network configuration (default values for base grid)
*Circuit electrical properties
*Circuit Re-ratings
*Group Constraint descriptions

By preference ''EMarketOffer'' automatically loads Transmission System data from any files found in the '''''EM6/COMIT sub-folders''''' on start-up (unless the '''Use Grid File Inputs box''' in the '''''Source Data Administration Window''''' is checked). A full 'map' of the historical and expected network is created, allowing the exact configuration of the Transmission System to be determined for any Trading Period covered by the file data. Where Transmission System data must be obtained from an online database, the 'map' is built up period by period in response to requests made through the Presentation Interface.

===Base Transmission System===
The Branch_Data file (obtainable as a downloaded from the EM6 service) defines the properties of all Circuits (including Summer and Winter ratings) and how they are connected. Both historical and expected changes to these properties are stored in the file. Files downloaded from EM6 should be placed in the ''Grid\EM6\BaseGrid'' sub-folder named according the the following convention (the same as used by EM6):

:'''Branch_Data_ddmmyyyy.csv'''

Where:
*''''Branch_Data'''' is the text 'Branch_Data'
*''''ddmmyyyy'''' is the date the file was downloaded from EM6

On start-up ''EMarketOffer'' uses the Branch_Data file with the latest date. Where no Branch_Data file exists, ''EMarketOffer'' attempts to access data through a database connection. Branch_Data files will always be used preferentially to source Transmission System data regardless of whether a current database connection for the data has been defined or not.

''EMarketOffer'' will fail to operate if no Transmission System data can be found from any source at startup.

----
NOTE: You will need to download new branch data files from EM6 from time to time. These should be named according to the above convention and placed in the ''EM6\BaseGrid'' sub-folder. It is not necessary to remove previous Branch_Data files as only data from the file with the latest date will be used.
----

'''WARNING!''' Files provided by the system operator no longer accurately reflect historical system changes prior to 2002. If you wish to model the pre-2002 system you should use Transpower branch data files that were created close to the time you are interested in analysing. ''This will require removing Branch_Date files with later create dates from the EM6\BaseGrid sub-folder.''

===Security Constraints===
Files, in CSV format, of historical and expected Security Constraints are provided through both the EM6 and COMIT free to air services. The following table summarises the files that can be processed by ''EMarketOffer:''

{|class="wikitable"
! Source !! Data Content !! EMO Location !! EMO File Name !! Comment
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Reratings
|
:branch_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (circuit re-ratings) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Equations
|
:constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (equations) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Constraints
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:COMIT
|
:Security Constraints
|
:Input\Grid\COMIT
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:Ellsoft
|
:Security Constraints
|
:Input\Grid\Ellsoft
|
:ellsoft_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains an Ellsoft generated file containing all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|}

'''Warning!''' Ellsoft provides Constraint files to cover the historical period from the beginning of the market to a date subsequent to the mass circuit renaming exercise conducted by Transpower on the 8th December 2007. The file has been constructed using historical TPIX and recent EM6 data files and for the period 1 Jan 2007 to 31 May 2008 has used the maximum capacities from arcflow data to construct realistic branch re-ratings. This file is intended to overcome inconsistencies in constraint source data. It is provided without guarantee of accuracy; you are advised to independently confirm any data found in this file.

===How ''EMarketOffer'' processes Transmission System data===
When constructing its internal Transmission System database, ''EMarketOffer'' first analyses the base grid and creates a period-by-period 'map' of the system which includes the tracking of all changes in rating, electrical properties and overall configuration. Secondly the Branch re-ratings are applied 'over the top' of the initial map. Finally the Group Constraints are examined and a parallel map of these Constraints is created.

Two rules are observed when loading Constraint data from file:

#Data is loaded from folders in the following preference order:
##Input\Grid\Ellsoft
##Input\Grid\EM6\Constraints
##Input\Grid\COMIT
##Input\Grid\EM6\Equations and Input\Grid\EM6\Reratings
#Where files in different folders contain data for overlapping time periods, data will be taken from the file with the latest data, and from the first date in that file (see Figure 1 below)

[[File:Transmission system data.PNG|500px|thumb|centre|Figure 1 - Constraints File Handling Example]]

----
TIP: It is recommended that you keep your constraint data files from the various sources in a separate library and transfer only those files you wish to use to the ''EMarketOffer'' input data folders before launching the program.
----

In most cases ''EMarketOffer'' will be configured to use either EM6/COMIT files or online database connections (combination of there two options are supported). If you are using the EM6/COMIT file option, changing to a database link will require considerable configuration (see '''''Connecting to a networked database'''''), however if you are using the database option simply place EM6/COMIT files in the appropriate input date folders to override the database connection for the associated type of data.

== Static (User-Defined) Data ==
''EMarketOffer'' uses static data to calculate or organise the dynamic data produced by the Run (see the Circuit Item example below). Static data tends to remain unchanged between the Trading Periods contained in a Run,

{|class="wikitable"
! Item !! Description !! Static Data !! Dynamic Data
|-
! Circuit
|
:A branch (Circuit) conducting power on the NZ Transmission System
|
*Name
*From Node
*To Node
*Contributes to HVDC (Yes/No)
*Resistance
*Reactance
*Capacity (Summer/Winter)
|
*Powerflow
*Price Separation
*Losses
*Loading
|}

Modelled Items are defined by static data. Some Items are global, i.e. common for all Definitions, and their associated static data must be pre-defined (e.g. '''''Companies''''' and '''''Regions''''') Most Modelled Items, and their static data, are created for each new Definition (e.g. Generators or Nodes).

----
NOTE: To avoid time-consuming effort re-creating data in Definitions, Modelled Items, with their static data descriptions, can be imported into a Definition using ''EMarketOffer's'', '''''data loading functions''''', once loaded these Items can be modified, added to or removed with the Definition editor.
----

Functionality to maintain global static data can be accessed through the '''''[[EMO:The Edit Menu|Edit Menu]]''''' in the Main Window.

Static data in Definitions can be accessed through the '''''[[EMO:The Forecast Definition Window|Forecast Definition Window]]''''' or the '''''[[EMO:The Powerflow Case Definition Window|Powerflow Case Definition Window]]'''''.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Data Management|Back]]

Special:Badtitle/NS106:Input Data

2013-02-08T01:18:31Z

Sisep: /* Offers Data */

{{#hidens:}}
The input data types processed by ''EMarketOffer:''
*'''''Generation Data'''''
*'''''Price data
*'''''Offers data
*'''''Demand data
*'''''Transmission data
*'''''Static (user-defined) data'''''

==Generation Data==
Generation data is used:
*As information-only in the'''''Units Information Grid''''' for the '''''Market Data Source'''''
*To calculate Powerflows for the Market Data Source
*To calculate Powerflows for the Powerflow Case Data Source

=== Loading Generation Data ===
Historical Generation data can be loaded from an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]'''''). Historical or user-created Generation data can be loaded into '''''Powerflow Case Definitions''''' from file as a '''''generation profile'''''.

==Price Data==
Historical Price data is used as information-only in the '''''Units Information Grid''''' for the '''''Market Data Source'''''

=== Loading Historical Price Data ===
Historical Price data can only be loaded into ''EMarketOffer'' through direct connection to an online database (see '''''[[EMO:Connecting_to_a_Database_Server|Connecting to a Networked Database]]''''').

== Offers Data ==
Historical Offers data is used:
*As information-only in the '''''Units Information Grid''''' for the '''''Market Data Source'''''
*As input into the seven-day '''''Offers Profile''''' in '''''forecast Definitions'''''
*As input into '''''Scenario Data Sources''''' based on historical Trading Periods (when available)

----
NOTE: ''EMarketOffer'' does NOT use Offers to calculate Powerflows for historical Trading Periods.
----

===Loading Historical Offers Data===
Historical Offers can be loaded either through:
*'''''direct connection to an online database'''''
*from pre-formatted '''''Offers files'''''
*Offers profile files (Forecast Definitions only)

Pre-formatted Offers files must use the same formatting used by the COMIT service for providing Offers data and be named in the following manner:

:'''Offersyyyymmdd.csv

Where:
:''''Offers'''' is the text 'Offers'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

You can download Offers files directly from COMIT and use them without further editing (once renamed).

----
NOTE: Date format is important - the date in the file name is used to allocate offers data to specific week days. One Offers files cannot contain data for periods from two different calendar days.
----

Offers files are processed row by row without regard to overall integrity. This allows you to edit the raw Offers files prior to loading to:
*Alter Offer Prices and Quantities
*Restrict the number of offered periods to less than a full day
*Restrict the number of offering Generation Units to less than those actually offering on the day

The formatting for the Offers files can be found in '''''[[EMO:Offers File Format|Offers File Format]].'''''

Offers profile files can also be used. An Offers profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition. The format of this file can be found in '''''[[EMO:Offers Profile File Format|Offers Profile File Format]].'''''

===Using Offers with Forecast Definitions===
''EMarketOffer's'' Forecast Definitions are based on a weekly Offers Profile, organised by day of week (as opposed to date). Offers for no more than seven consecutive days can be loaded into a Forecast Definition.

The following points should be noted:
*Where a '''''Forecast Effective Date Range''''' extends beyond seven days, Offers from the loaded profile will be re-used (by corresponding day of the week) from day eight onward
*Where Offers for fewer than seven days are loaded into a Forecast Definition , any periods in the Effective Date Range not covered by the loaded Offers will fail to dispatch
*New Offers can be loaded over the top of existing Offers. examples of how this might be useful include:
**producing modified historical offers by loading a week profile from the database and then loading an edited and/or partial Offers file containing different data
**Modifying a single generator's offering behaviour by loading an edited file containing Offers for only the Generation Unit and specific Trading Periods of interest leaving all previously loaded Offers (for this or any other generator) unchanged.
*Offers files are loaded consecutively by date from earliest to latest. Loading more than seven-days of Offers data from file will cause over-writing of the weekly Offer profile. for example if ten days of offers are loaded from file the first three days will be lost.

== Demand Data ==
Historical Demand data is used:
*As input into calculating the Powerflows in the '''''Market Data Source'''''
*As input into the seven-day Demand profile for '''''Forecast Definitions''''' and '''''Powerflow Cases'''''

===Which Demand data to use with ''EMarketOffer?''===
For best results ''EMarketOffer's'' database connection should be configured to use reconciled SPD Daily Demand data (available to subscribed COMIT users) as this most closely matches the historical Offers used in final dispatch. However, ''EMarketOffer'' can be configured to work with SCADA or other Demand data. Where SCADA data is used the net final Load at each Market Node should also be used.

===Loading Historical Demand Data===
Historical Demand can be loaded into ''EMarketOffer'' either through:
*'''''direct connection to an online database''''',
*from pre-formatted '''''Demand files'''''
*from demand profile files

===Pre-formatted files===
Pre-formatted Demand files must comply with ''EMarketOffer's'' Demand file format (see '''''[[EMO:Demand File Format|Demand File Format]]''''') and named in the following manner:

:'''Loadyyyymmdd.csv'''

Where:
:''''Load'''' is the text 'Load'
:''''yyyy'''' is the numerical 4 digit year (e.g. 2012)
:''''mm'''' is the numerical 2 digit month (e.g. 06)
:''''dd'''' is the numerical 2 digit day (e.g. 08)

----
NOTE: The date format is important. ''EMarketOffer'' uses the date in the file name to allocate the Demand data to specific days. One demand file cannot contain data for periods from more than one calendar day.
----

''EMarketOffer'' processes the Demand files row by row and does not check overall integrity. This allows you to edit the raw files prior to loading into ''EMarketOffer'' to:
*Alter Demand Quantities
*Restrict the number of Demand periods to less than a full day
*Restrict the number of Demand Nodes to less than those actually drawing down power on the day

The formatting for the Demand files can be found in '''''[[EMO:Demand File Format|Demand File Formats]]'''''.

===Demand Profile Files===
A Demand profile file uses a proprietary format designed to capture a complete seven-day profile suitable for importing into a Forecast Definition or Powerflow Case. The format of this file can be found in '''''[[EMO:Demand Profile File Format|Demand Profile File Formats]]'''''.

===Using Demand with Forecast Definitions and Powerflow Cases===
''EMarketOffer's'' Forecast and Powerflow Case Definitions are based on a weekly Demand profile, by day of week. Demand figures for no more than seven consecutive days can be loaded into a Definition.

The following points should be noted:
*Where a Forecast or Powerflow Case '''''Effective Date Range''''' extends beyond seven days ''EMarketOffer'' will re-use the loaded Demand by day of the week from the profile for all weeks in the Effective Date Range.
*Where Demand for fewer than seven days is loaded into a Definition, any periods in the Effective Date Range not covered by the loaded Demand will fail to dispatch
*New Demand data can be loaded over the top of existing data. For example it is possible to load a full week profile from the database and then load a partial Demand file that has been edited to contain Demand for a single Node for a specific number of periods on a specific day. In this case only those periods for the selected Node will be overwritten with new data, all other Demand figures will remain unchanged.
*Demand files are loaded consecutively by date from earliest to latest. '''''Loading more than seven days of data from file''''' will cause over-writing of the weekly Demand profile. For example if ten days of Demand figures are loaded from file the first three days will be lost.
*When '''''loading Demand data from database''''' it is possible to specify a date range covering several weeks of Demand data. In this case an averaged Demand profile is created by day-of-the-week and Trading Period. It is not possible to create averaged Demand profiles by loading from file.

== Transmission System Data ==
Transmission System data supplied by the System Operator through the EM6 and COMIT information services is used to:
*Calculate Powerflows for the '''''Market Data Source'''''
*Define the Transmission System configuration in Forecast and '''''Powerflow Case Definitions'''''

''EMarketOffer'' requires the following Transmission System data:
*Network configuration (default values for base grid)
*Circuit electrical properties
*Circuit Re-ratings
*Group Constraint descriptions

By preference ''EMarketOffer'' automatically loads Transmission System data from any files found in the '''''EM6/COMIT sub-folders''''' on start-up (unless the '''Use Grid File Inputs box''' in the '''''Source Data Administration Window''''' is checked). A full 'map' of the historical and expected network is created, allowing the exact configuration of the Transmission System to be determined for any Trading Period covered by the file data. Where Transmission System data must be obtained from an online database, the 'map' is built up period by period in response to requests made through the Presentation Interface.

===Base Transmission System===
The Branch_Data file (obtainable as a downloaded from the EM6 service) defines the properties of all Circuits (including Summer and Winter ratings) and how they are connected. Both historical and expected changes to these properties are stored in the file. Files downloaded from EM6 should be placed in the ''Grid\EM6\BaseGrid'' sub-folder named according the the following convention (the same as used by EM6):

:'''Branch_Data_ddmmyyyy.csv'''

Where:
*''''Branch_Data'''' is the text 'Branch_Data'
*''''ddmmyyyy'''' is the date the file was downloaded from EM6

On start-up ''EMarketOffer'' uses the Branch_Data file with the latest date. Where no Branch_Data file exists, ''EMarketOffer'' attempts to access data through a database connection. Branch_Data files will always be used preferentially to source Transmission System data regardless of whether a current database connection for the data has been defined or not.

''EMarketOffer'' will fail to operate if no Transmission System data can be found from any source at startup.

----
NOTE: You will need to download new branch data files from EM6 from time to time. These should be named according to the above convention and placed in the ''EM6\BaseGrid'' sub-folder. It is not necessary to remove previous Branch_Data files as only data from the file with the latest date will be used.
----

'''WARNING!''' Files provided by the system operator no longer accurately reflect historical system changes prior to 2002. If you wish to model the pre-2002 system you should use Transpower branch data files that were created close to the time you are interested in analysing. ''This will require removing Branch_Date files with later create dates from the EM6\BaseGrid sub-folder.''

===Security Constraints===
Files, in CSV format, of historical and expected Security Constraints are provided through both the EM6 and COMIT free to air services. The following table summarises the files that can be processed by ''EMarketOffer:''

{|class="wikitable"
! Source !! Data Content !! EMO Location !! EMO File Name !! Comment
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Reratings
|
:branch_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (circuit re-ratings) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Equations
|
:constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints (equations) in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:EM6
|
:Security Constraints
|
:Input\Grid\EM6\Constraints
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:COMIT
|
:Security Constraints
|
:Input\Grid\COMIT
|
:constraints_change_ddmmyyyy_ddmmyyyy.csv
|
:Contains all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|-
|
:Ellsoft
|
:Security Constraints
|
:Input\Grid\Ellsoft
|
:ellsoft_constraints_ddmmyyyy_ddmmyyyy.csv
|
:Contains an Ellsoft generated file containing all security constraints in effect on any Trading Period with the date range contained in the file name (naming as used by EM6).
|}

'''Warning!''' Ellsoft provides Constraint files to cover the historical period from the beginning of the market to a date subsequent to the mass circuit renaming exercise conducted by Transpower on the 8th December 2007. The file has been constructed using historical TPIX and recent EM6 data files and for the period 1 Jan 2007 to 31 May 2008 has used the maximum capacities from arcflow data to construct realistic branch re-ratings. This file is intended to overcome inconsistencies in constraint source data. It is provided without guarantee of accuracy; you are advised to independently confirm any data found in this file.

===How ''EMarketOffer'' processes Transmission System data===
When constructing its internal Transmission System database, ''EMarketOffer'' first analyses the base grid and creates a period-by-period 'map' of the system which includes the tracking of all changes in rating, electrical properties and overall configuration. Secondly the Branch re-ratings are applied 'over the top' of the initial map. Finally the Group Constraints are examined and a parallel map of these Constraints is created.

Two rules are observed when loading Constraint data from file:

#Data is loaded from folders in the following preference order:
##Input\Grid\Ellsoft
##Input\Grid\EM6\Constraints
##Input\Grid\COMIT
##Input\Grid\EM6\Equations and Input\Grid\EM6\Reratings
#Where files in different folders contain data for overlapping time periods, data will be taken from the file with the latest data, and from the first date in that file (see Figure 1 below)

[[File:Transmission system data.PNG|500px|thumb|centre|Figure 1 - Constraints File Handling Example]]

----
TIP: It is recommended that you keep your constraint data files from the various sources in a separate library and transfer only those files you wish to use to the ''EMarketOffer'' input data folders before launching the program.
----

In most cases ''EMarketOffer'' will be configured to use either EM6/COMIT files or online database connections (combination of there two options are supported). If you are using the EM6/COMIT file option, changing to a database link will require considerable configuration (see '''''Connecting to a networked database'''''), however if you are using the database option simply place EM6/COMIT files in the appropriate input date folders to override the database connection for the associated type of data.

== Static (User-Defined) Data ==
''EMarketOffer'' uses static data to calculate or organise the dynamic data produced by the Run (see the Circuit Item example below). Static data tends to remain unchanged between the Trading Periods contained in a Run,

{|class="wikitable"
! Item !! Description !! Static Data !! Dynamic Data
|-
! Circuit
|
:A branch (Circuit) conducting power on the NZ Transmission System
|
*Name
*From Node
*To Node
*Contributes to HVDC (Yes/No)
*Resistance
*Reactance
*Capacity (Summer/Winter)
|
*Powerflow
*Price Separation
*Losses
*Loading
|}

Modelled Items are defined by static data. Some Items are global, i.e. common for all Definitions, and their associated static data must be pre-defined (e.g. '''''Companies''''' and '''''Regions''''') Most Modelled Items, and their static data, are created for each new Definition (e.g. Generators or Nodes).

----
NOTE: To avoid time-consuming effort re-creating data in Definitions, Modelled Items, with their static data descriptions, can be imported into a Definition using ''EMarketOffer's'', '''''data loading functions''''', once loaded these Items can be modified, added to or removed with the Definition editor.
----

Functionality to maintain global static data can be accessed through the '''''[[EMO:The Edit Menu|Edit Menu]]''''' in the Main Window.

Static data in Definitions can be accessed through the '''''[[EMO:The Forecast Definition Window|Forecast Definition Window]]''''' or the '''''[[EMO:The Powerflow Case Definition Window|Powerflow Case Definition Window]]'''''.

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Data Management|Back]]

Special:Badtitle/NS106:Data Storage Directory Structure

2013-02-08T01:16:30Z

Sisep: /* The Forecast Folder */

{{#hidens:}}
''EMarketOffer'' creates two separate directory structures during installation:
*'''''Program files'''''
*'''''Data files'''''

=== Program Files ===
During the installation process you can choose where ''EMarketOffer'' places its program files (default C:\Program Files\Ellsoft\EMO). This folder contains the ''EMarketOffer'' executable and the dlls (dynamic link libraries) required by the program.

'''Warning!''' You should not attempt to modify or move the location of any of these files after installation - doing so will cause the program to malfunction.

== Data Files ==
[[File:Data Files.PNG|400px|thumb|right|Figure 1 - The Directory Structure]]
During installation ''EMarketOffer'' creates a data directory structure containing the folders to the right:

You can select the location for the data folders during the installation process. For performance reasons it is recommended that the data folder be located on the same computer as the ''EMarketOffer'' executable.

===The Connection Folder===
The information required to connect ''EMarketOffer'' to external databases is stored here.

{|class="wikitable"
! File !! Purpose !! Comment
|-
|
:''<Configuration_Name>''.xml Where <Configuration_Name> is the name of a previously created '''''Database Connectivity Configuration'''''.
|
:Stores database connectivity data referred to during program launch
|
:Editing, deleting or moving this file may cause ''EMarketOffer'' to malfunction.
|}

===The Forecast Folder===
This folder stores the definitions of Forecasts and Powerflow Cases (.xml format), and the results of '''''Forecasts, Powerflow Cases''''' and '''''Scenarios''''' (binary form).

{|class="wikitable"
! Sub-Folder !! File !! Purpose !! Comment
|-
! rowspan="2" | Forecasts
|
:''<My_Forecast>.ibn''
|
:Contains the output in binary form of the last run of the 'My_Forecast' Forecast Definition
|
:One .ibn file will exist for each currently available Forecast Definition that has been run. 
----
:NOTE: Deleting a Forecast Definition using the '''''[[EMO:Control_Panel_Components#The_Data_Source_Manager|Main Window Data Source Manager]]''''' will delete the associated .ibn file.
|-
|
:''Forecasts.xml''
|
:Contains the latest Definitions for all current Forecasts.
|
:In editing or deleting this file you run the risk of losing all current Forecast Definitions and associated results data.
|-
! rowspan="2" | Powerflows
|
:''<My_Powerflow>.ibn''
|
:Contains the output in binary form of the last run of the 'My_Powerflow' Powerflow Case Definition
|
:One .ibn file will exist for each currently available Powerflow Case Definition that has been run. 
----
:NOTE: Deleting a Powerflow Case Definition using the '''''[[EMO:Control_Panel_Components#The_Data_Source_Manager|Main Window Data Source Manager]]''''' will delete the associated .ibn file.
|-
|
:''Powerflows.xml''
|
:Contains the latest Definitions for all current Powerflow Cases.
|
:In editing or deleting this file you run the risk of losing all current Powerflow Case Definitions and associated results data.
|-
! Scenarios
|
:''<My_Scenario>.scn''
|
:Contains the output in binary form of the 'My_Scenario' Scenario
|
:One .scn file will exist for each currently available '''''Scenario'''''
|-
! Library
|
:''<My_Component>.xml''
|
:Contains the definition for a saved '''''Powerflow Case''''' or '''''Forecast Component'''''.
|
|}

===The Inputs Folder===
The Inputs folder contains several sub-folders:
[[File:Inputs_Folder.PNG|400px|thumb|centre|Figure 2 - Inputs Folder Directory]]

== Offers ==
''EMarketOffer'' can load Offers from a database or from file. Offers files in the standard CSV format as provided by the COMIT system can be placed in the Offers folder, which can also be used as a library for user-defined/amended Offers data.

NOTE: This folder is created during installation for convenience; however ''EMarketOffer'' can read properly formatted files from any location.

== Grid ==
''EMarketOffer'' uses accurate Transmission System data for analysis of historical periods and as input into '''''Forecast Definitions''''' and '''''Powerflow Cases'''''. Transmission System configuration data downloaded from The EM6 and COMIT information services must be placed in respective sub-folders under this folder. ''EMarketOffer'' 'looks for' preferentially for system configuration data in these files during startup.

The '''''[[EMO:Input Data|Input Data]]''''' section contains details on each file type.

===The Licence Folder===
Your customised license file is stored in the License folder.

'''Warning!''' This file contains important information about your installation. It is encrypted and you should not attempt to edit this file in any way. The program will fail to startup if the licence file is missing or altered.

===The Maps Folder===
Data required to support the Mapping function is stored here.
[[File:Maps Folder.PNG|300px|thumb|right|Figure 3 - The Maps Data Directory]]

{|class="wikitable"
! Folder !! Purpose
|-
|
:''NZ''
|
:Specifies a collection of maps (typically NZ) and contains the configuration data files (.config) for the full set of maps. Maps from only one collection can be displayed at a time.
|-
|
:''Large Maps''
|
:Contains low level detail Map graphics in .emf format
|-
|
:''Medium Maps''
|
:Contains medium level detail Map graphics in .emf format
|-
|
:''Small''
|
:Contains high level detail Map graphics in .emf format
|}

===The Positions Folder===
Map position details for Transmission System elements are stored here.

{|class="wikitable"
! File !! Purpose !! Comment
|-
|
:''NodePositions.csv''
|
:Contains Map positions of Market Nodes.
|
:Updated automatically as Nodes are added and re-positioned through the '''''[[EMO:The Presentation Interface|Presentation Interface]]''''' - should not be edited manually.
|-
|
:''SubstationPositions.csv''
|
:Contains Map positions of Substations
|
|}

===The Reports Folder===
This is the default folder for storing '''''Report Output''''' files, which are typically in .csv format and named according to their data content and date range.

===The Settings Folder===
User settings files are stored in here.

{|class="wikitable"
! File !! Purpose !! Comment
|-
|
:''EmbeddedUnits.xml''
|
:Contains the latest list of '''''Embedded Generators'''''
|
:Updated automatically as Nodes are added and re-positioned through the '''''Presentation Interface''''' - should not be edited manually
|-
|
:''WatchLists.xml''
|
:Contains the latest '''''lists of Watched items'''''
|
|-
|
:''Regions.xml''
|
:Contains the latest '''''Region''''' set definitions
|
|-
|
:''Views.xml''
|
:Contains the latest '''''Main Map View''''' specifications
|
|-
|
:''Companies.xml''
|
:Contains the latest '''''Company''''' definition details
|
|-
|
:''ExcludedEquations.xml''
|
:Contains the latest list of '''''Equations Constraints''''' to be excluded when loading Transmission System data into a '''''Forecast Definition'''''
|
|}

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Data Management|Back]]

Special:Badtitle/NS106:Data Storage Directory Structure

2013-02-08T01:14:04Z

Sisep: /* The Forecast Folder */

{{#hidens:}}
''EMarketOffer'' creates two separate directory structures during installation:
*'''''Program files'''''
*'''''Data files'''''

=== Program Files ===
During the installation process you can choose where ''EMarketOffer'' places its program files (default C:\Program Files\Ellsoft\EMO). This folder contains the ''EMarketOffer'' executable and the dlls (dynamic link libraries) required by the program.

'''Warning!''' You should not attempt to modify or move the location of any of these files after installation - doing so will cause the program to malfunction.

== Data Files ==
[[File:Data Files.PNG|400px|thumb|right|Figure 1 - The Directory Structure]]
During installation ''EMarketOffer'' creates a data directory structure containing the folders to the right:

You can select the location for the data folders during the installation process. For performance reasons it is recommended that the data folder be located on the same computer as the ''EMarketOffer'' executable.

===The Connection Folder===
The information required to connect ''EMarketOffer'' to external databases is stored here.

{|class="wikitable"
! File !! Purpose !! Comment
|-
|
:''<Configuration_Name>''.xml Where <Configuration_Name> is the name of a previously created '''''Database Connectivity Configuration'''''.
|
:Stores database connectivity data referred to during program launch
|
:Editing, deleting or moving this file may cause ''EMarketOffer'' to malfunction.
|}

===The Forecast Folder===
This folder stores the definitions of Forecasts and Powerflow Cases (.xml format), and the results of '''''Forecasts, Powerflow Cases''''' and '''''Scenarios''''' (binary form).

{|class="wikitable"
! Sub-Folder !! File !! Purpose !! Comment
|-
! rowspan="2" | Forecasts
|
:''<My_Forecast>.ibn''
|
:Contains the output in binary form of the last run of the 'My_Forecast' Forecast Definition
|
:One .ibn file will exist for each currently available Forecast Definition that has been run. 
----
:NOTE: Deleting a Forecast Definition using the '''''Main Window Data Source Manager''''' will delete the associated .ibn file.
|-
|
:''Forecasts.xml''
|
:Contains the latest Definitions for all current Forecasts.
|
:In editing or deleting this file you run the risk of losing all current Forecast Definitions and associated results data.
|-
! rowspan="2" | Powerflows
|
:''<My_Powerflow>.ibn''
|
:Contains the output in binary form of the last run of the 'My_Powerflow' Powerflow Case Definition
|
:One .ibn file will exist for each currently available Powerflow Case Definition that has been run. 
----
:NOTE: Deleting a Powerflow Case Definition using the '''''Main Window Data Source Manager''''' will delete the associated .ibn file.
|-
|
:''Powerflows.xml''
|
:Contains the latest Definitions for all current Powerflow Cases.
|
:In editing or deleting this file you run the risk of losing all current Powerflow Case Definitions and associated results data.
|-
! Scenarios
|
:''<My_Scenario>.scn''
|
:Contains the output in binary form of the 'My_Scenario' Scenario
|
:One .scn file will exist for each currently available '''''Scenario'''''
|-
! Library
|
:''<My_Component>.xml''
|
:Contains the definition for a saved '''''Powerflow Case''''' or '''''Forecast Component'''''.
|
|}

===The Inputs Folder===
The Inputs folder contains several sub-folders:
[[File:Inputs_Folder.PNG|400px|thumb|centre|Figure 2 - Inputs Folder Directory]]

== Offers ==
''EMarketOffer'' can load Offers from a database or from file. Offers files in the standard CSV format as provided by the COMIT system can be placed in the Offers folder, which can also be used as a library for user-defined/amended Offers data.

NOTE: This folder is created during installation for convenience; however ''EMarketOffer'' can read properly formatted files from any location.

== Grid ==
''EMarketOffer'' uses accurate Transmission System data for analysis of historical periods and as input into '''''Forecast Definitions''''' and '''''Powerflow Cases'''''. Transmission System configuration data downloaded from The EM6 and COMIT information services must be placed in respective sub-folders under this folder. ''EMarketOffer'' 'looks for' preferentially for system configuration data in these files during startup.

The '''''[[EMO:Input Data|Input Data]]''''' section contains details on each file type.

===The Licence Folder===
Your customised license file is stored in the License folder.

'''Warning!''' This file contains important information about your installation. It is encrypted and you should not attempt to edit this file in any way. The program will fail to startup if the licence file is missing or altered.

===The Maps Folder===
Data required to support the Mapping function is stored here.
[[File:Maps Folder.PNG|300px|thumb|right|Figure 3 - The Maps Data Directory]]

{|class="wikitable"
! Folder !! Purpose
|-
|
:''NZ''
|
:Specifies a collection of maps (typically NZ) and contains the configuration data files (.config) for the full set of maps. Maps from only one collection can be displayed at a time.
|-
|
:''Large Maps''
|
:Contains low level detail Map graphics in .emf format
|-
|
:''Medium Maps''
|
:Contains medium level detail Map graphics in .emf format
|-
|
:''Small''
|
:Contains high level detail Map graphics in .emf format
|}

===The Positions Folder===
Map position details for Transmission System elements are stored here.

{|class="wikitable"
! File !! Purpose !! Comment
|-
|
:''NodePositions.csv''
|
:Contains Map positions of Market Nodes.
|
:Updated automatically as Nodes are added and re-positioned through the '''''[[EMO:The Presentation Interface|Presentation Interface]]''''' - should not be edited manually.
|-
|
:''SubstationPositions.csv''
|
:Contains Map positions of Substations
|
|}

===The Reports Folder===
This is the default folder for storing '''''Report Output''''' files, which are typically in .csv format and named according to their data content and date range.

===The Settings Folder===
User settings files are stored in here.

{|class="wikitable"
! File !! Purpose !! Comment
|-
|
:''EmbeddedUnits.xml''
|
:Contains the latest list of '''''Embedded Generators'''''
|
:Updated automatically as Nodes are added and re-positioned through the '''''Presentation Interface''''' - should not be edited manually
|-
|
:''WatchLists.xml''
|
:Contains the latest '''''lists of Watched items'''''
|
|-
|
:''Regions.xml''
|
:Contains the latest '''''Region''''' set definitions
|
|-
|
:''Views.xml''
|
:Contains the latest '''''Main Map View''''' specifications
|
|-
|
:''Companies.xml''
|
:Contains the latest '''''Company''''' definition details
|
|-
|
:''ExcludedEquations.xml''
|
:Contains the latest list of '''''Equations Constraints''''' to be excluded when loading Transmission System data into a '''''Forecast Definition'''''
|
|}

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:Data Management|Back]]

Special:Badtitle/NS106:System Architecture

2013-02-08T01:08:03Z

Sisep: /* Logical Architecture */

{{#hidens:}}
===Technical Architecture===
''EMarketOffer'' is provided as a single executable program developed in C# using Microsoft's .net environment. It is designed to operate as a desktop application and can be configured to operate as a client of networked online databases. To support mobile users the system can also operate when disconnected from the corporate network using data uploaded from corporate databases or appropriately formatted files. Offline data must be stored in the local machine in relational database schema implemented in either Microsoft Access or SQL server.

'''''[[EMO:The Forecast Definition Window|Forecast Definitions]]''''' and forecast results are stored in binary or XML file formats and can be exported for use in applications such as Excel and other ''EMarketOffer'' implementations. Forecast results files can be stored either on the client PC's local drive or on a mapped network drive. Energy Link recommends Forecast output is stored on the local drive by default to minimise the effects of network latency on Forecast Run speed. Forecast files can subsequently exported to networked locations either in native XML format or translated into csv format.

====Logical Architecture====
The logical architecture of ''EMarketOffer'' is described in the table below.

{|class="wikitable"
! Component !! Purpose
|-
|
:''Online database(s)''
|
:''EMarketOffer'' can access networked online databases containing market and transmission system data
|-
|
:''EM6 Data Files''
|
:Transmission System data provided in standard format by the System Operator can be uploaded into ''EMarketOffer''
|-
|
:''COMIT Data Files''
|
:Offers and Demand data provided in standard format by the Market operator can be uploaded into ''EMarketOffer''
|-
|
:''User Data Files''
|
:You can create your own Transmission System, Demand and Offers data files to upload into ''EMarketoffer''.
----
:'''TIP:''' To avoid the time consuming task of creating of large, customised time-series data files from scratch, use ''EMarketOffer'' to create output files of Offer and Demand data from database downloads, and store them in the Library. These can be edited in Excel as required and uploaded into new Forecast Definitions.
|-
|
:''Presentation Interface''
|
:The collection of information presentation features that allow you to visualise the '''''Current Data Set'''''. These include the '''''[[EMO:The Map|Map]]''''', '''''[[EMO:Charts|Charts]]''''', '''''[[EMO:The_Display_Area_Information_Grids#Information_Windows|Information Windows]]''''', '''''[[EMO:The_Edit_Menu#Watch_Window|Watch Windows]]''''' and '''''[[EMO:The_Edit_Menu#The_Reports_and_Graphs_Option|Reports]]'''''.
|-
|
:''Data Source Manager''
|
:Manages the run-time data displayed through the Presentation Interface
|-
|
:''Forecaster''
|
:Forecasts a range of Market-related variables for a specific period of time, based on a user-defined Definition which can be constructed using a mix of real (historical) data and user-created data. A special forecasting feature (Scenario mode) allows immediate, multiple re-forecasts for a selected single period using interactively modified Scenario details.
|-
|
:''Powerflow Engine''
|
:The Powerflow engine calculates the powerflows for a specified network using given nodal injection and off-take. The engine provides essential support for historical (market) data analysis and powerflow case analysis.
|-
|
:''Reports and Export Files''
|
:Reports and Export Files are produced by ''EMarketOffer's'' reporting capabilities.
|}

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:System Overview|Back]]

Special:Badtitle/NS106:System Architecture

2013-02-08T01:05:47Z

Sisep: /* Logical Architecture */

{{#hidens:}}
===Technical Architecture===
''EMarketOffer'' is provided as a single executable program developed in C# using Microsoft's .net environment. It is designed to operate as a desktop application and can be configured to operate as a client of networked online databases. To support mobile users the system can also operate when disconnected from the corporate network using data uploaded from corporate databases or appropriately formatted files. Offline data must be stored in the local machine in relational database schema implemented in either Microsoft Access or SQL server.

'''''[[EMO:The Forecast Definition Window|Forecast Definitions]]''''' and forecast results are stored in binary or XML file formats and can be exported for use in applications such as Excel and other ''EMarketOffer'' implementations. Forecast results files can be stored either on the client PC's local drive or on a mapped network drive. Energy Link recommends Forecast output is stored on the local drive by default to minimise the effects of network latency on Forecast Run speed. Forecast files can subsequently exported to networked locations either in native XML format or translated into csv format.

====Logical Architecture====
The logical architecture of ''EMarketOffer'' is described in the table below.

{|class="wikitable"
! Component !! Purpose
|-
|
:''Online database(s)''
|
:''EMarketOffer'' can access networked online databases containing market and transmission system data
|-
|
:''EM6 Data Files''
|
:Transmission System data provided in standard format by the System Operator can be uploaded into ''EMarketOffer''
|-
|
:''COMIT Data Files''
|
:Offers and Demand data provided in standard format by the Market operator can be uploaded into ''EMarketOffer''
|-
|
:''User Data Files''
|
:You can create your own Transmission System, Demand and Offers data files to upload into ''EMarketoffer''.
----
:'''TIP:''' To avoid the time consuming task of creating of large, customised time-series data files from scratch, use ''EMarketOffer'' to create output files of Offer and Demand data from database downloads, and store them in the Library. These can be edited in Excel as required and uploaded into new Forecast Definitions.
|-
|
:''Presentation Interface''
|
:The collection of information presentation features that allow you to visualise the '''''Current Data Set'''''. These include the '''''[[EMO:The Map|Map]]''''', '''''[[EMO:Charts|Charts]]''''', '''''Information Windows, Watch Windows''''' and '''''[[EMO:Reports|Reports]]'''''.
|-
|
:''Data Source Manager''
|
:Manages the run-time data displayed through the Presentation Interface
|-
|
:''Forecaster''
|
:Forecasts a range of Market-related variables for a specific period of time, based on a user-defined Definition which can be constructed using a mix of real (historical) data and user-created data. A special forecasting feature (Scenario mode) allows immediate, multiple re-forecasts for a selected single period using interactively modified Scenario details.
|-
|
:''Powerflow Engine''
|
:The Powerflow engine calculates the powerflows for a specified network using given nodal injection and off-take. The engine provides essential support for historical (market) data analysis and powerflow case analysis.
|-
|
:''Reports and Export Files''
|
:Reports and Export Files are produced by ''EMarketOffer's'' reporting capabilities.
|}

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:System Overview|Back]]

Special:Badtitle/NS106:System Architecture

2013-02-08T01:04:46Z

Sisep: /* Technical Architecture */

{{#hidens:}}
===Technical Architecture===
''EMarketOffer'' is provided as a single executable program developed in C# using Microsoft's .net environment. It is designed to operate as a desktop application and can be configured to operate as a client of networked online databases. To support mobile users the system can also operate when disconnected from the corporate network using data uploaded from corporate databases or appropriately formatted files. Offline data must be stored in the local machine in relational database schema implemented in either Microsoft Access or SQL server.

'''''[[EMO:The Forecast Definition Window|Forecast Definitions]]''''' and forecast results are stored in binary or XML file formats and can be exported for use in applications such as Excel and other ''EMarketOffer'' implementations. Forecast results files can be stored either on the client PC's local drive or on a mapped network drive. Energy Link recommends Forecast output is stored on the local drive by default to minimise the effects of network latency on Forecast Run speed. Forecast files can subsequently exported to networked locations either in native XML format or translated into csv format.

====Logical Architecture====
The logical architecture of ''EMarketOffer'' is described in the table below.

{|class="wikitable"
! Component !! Purpose
|-
|
:''Online database(s)''
|
:''EMarketOffer'' can access networked online databases containing market and transmission system data
|-
|
:''EM6 Data Files''
|
:Transmission System data provided in standard format by the System Operator can be uploaded into ''EMarketOffer''
|-
|
:''COMIT Data Files''
|
:Offers and Demand data provided in standard format by the Market operator can be uploaded into ''EMarketOffer''
|-
|
:''User Data Files''
|
:You can create your own Transmission System, Demand and Offers data files to upload into ''EMarketoffer''.
----
:'''TIP:''' To avoid the time consuming task of creating of large, customised time-series data files from scratch, use ''EMarketOffer'' to create output files of Offer and Demand data from database downloads, and store them in the Library. These can be edited in Excel as required and uploaded into new Forecast Definitions.
|-
|
:''Presentation Interface''
|
:The collection of information presentation features that allow you to visualise the '''''Current Data Set'''''. These include the '''''Map''''', '''''Charts''''', '''''Information Windows, Watch Windows''''' and '''''Reports'''''.
|-
|
:''Data Source Manager''
|
:Manages the run-time data displayed through the Presentation Interface
|-
|
:''Forecaster''
|
:Forecasts a range of Market-related variables for a specific period of time, based on a user-defined Definition which can be constructed using a mix of real (historical) data and user-created data. A special forecasting feature (Scenario mode) allows immediate, multiple re-forecasts for a selected single period using interactively modified Scenario details.
|-
|
:''Powerflow Engine''
|
:The Powerflow engine calculates the powerflows for a specified network using given nodal injection and off-take. The engine provides essential support for historical (market) data analysis and powerflow case analysis.
|-
|
:''Reports and Export Files''
|
:Reports and Export Files are produced by ''EMarketOffer's'' reporting capabilities.
|}

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:System Overview|Back]]

Special:Badtitle/NS106:Functions

2013-02-08T01:04:03Z

Sisep:

{{#hidens:}}
''EMarketOffer'' has three main functions:
#To analyse and present key historical electricity Market data through a powerful set of information presentation tools in the '''''[[EMO:The Presentation Interface|Presentation Interface]]'''''
#To simulate the operation of the Market for situations of potential interest (generally termed Forecasting) and presenting the results through the '''''[[EMO:The Presentation Interface|Presentation Interface]]'''''
#To conduct Powerflow analysis for a defined network using given nodal demand and generation.

===Supporting Functions===
To achieve these main functions ''EMarketOffer'' must:

For Historical Periods
*Calculate Powerflows using Generation and Demand figures
*Interpret Transmission System data including dynamic changes such as:
**Circuit re-routing or retirement
**Circuit re-rating
**Outages
**Application of Equation Constraints

For Forecast Periods
*Replicate the operation of the SPD Market clearance engine for both the energy and reserves market, to a high degree of accuracy and at high speed
*Load large volume data sets directly from online databases or from file
*Allow easy manipulation of historical Market and Transmission System data to quickly create new '''''[[EMO:The Forecast Definition Window|Forecast Definitions]]'''''

===Presentation Interface Functions===
The '''''[[EMO:The Presentation Interface|Presentation Interface]]''''' provides the following analysis support functions:
*Multi-featured Map
*Flexible tabulation of data including separate floating Windows to monitor specific items of interest
*Forecast Summaries to quickly identify the key features of a Forecast Run
*Charting of Offer stacks, Demand Profiles and Supply Curves by user-definable periods
*General purpose charting of any output variable
*Single-period 'Scenario' mode allowing any Dispatch Factor to be amended and the effects seen instantly through an immediate re-dispatch
*Marginal Analysis mode allowing the sensitivity of dispatch to changes in any Dispatch Factor

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:System Overview|Back]]

Special:Badtitle/NS106:Data Sources

2013-02-08T00:56:14Z

Sisep:

The ''EMarketOffer'' '''''[[EMO:The Presentation Interface|Presentation Interface]] '''''provides four generic data presentation modes, referred to as data sources. These are:
*Market
*Forecast
*Powerflow Case
*Scenario

=== Market Mode ===
Market mode is available for any historical day for which ''EMarketOffer'' can access data. It is not available for future dates. For the Trading Period under review, Market-sourced data automatically includes the best available:
*Transmission System configuration including Branch Constraints
*Group Constraints
*Prices
*Load
*Generation
While in Market Mode:
*Powerflows are automatically calculated and displayed on the Map and in '''''[[EMO:The_Edit_Menu#Watch_Window|Watch Windows]]'''''
*Probable (not actual) constraints are highlighted on the Map and can be viewed in Watch Windows
Market Mode is passive: while viewing and reporting Market-sourced information you cannot alter any of the underlying data or the derived Powerflows. There is only one version of Market data for any given Trading Period, and therefore in Market Data Source Mode views are simply identified by date and Trading Period in the '''''[[EMO:Main Window Title Bar|Main Window Title Bar]].'''''

=== Forecast Mode ===
Forecast Mode allows you to view Forecast output using the same viewing and reporting features used in Market Mode. Once created, Forecast data is stored in the Forecast Library and like Market-sourced data, cannot be amended. However, by running many Forecast Definitions is it possible to create multiple, different, Forecasts for the same Trading Period. The '''''Effective Date Range''''' for a Forecast can start in future or the past. Forecasts are identified by the name you give them at definition time, which appears with the "current" forecast date and Trading Period in the Main Window title bar.

=== Powerflow Case ===
Powerflow Mode displays the results of a Powerflow Case analysis output through the Presentation Interface. Powerflow Cases are defined in a similar way to Forecast Definitions. However, while Forecasts use the SPD dispatch engine and require Offers data, Powerflow cases execute ''EMarketOffer's'' powerflow engine and use Generation data as input.

=== Scenario Mode ===
A scenario is a snapshot of a single Trading Period taken from either Market (historical) or Forecast data. Once saved a Scenario cannot be changed. However, in this Mode any factor influencing the dispatch for the Period can be temporarily altered and a re-dispatch of the period immediately invoked. The result of a re-dispatched Scenario can be saved as a new Scenario and recalled independently at a future date. While in Scenario mode the Main Window title bar carries the Scenario name, which by default contains the date and Trading Period from which the scenario was originally created.

----
NOTE: Scenarios should not be created from a Market Data Source on days for which no valid Offers exist

[[EMO:EMarketOffer User Manual|Home]] - [[EMO:The Presentation Interface|Back]]

Special:Badtitle/NS106:Connecting to a Database Server

2013-02-08T00:50:06Z

Sisep: /* Creating a Database Connectivity Configuration */

{{#hidens:}}
'''WARNING!''' Connecting to a local or networked database server is complex and can be time-consuming. If no people in your company have previous experience of configuring ''EMarketOffer'' it is strongly recommended that you consult your company's technical support and also the Energy Link helpdesk for advice on the best approach.
----

===Terminology===
''EMarketOffer's'' data design consists of 'things' included in the model, and their characteristics'. In this manual the 'things' are called '''Entities''' and their characteristics are called '''Traits'''. Traits may be static or dynamic.

For example the ''Circuit'' Entity can have the static Traits of ''Start Node'' and ''End Node'', and the dynamic Traits of ''Powerflow'' and ''Loss'' by Trading Period.

===Input Data Types===
''EMarketOffer'' can load the folowing types of '''''Input data''''' directly from external databases.
{|class="wikitable"
! Entity !! Type !! Trait !! Mandatory !! Defined by
|-
!Node
|
:Demand (Load)
|
:''Node.Load''
|
:No
|
:Date, TP, Node, Load
|-
!
|
:Price
|
:''Node.Price''
|
:No
|
:Date, TP, Node, Load
|-
!Unit
|
:Unit Node (Links Units to Nodes)
|
:''Unit.Node''
|
:Yes
|
:Unit, Node
|-
!
|
:Generator Unit (Links Units to Generators)
|
:''Unit.Generator''
|
:Yes
|
:Unit, Generator (Generators are always three alpha codes e.g. HLY = Huntly
|-
!
|
:Unit Generation
|
:''Unit.Generation''
|
:No
|
:Date, TP, Unit, Generation
|-
!
|
:Unit Offer Quantity
|
:''Unit.OfferQn (where n = band)''
|
:No
|
:Date, TP, Unit, OfferQn
|-
!
|
:Unit Offer Price
|
:''Unit.OfferPn (Where n = band)''
|
:No
|
:Date, TP, Unit, OfferPn
|-
!Circuit
|
:Circuit
|
:''Circuit.Name''
|
:Yes
|
:Date, TP, Name
|-
!
|
:Node From
|
:''Circuit.NodeFrom''
|
:Yes
|
:Date, TP, Circuit, NodeFrom
|-
!
|
:Node To
|
:''Circuit.NodeTo''
|
:Yes
|
:Date, TP, Circuit, NodeTo
|-
!
|
:Reactance
|
:''Circuit.Reactance''
|
:Yes
|
:Date, TP, Circuit, Reactance
|-
!
|
:Resistance
|
:''Circuit.Resistance''
|
:Yes
|
:Date, TP, Circuit, Resistance
|-
!
|
:Fixed Losses
|
:''Circuit.FixedLosses''
|
:Yes
|
:Date, TP, Circuit, FixedLosses
|-
!
|
:Capacity
|
:''Circuit.Capacity''
|
:Conditional
|
:Date, TP, Circuit, Capacity
|-
!
|
:Winter Capacity
|
:''Circuit.WinterCapacity''
|
:Conditional
|
:Date, TP, Circuit, WinterCapacity
|-
!
|
:Summer Capacity
|
:''Circuit.SummerCapacity''
|
:Conditional
|
:Date, TP, Circuit, SummerCapacity
|-
!Equation
|
:Formula
|
:''Equation.Formula''
|
:Yes
|
:Date, TP, Equation, Formula
|-
!
|
:Limit
|
:''Equation.Limit''
|
:Yes
|
:Date, TP, Equation, Limit
|-
!
|
:Type
|
:''Equation.EquationType''
|
:No
|
:Date, TP, Equation, EquationType
|-
!
|
:Season
|
:''Equation.Season''
|
:No
|
:Date, TP, Equation, Season
|-
!
|
:Active
|
:''Equation.Active''
|
:No
|
:Date, TP, Equation, Active
|}
----
Note:
#Mandatory Traits are the minimum required for ''EMarketOffer'' to operate
#Either a single Circuit capacity OR two seasonal Circuit Capacities are mandatorily required
----

===Input Data Structure===
''EMarketOffer'' requires input data to generate the '''''Market Data Source''''' for viewing historical market conditions and to create the '''''Forecast Definitions''''' used to generate detailed market '''''[[EMO:Forecasting|Forecasts]]'''''. Input data can be retrieved from external databases using standard SQL queries, which can be grouped together into '''''Database Connection Configurations.'''''

NOTE: Only one Database Connection Configuration can be in use at any one time. This "current" configuration tells ''EMarketOffer'' where to find the input data for the Market Data Source and for loading data into Forecast Definitions. The most recently used configuration is opened automatically as the default configuration on startup. The '''''Data Sources Option''''' in the '''''[[EMO:The_File_Menu|Main Window File Menu]]''''' can be used to make an alternative configuration current.

===Input Data Handling Mechanism===
A Configuration contains a collection of queries that may reference more than one online database. Each query can reference a single database using a pre-defined Connection containing the information required to interact with a database server; once defined a Connection can be used repeatedly by the queries in a Configuration.

The main component of a Connection is the Connection String which ''EMarketOffer'' uses to locate and connect to online database servers. The string varies in format depending on the target database server type and its specific access security provisions. It may contain:
*A data provider: this specifies the type of database server to be accessed (e.g. Oracle or MS SQL) and the specific access protocol to be used (e.g. OleDB or ODBC)
*Name of the database to be accessed (if required)
*The access parameters (typically User Name and Password)
*A Data Source Name (DSN) file containing the above information

See the section on '''''[[EMO:Securing Database Access|Securing Database Access]]''''' for more information on Connection Strings.

A separate query is used for each data item to be imported. Each query must be associated with a Connection and contain a standard SQL-compliant query string defining the specific data to be returned. Query strings should:
*Be tested first for syntax against your database
*Contain the return fields required by ''EMarketOffer''
*Contain a 'Where' clause defining a select date range using the variables $DateFrom$ and $DateTo$
*Allow aliasing of column names with attribute (trait) names used by ''EMarketOffer''
----
Notes:
#All queries must contain the name of the Trait on each returned row
#All queries (except for the Unit.Generator query) return a row for each Trading Period in the entered data range
----
'''WARNING!''' It is not possible to join data from two different databases within the same query, however ''EMarketOffer'' is designed to use a separate query for each input data item so this restriction should not constrain the program's operation. If, for any reason, you do wish to use a data item that is sourced from more than one database, you should first create a view/query in an external database that combines the required data, and then use ''EMarketOffer's'' to connect to this view.

===Creating a Database Connectivity Configuration===
The functions used in Configuration setup are fully described in the '''''[[EMO:Menu Bar|Menu Bar]]''''' segment under the '''''[[EMO:The Main Window|Main Window]]''''' section.

[[EMO:Getting Started|Back]] - [[EMO:EMarketOffer User Manual|Home]]

Special:Badtitle/NS106:Connecting to a Database Server

2013-02-08T00:49:38Z

Sisep: /* Input Data Handling Mechanism */

{{#hidens:}}
'''WARNING!''' Connecting to a local or networked database server is complex and can be time-consuming. If no people in your company have previous experience of configuring ''EMarketOffer'' it is strongly recommended that you consult your company's technical support and also the Energy Link helpdesk for advice on the best approach.
----

===Terminology===
''EMarketOffer's'' data design consists of 'things' included in the model, and their characteristics'. In this manual the 'things' are called '''Entities''' and their characteristics are called '''Traits'''. Traits may be static or dynamic.

For example the ''Circuit'' Entity can have the static Traits of ''Start Node'' and ''End Node'', and the dynamic Traits of ''Powerflow'' and ''Loss'' by Trading Period.

===Input Data Types===
''EMarketOffer'' can load the folowing types of '''''Input data''''' directly from external databases.
{|class="wikitable"
! Entity !! Type !! Trait !! Mandatory !! Defined by
|-
!Node
|
:Demand (Load)
|
:''Node.Load''
|
:No
|
:Date, TP, Node, Load
|-
!
|
:Price
|
:''Node.Price''
|
:No
|
:Date, TP, Node, Load
|-
!Unit
|
:Unit Node (Links Units to Nodes)
|
:''Unit.Node''
|
:Yes
|
:Unit, Node
|-
!
|
:Generator Unit (Links Units to Generators)
|
:''Unit.Generator''
|
:Yes
|
:Unit, Generator (Generators are always three alpha codes e.g. HLY = Huntly
|-
!
|
:Unit Generation
|
:''Unit.Generation''
|
:No
|
:Date, TP, Unit, Generation
|-
!
|
:Unit Offer Quantity
|
:''Unit.OfferQn (where n = band)''
|
:No
|
:Date, TP, Unit, OfferQn
|-
!
|
:Unit Offer Price
|
:''Unit.OfferPn (Where n = band)''
|
:No
|
:Date, TP, Unit, OfferPn
|-
!Circuit
|
:Circuit
|
:''Circuit.Name''
|
:Yes
|
:Date, TP, Name
|-
!
|
:Node From
|
:''Circuit.NodeFrom''
|
:Yes
|
:Date, TP, Circuit, NodeFrom
|-
!
|
:Node To
|
:''Circuit.NodeTo''
|
:Yes
|
:Date, TP, Circuit, NodeTo
|-
!
|
:Reactance
|
:''Circuit.Reactance''
|
:Yes
|
:Date, TP, Circuit, Reactance
|-
!
|
:Resistance
|
:''Circuit.Resistance''
|
:Yes
|
:Date, TP, Circuit, Resistance
|-
!
|
:Fixed Losses
|
:''Circuit.FixedLosses''
|
:Yes
|
:Date, TP, Circuit, FixedLosses
|-
!
|
:Capacity
|
:''Circuit.Capacity''
|
:Conditional
|
:Date, TP, Circuit, Capacity
|-
!
|
:Winter Capacity
|
:''Circuit.WinterCapacity''
|
:Conditional
|
:Date, TP, Circuit, WinterCapacity
|-
!
|
:Summer Capacity
|
:''Circuit.SummerCapacity''
|
:Conditional
|
:Date, TP, Circuit, SummerCapacity
|-
!Equation
|
:Formula
|
:''Equation.Formula''
|
:Yes
|
:Date, TP, Equation, Formula
|-
!
|
:Limit
|
:''Equation.Limit''
|
:Yes
|
:Date, TP, Equation, Limit
|-
!
|
:Type
|
:''Equation.EquationType''
|
:No
|
:Date, TP, Equation, EquationType
|-
!
|
:Season
|
:''Equation.Season''
|
:No
|
:Date, TP, Equation, Season
|-
!
|
:Active
|
:''Equation.Active''
|
:No
|
:Date, TP, Equation, Active
|}
----
Note:
#Mandatory Traits are the minimum required for ''EMarketOffer'' to operate
#Either a single Circuit capacity OR two seasonal Circuit Capacities are mandatorily required
----

===Input Data Structure===
''EMarketOffer'' requires input data to generate the '''''Market Data Source''''' for viewing historical market conditions and to create the '''''Forecast Definitions''''' used to generate detailed market '''''[[EMO:Forecasting|Forecasts]]'''''. Input data can be retrieved from external databases using standard SQL queries, which can be grouped together into '''''Database Connection Configurations.'''''

NOTE: Only one Database Connection Configuration can be in use at any one time. This "current" configuration tells ''EMarketOffer'' where to find the input data for the Market Data Source and for loading data into Forecast Definitions. The most recently used configuration is opened automatically as the default configuration on startup. The '''''Data Sources Option''''' in the '''''[[EMO:The_File_Menu|Main Window File Menu]]''''' can be used to make an alternative configuration current.

===Input Data Handling Mechanism===
A Configuration contains a collection of queries that may reference more than one online database. Each query can reference a single database using a pre-defined Connection containing the information required to interact with a database server; once defined a Connection can be used repeatedly by the queries in a Configuration.

The main component of a Connection is the Connection String which ''EMarketOffer'' uses to locate and connect to online database servers. The string varies in format depending on the target database server type and its specific access security provisions. It may contain:
*A data provider: this specifies the type of database server to be accessed (e.g. Oracle or MS SQL) and the specific access protocol to be used (e.g. OleDB or ODBC)
*Name of the database to be accessed (if required)
*The access parameters (typically User Name and Password)
*A Data Source Name (DSN) file containing the above information

See the section on '''''[[EMO:Securing Database Access|Securing Database Access]]''''' for more information on Connection Strings.

A separate query is used for each data item to be imported. Each query must be associated with a Connection and contain a standard SQL-compliant query string defining the specific data to be returned. Query strings should:
*Be tested first for syntax against your database
*Contain the return fields required by ''EMarketOffer''
*Contain a 'Where' clause defining a select date range using the variables $DateFrom$ and $DateTo$
*Allow aliasing of column names with attribute (trait) names used by ''EMarketOffer''
----
Notes:
#All queries must contain the name of the Trait on each returned row
#All queries (except for the Unit.Generator query) return a row for each Trading Period in the entered data range
----
'''WARNING!''' It is not possible to join data from two different databases within the same query, however ''EMarketOffer'' is designed to use a separate query for each input data item so this restriction should not constrain the program's operation. If, for any reason, you do wish to use a data item that is sourced from more than one database, you should first create a view/query in an external database that combines the required data, and then use ''EMarketOffer's'' to connect to this view.

===Creating a Database Connectivity Configuration===
The functions used in Configuration setup are fully described in the [[EMO:Menu Bar|Menu Bar]] segment under the [[EMO:The Main Window|Main Window]] section.

[[EMO:Getting Started|Back]] - [[EMO:EMarketOffer User Manual|Home]]

Special:Badtitle/NS106:Connecting to a Database Server

2013-02-08T00:47:27Z

Sisep: /* Input Data Handling Mechanism */

{{#hidens:}}
'''WARNING!''' Connecting to a local or networked database server is complex and can be time-consuming. If no people in your company have previous experience of configuring ''EMarketOffer'' it is strongly recommended that you consult your company's technical support and also the Energy Link helpdesk for advice on the best approach.
----

===Terminology===
''EMarketOffer's'' data design consists of 'things' included in the model, and their characteristics'. In this manual the 'things' are called '''Entities''' and their characteristics are called '''Traits'''. Traits may be static or dynamic.

For example the ''Circuit'' Entity can have the static Traits of ''Start Node'' and ''End Node'', and the dynamic Traits of ''Powerflow'' and ''Loss'' by Trading Period.

===Input Data Types===
''EMarketOffer'' can load the folowing types of '''''Input data''''' directly from external databases.
{|class="wikitable"
! Entity !! Type !! Trait !! Mandatory !! Defined by
|-
!Node
|
:Demand (Load)
|
:''Node.Load''
|
:No
|
:Date, TP, Node, Load
|-
!
|
:Price
|
:''Node.Price''
|
:No
|
:Date, TP, Node, Load
|-
!Unit
|
:Unit Node (Links Units to Nodes)
|
:''Unit.Node''
|
:Yes
|
:Unit, Node
|-
!
|
:Generator Unit (Links Units to Generators)
|
:''Unit.Generator''
|
:Yes
|
:Unit, Generator (Generators are always three alpha codes e.g. HLY = Huntly
|-
!
|
:Unit Generation
|
:''Unit.Generation''
|
:No
|
:Date, TP, Unit, Generation
|-
!
|
:Unit Offer Quantity
|
:''Unit.OfferQn (where n = band)''
|
:No
|
:Date, TP, Unit, OfferQn
|-
!
|
:Unit Offer Price
|
:''Unit.OfferPn (Where n = band)''
|
:No
|
:Date, TP, Unit, OfferPn
|-
!Circuit
|
:Circuit
|
:''Circuit.Name''
|
:Yes
|
:Date, TP, Name
|-
!
|
:Node From
|
:''Circuit.NodeFrom''
|
:Yes
|
:Date, TP, Circuit, NodeFrom
|-
!
|
:Node To
|
:''Circuit.NodeTo''
|
:Yes
|
:Date, TP, Circuit, NodeTo
|-
!
|
:Reactance
|
:''Circuit.Reactance''
|
:Yes
|
:Date, TP, Circuit, Reactance
|-
!
|
:Resistance
|
:''Circuit.Resistance''
|
:Yes
|
:Date, TP, Circuit, Resistance
|-
!
|
:Fixed Losses
|
:''Circuit.FixedLosses''
|
:Yes
|
:Date, TP, Circuit, FixedLosses
|-
!
|
:Capacity
|
:''Circuit.Capacity''
|
:Conditional
|
:Date, TP, Circuit, Capacity
|-
!
|
:Winter Capacity
|
:''Circuit.WinterCapacity''
|
:Conditional
|
:Date, TP, Circuit, WinterCapacity
|-
!
|
:Summer Capacity
|
:''Circuit.SummerCapacity''
|
:Conditional
|
:Date, TP, Circuit, SummerCapacity
|-
!Equation
|
:Formula
|
:''Equation.Formula''
|
:Yes
|
:Date, TP, Equation, Formula
|-
!
|
:Limit
|
:''Equation.Limit''
|
:Yes
|
:Date, TP, Equation, Limit
|-
!
|
:Type
|
:''Equation.EquationType''
|
:No
|
:Date, TP, Equation, EquationType
|-
!
|
:Season
|
:''Equation.Season''
|
:No
|
:Date, TP, Equation, Season
|-
!
|
:Active
|
:''Equation.Active''
|
:No
|
:Date, TP, Equation, Active
|}
----
Note:
#Mandatory Traits are the minimum required for ''EMarketOffer'' to operate
#Either a single Circuit capacity OR two seasonal Circuit Capacities are mandatorily required
----

===Input Data Structure===
''EMarketOffer'' requires input data to generate the '''''Market Data Source''''' for viewing historical market conditions and to create the '''''Forecast Definitions''''' used to generate detailed market '''''[[EMO:Forecasting|Forecasts]]'''''. Input data can be retrieved from external databases using standard SQL queries, which can be grouped together into '''''Database Connection Configurations.'''''

NOTE: Only one Database Connection Configuration can be in use at any one time. This "current" configuration tells ''EMarketOffer'' where to find the input data for the Market Data Source and for loading data into Forecast Definitions. The most recently used configuration is opened automatically as the default configuration on startup. The '''''Data Sources Option''''' in the '''''[[EMO:The_File_Menu|Main Window File Menu]]''''' can be used to make an alternative configuration current.

===Input Data Handling Mechanism===
A Configuration contains a collection of queries that may reference more than one online database. Each query can reference a single database using a pre-defined Connection containing the information required to interact with a database server; once defined a Connection can be used repeatedly by the queries in a Configuration.

The main component of a Connection is the Connection String which ''EMarketOffer'' uses to locate and connect to online database servers. The string varies in format depending on the target database server type and its specific access security provisions. It may contain:
*A data provider: this specifies the type of database server to be accessed (e.g. Oracle or MS SQL) and the specific access protocol to be used (e.g. OleDB or ODBC)
*Name of the database to be accessed (if required)
*The access parameters (typically User Name and Password)
*A Data Source Name (DSN) file containing the above information

See the section on '''''[[EMO:Control_Panel_Components#The_Data_Source_Manager|Data Sources Options]]''''' for more information on Connection Strings.

A separate query is used for each data item to be imported. Each query must be associated with a Connection and contain a standard SQL-compliant query string defining the specific data to be returned. Query strings should:
*Be tested first for syntax against your database
*Contain the return fields required by ''EMarketOffer''
*Contain a 'Where' clause defining a select date range using the variables $DateFrom$ and $DateTo$
*Allow aliasing of column names with attribute (trait) names used by ''EMarketOffer''
----
Notes:
#All queries must contain the name of the Trait on each returned row
#All queries (except for the Unit.Generator query) return a row for each Trading Period in the entered data range
----
'''WARNING!''' It is not possible to join data from two different databases within the same query, however ''EMarketOffer'' is designed to use a separate query for each input data item so this restriction should not constrain the program's operation. If, for any reason, you do wish to use a data item that is sourced from more than one database, you should first create a view/query in an external database that combines the required data, and then use ''EMarketOffer's'' to connect to this view.

===Creating a Database Connectivity Configuration===
The functions used in Configuration setup are fully described in the [[EMO:Menu Bar|Menu Bar]] segment under the [[EMO:The Main Window|Main Window]] section.

[[EMO:Getting Started|Back]] - [[EMO:EMarketOffer User Manual|Home]]

EMK:Water Values Bulletins

2013-02-07T22:39:26Z

Sisep:

{{#hidens:}}

[[EMK:Multi Dimensional Water Values | Multi Dimensional Water Values]]

[[EMK:Water Values and Hydro Offers | Water Values and Hydro Offers]]

[[EMK:Water Values in the EMarket Model | Water Values in the EMarket Model]]

[[EMK:Improvements to Water Values | Improvements to Water Values (2009)]]

[[EMK:Technical Bulletins | Back to Technical Bulletins]]

EMK:Glossary

2013-02-07T22:37:17Z

Sisep: /* W */

{{#hidens:}}

==B==
'''Batch Resource'''

A collection of individual run definitions for consecutive execution. The sequence of runs in a Batch is important for correct operation of the multi-year run function.

'''Binary Output Specification Resource'''

A specification of which binary data will be output when running a run definition.

==C==
'''Circuit'''

A link, either a line or a transformer, with electrical characteristics such as Resistance and Reactance and which is capable of carrying a Power Flow on the Transmission System.

'''Company Resource'''

Defines the Generator ownership, retail load and hedge contract information for a specific Company. EMarket allows different versions of the same Company to be defined.

'''Connections Resource'''

Specifies a single physical network configuration as a collection specific Generator, Company and grid resources.

'''Constraint'''

A constraint occurs when one or more Circuits reaches its local maximum carrying capacity set by the System Operator to maintain grid security or power quality.

==D==
'''Demand'''

The rate of consumption of electrical energy measured in MW/h (1 MW/h = the electrical energy consumed by a 1,000,000 watt appliance in one hour

'''Dispatch'''

The process of calculating the key Market and Transmission System data for a single Tick using a Run Definition(Demand, Offers and Transmission System configuration) to calculate Price, Generation and Power Flows.

==E==
'''Entity.Trait Model'''

EMarket's internal data model is composed of entities and traits.

Entities represent elements of the modelled electricity market (e.g Node or Generator). There can be many instances of the same entity type (e.g. Manapouri and Huntly are both instances of the Generator entity type.

Traits are data elements that describe an entity. For example, each Generator entity has several traits such maximum output or Injection Node.

Entity.Trait references can be used constructing formulae in most EMarket data entry fields.

==F==
'''Forecast (n)'''

The output (data set) from a single execution (Run) of the model using a Run definItion

'''Forecast (v)'''

The process of using the EMarket model to simulate the behaviour of the NZ electricity Market under a specific set of conditions.

==G==
'''Generation'''

The electrical energy produced by one or more generators measure in MW or MW/h.

'''Generation Company'''

A real or supposed Company offering electricity into the NZ Electricity Market.

'''Generator Resource'''

Specifies the physical characteristics of a generation facility (e.g. maximum capacity, efficiency etc). EMarket allows several 'logical' versions of the same physical Generator to be defined.

'''Grid Resource'''

Specifies the physical arrangement of circuits and Nodes in a specific Grid configuration.

==I==
'''Infeasibility'''

A situation in which a feasible dispatch solution cannot be calculated from a given set of Dispatch Factors.

==M==
'''Map'''

A multi-featured map-based graphical presentation of the Current Data Set.

'''Marginal Generator'''

A Generation Unit, the output of which varies in response to changes in load while the output of other Units is held constant.

'''Market'''

The New Zealand electricity wholesale market where the spot price for electricity is determined.

'''Market Operator'''

A collective term for the service providers of number of operational and administrative services to the market under the wholesale and retail sections of the rules

'''Market Participant'''

Any party involved in the regulation and operation of the NZ electricity Market:
*retailers
*distributors
*generation companies
*consumers
*line owners
*service providers
*metering equipment owners
*ancillary service agents
*data administrators
*system operator

==O==
'''Offer'''

An Offer by a Generation Company to sell a quantity of electricity at a specified price for a specific Trading Period.

==P==
'''Power Flow'''

For a given circuit, the average quantity of electricity (in MW) calculated to be flowing in the circuit in any one trading period.

'''Price'''

The price of electricity calculated through dispatch for a single Market Node in a single Tick.

==R==
'''Regions Resource'''

A collection of user-defined geographic Regions (as a set of Nodes).

'''Reports Resource'''

Specifies which pre-designed Reports and data summaries will be created at Forecast run time. Reports not generated at run time can be generated from the Binary Output data at any subsequent time (providing the binary data has been generated through the binary data output specifications).

'''Resource'''

The Resource is a core building block in EMarket. It is a specific configuration of a modelling, scheduling or data formatting element used to:
*specify a Forecast scenario
*schedule and control the production of a Forecast
*format the output of Forecast data

'''Run'''

The operation where EMarket creates a Forecast by executing the model using a specific instance of a run definition.

'''Run Definition'''

The set of input data required by EMarket to effectively execute a single Forecast Run of the model.

==S==
'''Schedule Resource'''

Schedule Resources allow you to specific in-run control over various aspects of a Run during its execution.

'''SPD'''

The Scheduling, Pricing and Dispatch software used by the System Operator and the pricing manager.

'''Supply Curve'''

The Supply Stack for a given Tick represented as a chart of Offer Price versus Offer Quantity.

'''System Operator'''

A service provider responsible for day-to day scheduling and dispatching of electricity on the Transmission System (currently Transpower).

==T==
'''Tick'''

The term 'Tick' refers to Demand information which EMarket requires to drive a simulation One Tick is a specified period, over which EMarket simulates the market with one dispatch. A Tick can be as short as half an hour, or as long as a whole month.

'''TPIX'''

Transpower Information Exchange: a computer system providing SCADA events and energy, wholesale market prices and schedules, network model and Reserve market information to market participants.

'''Trading Period'''

The fundamental half-hour interval for which individual Market Prices are calculated.

'''Transmission System'''

The national grid: a high-voltage electricity transmission network owned and operated by Transpower, and used to transmit electricity throughout New Zealand.

==W==
'''Windows Clipboard'''

A Windows feature that allows you to copy and move information within an application or among applications. The Windows Clipboard temporarily stores the information in memory before you paste the data in a new location.

'''Working Directory'''

A set of folders containing an independent EMarket modelling environment. Instructing EMarket to change the current Working Directory causes the program to reconfigure itself using the resources available in the new directory.

[[EMK:EMarket User Manual | Back to User Manual Main Page]]

2013-02-07T21:10:11Z

Sisep: /* The Connections Grid Box */

{{#hidens:}}
[[File:Connections_Window.jpg|400px|thumb|right|Example Connections Window]]
This is where all the physical grid data is assembled. '''''[[EMK:Generators | Generators]]''''', '''''[[EMK:Grid | the Grid]]''''', '''''[[EMK:Demand Profile | the underlying Demand Profile]]''''' and '''''[[EMK:Companies | Companies]]''''' are all grouped together.

The Connections Window contains three main components:
*Connections '''''[[EMK:grid | Grid]]''''' box
*Connections '''''[[EMK:generators | Generators]]''''' box
*Connections '''''[[EMK:companies | Companies]]''''' box

==The Connections Grid Box==

The Connections Grid Box is found at the top of the Connections Window and is used to specify the Grid-related details associated with the Connections Resource.

'''Connections Grid Box Fields'''
{|class="wikitable"
|-
| ''Name'' || The name of the Connections Resource Instance.
|-
| ''Description'' || A free text field for supplementary details about the Connections Resource.
|-
| ''Grid'' || Specifies the Grid Resource associated with this Connections resource. Double clicking on this field opens the '''''[[EMK:Grid | Grid Window]]''''' for the specified Grid.
|}

Warning: The ''Demand Profile'' field in the '''''[[EMK:Regions | Regions Window]]''''' does NOT have any influence on the Demand at the Nodes during a Run - its purpose is to provide a list of Nodes to assign to Regions.

==The Connections Generators Box==

The Connections Generators Box is found on the left of the ''Connections Window'' and is used to specify the list of Generators attached to the Grid.

To add a Generator, click on any field in the box and press the ''<insert>'' key. A popup window with a drop down list for selecting valid generators will appear, highlight the required generator and press ''<Enter>'' to make selection, click OK to confirm or cancel to discard the selection.

'''Connections Generators Box fields'''
{|class="wikitable"
|-
| ''Name'' || The name of the Generator attached to the Grid. Double clicking on this field causes the Generator Window for the selected Generator instance to open.
|-
| ''Version'' || Specifies the version of a particular Generator. Allows quick substitution of different versions of the same Generator Instance into a Run Definition without having to change any references made to the Generator Resource elsewhere in the setup.
|-
| ''Node'' || Displays the Generator's injection Node as entered through the [[EMK:Generators | Generator Window]]. Note that the ''Node'' field cannot be edited from this window.
|-
| ''Class'' || Specifies the class of Generator (Hydro System, Hydro Station, or Generic). Note that the ''Class'' field cannot be edited from this window.
|}

==Connections Companies Box==

The Connections Companies box is found on the right hand side of the ''Connections Window'' and is used to specify the Companies included in the Connection Resource.

'''Connections Companies Box Fields'''
{|class="wikitable"
|-
| ''Name'' || Company name. Double clicking this field will bring up the [[EMK:Companies | Company Window]] for that particular Company.
|-
| ''Version'' || A Company version name or number.
|}

Note that it is not necessary to have Companies present in the Connections Window for a run to proceed.

[[EMK:Connections | Back to Connections Main Page]]