Load Forecasting Trial FAQ

Questions & answers arising from Transpower's trial for the Load Forecasting Model.

This FAQ includes questions and answers previously provided as well as all the latest clarifications, corrections, questions, and answers issued to date for Transpower's load forecasting trial.

New FAQs 10 Feb 2012

Trial
Q. Will there be a process in place that informs the pilot participants of any exceptions with regard to their forecast submissions?
A. Provided you comply with the web service (Trial System) schema and rules, your forecasts should be submitted, received, and recorded. Aside from exception messages from the web service itself to you, the process for informing participants of any exceptions with regard to their forecasts will be around Transpower's inspection of the forecasts. This will be done approximately weekly, and if Transpower notices missing forecasts, that will be taken up with the relevant Participant, and dealt with on a case-by-case basis. It is noted that this process is still being put in place, and that inspections will start as soon as possible after the start of the Trial (when we start to receive forecasts). The key is that Transpower is interested in ascertaining accuracy of your forecasts during the Trial. Clearly, in order to do this, your forecasts must be received, but we will be reasonable about this, and understand that there may be some 'bedding in' at the start of the Trial.

GXPs
Q. The confirmed list of grid point locations contains 180 GXPs and HLY2201 & MAT1101 are on this list. However SCADA data for those two GXPs is consistently missing in the web service response. Can you confirm whether HLY2201 and MAT1101 are still on the confirmed list of GXPs?
A. They are on the confirmed list. However Transpower is still looking into these GXPs, and understands that the SCADA data is not available to the Trial at present. Transpower will exclude them from the analysis of performance until the provision of SCADA data for them has been resolved. So you do not need to produce load forecasts for HLY2201 and MAT1101 until the SCADA data becomes available. We will notify you when that happens.

Public holidays
Q. Can you provide dates of national and provincial holidays?
A. Public and provincial holidays can be found at the following link: http://www.dol.govt.nz/er/holidaysandleave/publicholidays/publicholidaydates/ School holiday information can be found at the following link: http://www.minedu.govt.nz/theMinistry/EducationInNewZealand/SchoolTermsAndHolidays.aspx

GXPs with Zero or Negative Load
Q. Would Transpower consider removing the WPA2201, SFD2201, SWN2201, BPD1101, and PEN0251 GXPs from the Trial, because they have multiple load of zero, and negative loads?
A. At this stage Transpower intends to keep these GXPs in the Trial. By way of explanation of the load at these:
WPA2201, SFD2201, and SWN2201 are all grid connected generators, which show zero load when operation, but may show some load (in the order of a few MWs) when the generators are out of service, or not generating. This is local service load at the generators. Not all grid connected generators have local service load from the main GXP, as it is supplied through the distribution network. Where known, we have removed these generators, which do not have local supply load from the main GXP, from the Trial.

BPD1101 has predominantly irrigation load connected to it, as does the BPT1101 GXP.

PEN0251 is a GXP for electric railway supply, although this supply has not been commissioned at present – hence its load is zero. A commissioning date is not known at present. For your information, MAN0551, TMN0551 (HM zone), BPE0551 and TNG0551 (PN zone) are all electric railway supplies. In some cases these trains may have regenerative braking (i.e. they generate and inject into the grid when braking).

Trial
Q. What if something unplanned, and beyond our control, happens during the Trial, which results in our forecast not being received by Transpower for a period of time?
A. Transpower will consider these on a case-by-case basis, and has the ability to exclude exceptional days during the Trial when assessing performance.

GXP Changes
Q. Is it possible that GXPs may change during the Trial?
A. GXPs are occasionally decommissioned, and new ones commissioned at times. Transpower will endeavour to provide plenty of notice of any chances. The next change expected is: New GXP, KOE1101 from 1/4/2012 period 1, in Zone NL. Decommission GXPs, KTA0331 & KOE0331, data supplied until 31/3/2012 period 48. Essentially KTA0331 & KOE0331 are being replaced by KOE1101. For this change, Transpower will endeavour to make the SCADA and GXP file data available for the new GXP from the start date.

FAQs 25 Jan 2012
SCADA Data
Q. Can you point out the difference between half-hourly and 5 minute SCADA?
A. The two SCADA data services of 5mins and 30 min data are provided to ensure accuracy of the data. SCADA receives multiple readings from the GXPs. The 5min data is the initial reading for the GXP for the previous trading period, but is not completely accurate. The 30 minute reading (which is not received until 27 min after the trading period has completed) is intended to be the most accurate SCADA reading for the GXP as all available data should have been collected.

Q. What happens if the Trial system goes off line for a while, and we miss the 5 minute data? I assume there is no possibility of retrieving that at all?
A. Yes you are correct, it cannot be retrieved. If available, the 30 minute data should be more accurate and should be requested in this case. It is more likely that an upstream system will be offline rather that the Trail System. If the Trial System does not have the data, then it will be unavailable to the Trial. An alternative might be to retrieve the 30 minute data should this occur.

GXP File Negative Values
Q. What do negative values in the GXP file mean?
A. Negative values in the GXP file represent injection.

GXP File
Q. Can you confirm that: (1) GXP data will only be available once per day, and (2) performance of the load forecast will be judged on GXP data, not on the SCADA data.
A. (1) Correct, (2) Correct

GXP File
Q. Why does the data for WPA2201 and HLY2201 stop at 30-11-2011?. Since these are in the trial, we need to bring them up to date with the others.
A. The two GXPs don't have data up to 31-12-2011 as they come from a different source, and their data was not available when the GXP file was created. SCADA data will be provided for these at the 5 minute intervals and 30 minute intervals. The GXP file may not be available for them each day; in these two cases it is suggested to use the SCADA data instead.

Historical GXP File
Q. Why is the Historical GXP File data different in magnitude to the SCADA data.
A. The data in the GXP file is in kWh, not MW. To convert to MW please multiply by 2 and divide by 1000. The reason it is in kWh is because it is revenue metering data.

Historical Data
Q. Is the latest round of data you sent simply the historical GXP data (i.e. the data we have uploaded already) in kWh and not in MW?
A. The latest round of data sent (on 16-1-2012) is the historical GXP data (strictly speaking, called the Bulk GXP File). This is in kWh, because it comes from revenue meters, which are different meters to the SCADA meters. Transpower did not convert this to MW, as we wanted to provide the exact data that came from that metering source. The data provided to you previously (in 2011) is the Bulk SCADA File.

During the Trial, Transpower will provide two types of data, which mirror the historical data already provided:
(1) SCADA data (MW), available every 30 minutes, at 5 minute or 30 minute intervals (called the SCADA File).
(2) GXP data (kWh), available once per day, at 30 minute intervals (called the GXP File).

Transpower requires a load forecast for each half-hour in MW. In assessing that load forecast, Transpower will compare this to the GXP File (converting units appropriately). This was stated in Q#36 from the NTT # 10 provided during the tender, which states that "Transpower intends to use the revenue meter data (the GXP file) as the actual demand when comparing load forecast with actual demand."

It would be ideal if the GXP File was available each half-hour, however this is not the case. Hence Transpower is providing the SCADA File each half-hour.

Load Forecast Units
Q. Can you confirm that Transpower is expecting load forecasts in units of kWh and not MW?
A. Transpower expects the load forecasts in units of MW, and Transpower will do the comparison in MWs. Hence Transpower will convert the GXP file from kWh to MW for the purpose of the forecast comparisons.

Load characteristics
Q. Is Transpower open, as part of the trial, to reviewing the ‘odd cases’ with the goal of getting a better understanding of what is driving the loads?
A. Yes, on a best endeavours basis.

Load characteristics : ADD0661 Load
Q. Can you explain why the ADD0661 load has dropped off? It is clearly related to the Christchurch earthquake; what is the expectation of load recovering at ADD0661?
A. The Addington substation supplies a number of distribution network feeders into the Christchurch CBD. With the extensive damage to the CBD, the CBD being ‘red zoned’ (i.e. no access allowed to it), and the resulting move of businesses away, the load dropped off, and has not recovered to its full level. The substation itself is undamaged, and the timing of load recovering at ADD0661 substation, is linked to the re-building programme of the Christchurch CBD. That programme is still not fully determined.

Load characteristics : ABY0111 and Negative loads
Q. ABY0111 appears to be one of the meters with both positive and negative flows. Forecasting this type of load can be challenging, as flow direction reflects operational decisions. Can you explain why Transpower requires forecasts at these GXPs?
A. There is an embedded generator in the distribution network connected to ABY0111. The key point is: that the users of the load forecast (such as Transpower System Operators, Pricing Schedules, Demand Side Initiatives, and Operations Planners), require the net load to or from the grid. In turn Transpower requires the net load for each zone, island, and country. This is the rationale for FR-0620 from the tender (Transpower requires that the supplier should have a load forecasting solution that explicitly allows for the effect of generation from embedded wind and other embedded generation). While Transpower realises that forecasting load at GXPs with embedded generators is challenging, the embedded generators do contribute to the net load.

For your information, the following link lists the publicly available information on known generating plant in the NZ power system:
http://www.ea.govt.nz/industry/modelling/long-term-generation-development/list-of-generation-projects/

Additional Forecasts
Q. We will use our own indepently sourced weather forecast for the load forecast Trial. We may also be able to supply an additional load forecast, based on the weather from the Transpower web service (Trial System). Would this be something Transpower would be able to accommodate?
A. Transpower encourages this as supporting the trial goals of finding the best load forecast, and will accommodate it with the following provisos:
(1) It is set up as two distinct systems, with another Transpower Web Service login.
(2) The Transpower Trial System has the capacity to communicate with, and to store this additional forecast; if any capacity limitations are found during the Trial, Transpower may discontinue an such additional load forecasts.
(3) Its purpose is to provide a load forecast from a different weather data source only.
(4) There is no additional cost to Transpower.
Participants are requested to let Transpower know as soon as possible whether they wish to do this.

Non-conforming GXPs
Q. From the data historical data provided, the implication is that Transpower does not expect any vendors to provide forecasting for the non-conforming GXPs. As an overall picture of demand forecasting, how does Transpower handle the non-conforming portion of the load?
A. Correct, Transpower does not require load forecasts for the non-conforming nodes (FR-0160). Transpower requires the loads at non-conforming GXPs to provide, what are essentially, forecasts of their load. This is mainly because the majority of the non-conforming GXPs are heavy industry, and their load shifts are based on process management and production levels – neither of which are factors that Transpower has access to.

Web service
Q. Have the GXPs on the web service (Trial System) been updated?
A. GXPs on the Trial System have now been updated, to match the list ‘LF_Trial_GXPs_23-12-2011.xls’

Weather data
Q. There are often temperature, and other fields, missing in the observed weather data.
A. This has been escalated to the weather data provider. Changes are being made at present (25-1-2012) to deal with this.

Older FAQ's

Dates/Timing
Q. What is the trial start date?
A. 13 February 2012

Q. Is the Web service still scheduled to be up and running on the 1st of December?
A. The web service is now available.

URL
Q. What is the URL for requesting information?
A. For Requesting Information it is https://ws.ems.co.nz/services/RequestingInformation

Q. What is the URL for the forecasting information?
A. For forecasting it is https://ws.ems.co.nz/services/Forecasting

Trial System Authentication
Q. What access credentials will be used for the trial system?
A. It will be via a user name token. Transpower has issued this token with a password to each Trial Participant.

Web Service Planned Outages
Q. Confirm what is Transpower’s maintenance window?
A. Transpower does have a maintenance window for its non-EMS systems from Wednesday 12noon to 1pm, which may affect the availability of data. The Trial System will always provide the data if it is available. However as it is a ‘down-stream’ system, data it requires may not always be available. The Trial System / web interface is hosted from EMS, who do not have a regular scheduled maintenance window. However outages of the em6 system may occur when a new release is made to the application. Since the em6 system is an ‘upstream’ provider of data to the Trial System, outages may result in the loss of some SCADA data.

Web Service Start Date
Q. Is the web service still scheduled to be up and running on the 1st of December?
A. The web-service has been delayed but is expected to be running within a few weeks of 1 December. Trial participants will be informed when the system is ready.

Forecast Format
Q. Does Transpower require forecasts for just the GXPs, or aggregates (zones, islands, and country)?
A. Transpower requires the load forecast to the GXP, zone, island, and national level (consistent with Requirements FR-0110, FR-0120, FR-0130, and FR-0140).

Forecast Timing
Q. To fully understand the amount of time we have available to produce the 30 minute resolution forecast, are the following examples correct:
(1) Just after 1pm, load data up to 1pm point will be available for all the GXP’s via the Transpower Web Service for download.
(2) We will then have until just before 1:30pm to produce and send a ‘short forecast’ file with 30 minute resolution to the Transpower Web Service.
A. Your understanding is correct. See also Question 8 of NTT#10.

Long Forecast Submission Deadline
Q. When does Transpower require the ‘long forecast’ file (the 30 minute resolution forecast produced to cover 14 days)?
A. Daily at or before 0130 hours, consistent with Requirement FR-0215 of the RFP.

2 Day Forecast Window
Q. Will the two day forecast (the ‘short forecast’) just need to go to the end of the next day, or will it need to be a 48hour window. i.e. which would be correct:
(1) At 1:30pm on the 1st, you would expect a forecast going to midnight on the 2nd, or
(2) 1:30pm on the 1st, you would expect a forecast going to 1pm on the 3rd?
A. The short forecast is definitely required for a 48 hour window (horizon), consistent with Requirement FR-0210 of the RFP. So '2' is correct.

Trading period
Q. The data received from the web service, is it period beginning or period ending?
A. Trading periods relate to the time of the day as follows:
Period 1 is for 00:00 to 00:29
Period 2 is for 00:30 to 00:59
Period 3 is for 01:00 to 01:29
Etc. to
Period 48 is for 23:30 to 23:59

Five Minute Data
Q. Please confirm that there will be no need for 5 minute resolution forecasts throughout the entire trial period, and that if we’re selected after the trial, advanced warning will be given about the start date for producing a 5 minute forecast.
A. Your understanding is correct. During the Trial SCADA data will be provided by Transpower at 5 minute resolution, however Forecasts are only required at 30 minute resolution.

Trial System Inferface Specifications
Q. We presently have Trial System specifications dated Thur 15th Sep, are any updates available?
A. A build pack and updated ‘resource bundle’ was emailed on Friday 16 December 2011. Please use these specifications as the most up-to-date version.

Historical Weather Data
Q. There are a number of weather stations missing temperature data.
A. From the weather data provider: The explanation for most sites is basically that there are times when the automatic weather stations have missing data. Normally because a sensor develops a fault or the station goes off line for some reason. If we find that the information from a station is suspect we will turn that parameter off until we can service or replace the sensor. This ensures that such data are missing rather than bad. Depending on where the station is, or the nature of the fault, there can be a number of days before we can fix the fault.

Q. The J’ville Helston Rd has no temperature readings at all.
A. This will be replaced with Otaki – data will be sent out when it is available. Note: J’ville Helston Rd and Otaki are both historical and observation stations only; forecasts will not be provided for these stations.

Q. The Newlands Off-ramp file (93408) appears to be missing altogether, but there is an extra file (93439).
A. It was found that the Newlands off ramp (93408) did not report temperature and it has therefore been replaced with Wellington Airport (93439). Note: Newlands Off-ramp and Wellington Airport are both historical and observation stations only; forecasts will not be provided for these stations.

Q. Some files are incomplete (i.e. they do not start 1-10-2008).
A. This is because not all stations were operational or recording that far back.

Forecast Weather Data
Q. Will forecasts for all stations listed in the file ‘Station List.xlsx’ be provided?
A. No, only those stations with an ‘x’ in the column ‘Fcst site’ will be forecast, and provided with actual weather recordings. That makes a total of 21 forecast stations.

Historical SCADA Data
Q. A number of GXPs listed in Appendix F are missing from the historical SCADA data provided.
A. There are two reasons for this. The first is that non-conforming nodes are excluded from the historical SCADA data, because forecasts for non-conforming nodes are not required (FR-0160 and FR-0170 from the tender).
The second is that a number of generator nodes have been excluded from the SCADA data, but are listed in Appendix F. Transpower has reviewed the GXPs for the Trial and removed some generator GXPs and some decommissioned GXPs.
Transpower has released an updated GXP list dated 23-12-2011.

Q. A number of GXPs in the historical SCADA data have missing readings.
A. Each GXP should have 54,048 readings covering the 3 year and 1 month (October) period in the historical data. This is (365 x 3 + 31) x 48 – 6 trading periods (skipped in the September transition to daylight saving) + 6 trading periods (inserted in the April transition from daylight saving). GXPs with missing load readings fall into three categories:
• They have been decommissioned since 1-10-2008. Decommissioned GXPs have been excluded from the new GXP list if not already removed (see previous FAQ). Decommissioned GXPs are: MRA0111, MTM0111, and OKI0111.
• They are new nodes in the market model since 1-10-2008. New GXPs (and the date their readings start) are: BPD1101 (17-10-2010), HLY0331 (16-12-2008), KMO0331 (10-6-2009), NAP2201 (1-10-2009), NPL0331 (1-7-2010), and PEN0251 (27-4-2011).
• Data are missing. In a small number of cases load readings have been lost. This appears to coincide with Transpower’s weekly maintenance window on a Wednesday, or monthly maintenance window on a Tuesday. In all but one case just one load value is missing. These nodes are: ABY0111 (p29 18-5-2011), CST0331 (p26 27-10-2010), FHL0331 (p26 13-4-2011), KIN0331 (p30 8-12-2010), STK0331 (p31 10-8-2010), and WVY0111 (p32 16-11-2008). The GPX MER0331 has periods 26 on 6-5-2009 to 24 on 7-5-2009 inclusive missing.

UTC Format
Q. Please confirm that our understanding and interpretation of the format of a UTC date is correct. We understand that the format of a UTC date for use in Web Services is YYYY-MM-DDZ where: - is a field separator, YYYY is the year, MM is the 2-digit (leading 0) month, DD is the 2-digit (leading 0) day, and Z is a specific UTC Time Zone identifier.
A. Your understanding is correct.

SCADA file and GXP file
Q. What do the terms ‘MV90’ and ‘GXP file’ mean?
A. The term MV90 is sometimes used to refer to the 'GXP file', referred to in the Trial Agreement.
The GXP file is the data used for market reconciliation and billing, and hence load forecast accuracy will be measured against this. However it is only available daily, for each trading period of the day. That is why we are providing SCADA data also via the Trial System web service - at 5 minute intervals, updated each half-hour.

Q. It seems to be possible to obtain half-hourly SCADA data. I understood that SCADA data was to be 5 minute averages?
A. SCADA data is half hourly when requested with a date range. When requested without a date range the latest finished period’s data is returned and that data is aggregated into 5 minute intervals. However half-hourly SCADA data is not available until about 27 minutes after the start of the current trading period, for the previous trading period. By contrast, the 5 minute SCADA data is available more rapidly (usually within 5 minutes after the end of the previous 5 minute period).

Q. The request for the recent SCADA data (ie. without the DateRange element) return seems to return the six intervals for the recently completed period. Is this the final implementation, because there are some notes that the recent data would be aggregated to the 30 minute period. The aggregation is not an issue - the question is whether this implementation is the final implementation or still subject to change.
A. Yes, the aggregation of recent SCADA data into 5 minute intervals is the final implementation for recent SCADA data (i.e. requests with no date range specified). SCADA data requests with a date range specified will provide whole period data and will not include interval data (see correction to the build pack).

Web Service
Q. Currently all missing values are being communicated as NULL. Please can you confirm that this is the final implementation or whether the returned data is still subject to change e.g. returning "N/A"?
A. Yes, missing data in any of the retrievable datasets will be indicated by the absence of that data time (i.e. NULL) in the payload. This is the final implementation and there is no intention to change it (any change would be as a result of a business change and would require a reissue of the schemas).

Build Pack
Q. Correction to Build Pack
A. The build pack has an interval in the example for the SCADA data request with a date range. This is wrong. (This is the 2nd example on Page 7). The correct example has been emailed to all Trial Participants on 10-1-2012.