CSIRO advises that the information contained in this publication comprises general statements based on scientific research. The reader is advised and needs to be aware that such information may be incomplete or unable to be used in any specific situation. No reliance or actions must therefore be made on that information without seeking prior expert professional, scientific and technical advice. To the extent permitted by law, CSIRO (including its employees and consultants) excludes all liability to any person for any consequences, including but not limited to all losses, damages, costs, expenses and any other compensation, arising directly or indirectly from using this publication (in part or in whole) and any information or material contained in it.
CSIRO is committed to providing web accessible content wherever possible. If you are having difficulties with accessing this document please contact csiroenquiries@csiro.au
Figure 1 Schematic of high-level applications architecture. Either MVApp or SWCApp can be called directly as stand-alone programs. Alternatively, the web portal can be used. ...................... 14
Figure 2 Screenshot of the main buildings
Figure 3 Screenshot of the baseline model eligibility window in the Web Portal. 17
Figure 4 Visualisation of the difference between measured (or targeted) energy savings and claimable energy savings. Minimum claimable savings is defined as the intercept of the claimable savings line (black line) with the vertical axis. Here minimum claimable savings is 3.9%, measured or estimated savings is 11.6% and actual claimable savings is 4.6%.
Figure 7 Schematic relationship between electricity metering streams. **Note: consumption is not a stream class.
Figure 8 Overview of MVApp algorithm key stages
Figure 9 Schematic showing piecewise continuous temperature changepoints and associated coefficients. ........................................................................................................................................
Figure 10 Screenshot of all M&V applications with results shared on the EVO-portal (as of 26th July 2023). DCH-M&V application results as indicated. 51
Figure 11 Cumulative histograms of R2, CVRMSE and number of over or under-estimated months, and boxplots of CVRMSE for interval baseline models for EVO portal buildings. ............... 52
Figure 12 Magnitude of estimated and actual energy saving for sites with valid building models (interval model and daily model results shown). Interval model slope = 0.9798. Daily model slope = 0.9689. 54
Figure 13 Distribution of credited percentage saving (left) and boxplots of estimated saving percentage as a function of CVRMSE (right) for sites with a valid baseline model based on the interval analysis model. ..................................................................................................................... 54
Figure 14 Distribution of credited percentage saving (left) and boxplots of emitting saving percentage as a function of CVRMSE (right) for sites with a valid baseline model based on the daily analysis model. 55
Figure 15 Increase in CVRMSE between baseline consumption daily model and net meter based daily model as a function of relative generation amount. ................................................................ 58
Figure 16 Comparison of estimated percentage savings and uncertainty ranges as computed using hourly, hourly net, daily and daily net models for seven sites................................................. 59
Figure 17 Correlation between daily site generation and daily horizontal irradiance (left) and hourly site generation and hourly horizontal irradiance (right) for Site 6. Dashed lines show fitted linear regression model with zero intercept. ........................................................................... 59
Figure 18 Boxplots of hourly irradiance – generation linear regression residual as a function of hour of the day. .................................................................................................................................. 60
Figure 19 Daily energy use variation plots. Actual daily energy use during the baseline period (blue line), monthly mean estimated energy use (black line), expected range of monthly estimated (blue shaded region), daily outliers (symbols) and monthly outliers (pink and light green lines). Top left: case just failing CVRMSE criteria only (201). Top right: case failing time trend (203). Centre left: case failing CVRMSE and showing large seasonal variation (204). Centre right: large spikes in consumption coupled with periods of lower and higher consumption (207) Bottom left: data quality issues confounded by the presence of many meters (509). Bottom right: case passing all criteria (311). 64
Figure 20 Variation of estimated savings as a function of percentage of excluded data for different data exclusions strategies. .................................................................................................. 68
Quick Reference Chart
User story
How do I use the M&V Web portal and what is it’s basic functionality?
My baseline model failed to meet the criteria – why?
What are the baseline model criteria?
How is the data prepared?
My data is in CSV or not in Senaps/DCH, how do I run an M&V analysis on it?
Why is there more than one M&V application
How do I run a custom M&V analysis using the DCH building semantic models?
How do I run a custom M&V analysis specifying the data-streams or sources to use?
How are savings calculated?
What algorithm is used for the baseline model?
How are non-routine events handled?
Can MVApp be used to compute normalised year savings?
Do the algorithms work with net meter data?
Reference
Section 2.3
Section 2.3.3, 3.4 and e3.9.2
Section 3.4
Section 3.1, 2.4.3.4, 2.4.3.3
Section 2.4.3.1
Section 2.1, 1.4
Section 2.5
Section 2.4
Section 3.6.3, 3.6
Section 3.3
Section 2.4.3.11, 3.10.3
Section 2.4.4.1, 3.10.4
Section 2.4.3.2, 3.8.1,
Part I Applications guide
1 Introduction
1.1 What is M&V?
Measurement and Verification (M&V) is the formal process of measuring the energy consumption of a site, building or specific equipment both before and after making changes to the equipment or its operation, for the purposes of determining energy savings. The NSW Metered Baseline Method Guide [1] and Project Impact Assessment with M&V (PIA M&V) guide [2] provide detailed information on the methodology that must be followed to be eligible for certificates under the NSW Energy Efficiency Scheme. Other jurisdictions within Australia such as Victoria [3] and South Australia [4] also provide guidance on their own M&V based methods. Internationally the Efficiency Valuation Organisation (EVO) [5] has published several guidelines, and in the US, ASHRAE Guideline 14 [6] relates to M&V methods.
Under the M&V methodology, energy savings are calculated under normalised conditions by using the pre-upgrade (baseline) data to develop a model for estimating what energy consumption would have been post-upgrade, had the upgrade not taken place. That is, energy savings are not directly measured because they represent a counter-factual; a difference between what actually occurs, and what would have occurred, had no changes been made.
In some instances, separate models are fitted to both the baseline and post-upgrade periods and used to estimate future savings for a normalised or reference year. In this respect, the process of developing a baseline model for M&V is similar to developing a generic load forecasting model. The main difference being baseline M&V models are used either to ‘back-cast’ (i.e. forecast for intervals of time in the past) or to forecast for normalised or reference conditions as opposed to expected actual conditions. Another potential distinction is that M&V models must make their predictions over long horizons, whereas load forecasting algorithms typically only need to provide forecasts a short time (e.g. 24 hours) ahead. Notwithstanding this, algorithms that work well for load forecasting may well be suited to M&V baselining, and vice-versa.
Central to the M&V process is the determination of a baseline model which includes establishing the key independent variables that influence energy consumption. Other important elements include the data cleaning and processing methods, the method to estimate uncertainty and the calculation of derived, aggregated and savings results. In the context of the broader M&V process, documentation of the project - including providing evidence of the timing of upgrades, metering installations and sensor calibration, measurement of independent variables, and proof of continued service delivery - are central elements for auditing purposes.
1.2 Automated M&V: background & scope
Previous work (see [7] and associated references) has highlighted a key challenge of M&V which is using automation to reduce the human effort required to undertake an analysis. This is important to broaden participation in energy efficiency schemes by reducing the cost-benefit ratio. More broadly it can also unlock additional opportunities to better understand building energy use
changes over time, energy use in comparison to similar buildings, and energy use changes from portfolios or aggregations of buildings under different types of interventions.
This need to improve automation has driven focus toward the use of net electricity meter data from utility or ‘billing’ meters (i.e. NMI meters) for automated whole-of-site 1 analysis as opposed to direct metering of equipment that may be the specific focus of the upgrade. That is, using EVO terminology, to ‘Option C’ type M&V analysis. Use of NMI meters not only reduces the onus on the M&V practitioner to setup and validate metering equipment but, crucially, it also means that historical baselining data may be available from the commencement of the project. However, it does have the effect of limiting the scope to electrical energy consumption and not other forms of energy use such as gas and liquid fuels.
Several works in the literature report on approaches to improve automation of M&V algorithms. These include approaches to detect and handle non-routine events [8] [9] [10], methods of calculating model quality and quantifying uncertainty ranges [10] [11] [12] [13] [14], and methods of automating the end-to-end process [15]. Often an algorithm will work well on some but not all of the buildings in any given study. In general, we should not expect a streamlined, automated process to be able to capture the nuance and variability that exists across different sites, buildings, and equipment. The inclusion of additional model complexity or human-in-the-loop adjustability (to account for this variability), risks undermining the transparency of an automated approach that is so important for accountability. Hence part of the development scope was to understand the extent to which a relatively ‘one-size-fits-all’ type algorithm can be applied. As noted above creating an auditable reporting record is a central component of the M&V process but is not specifically the focus of the applications described here. This is considered to be a relatively straightforward additional step once a plan is established for adopting the automated M&V algorithms at scale.
1.3 Application design philosophy
The M&V algorithms and tools described here have been developed following several guiding principles. In general, they should:
use automated and data-driven processes that limit as much as possible any requirement for human or manual input,
favour simpler, more robust models including those that can be used to deliver additional insights in contrast to overly complex or obfuscated models,
favour models/methods that reliably deliver reasonable results as opposed to those with erratic or variable performance,
not require expert knowledge to run or to understand key outputs but include the ability to display or output more detailed results for expert users,
provide the framework for creating auditable records of the analysis & results,
1 To the extent that many NMI meters set up to measure the net electricity flow to an entire site. Some sites such as those with tenancies may have multiple NMI meters.
use methods that allow simple interfacing with other programs or workflows,
be constructed in a modular fashion to enable future additions and/or modifications, and
require minimal computational effort to perform an analysis.
1.4 Use cases
CSIRO’s core M&V algorithms are described in this report. The explanations provided are general to many possible use-cases (not just specific to the energy saving certificate calculation use case).
The algorithms (and its codification into the ‘MVApp’ stand-alone application (§2.4)) was originally developed for the following two primary use cases:
i) calculating energy savings from energy efficiency upgrades and retrofits (i.e. traditional M&V analysis), and
ii) continuous benchmarking to track changes in energy use at a site.
In addition to these cases, the M&V algorithm is also easily adapted to:
iii) making projections of future energy, cost and emissions for one or multiple sites, and tracking progress towards targets,
iv) providing real-time forecasts of site energy consumption,
v) providing fault detection and some diagnosis capability
vi) calculating flexible demand capacity,
vii) calculating financial settlement from delivery of flexible demand.
These additional use cases could be built upon the core baseline algorithms with relatively minimal effort. The Streamlined White Certificate application (‘SWCapp’) (§2.5) and Web portal (§2.3) are one example of a use case of the M&V algorithm. Many aspects of the M&V tools such as data I/O, data cleaning and semantic model processing also share many commonalities across different buildings applications.
2 Using the M&V tools
2.1 App eco-system
The M&V application is in fact not a single application but a family of separate pieces of software as shown in the simplified schematic in Figure 1. Originally this was due to the historical development process, but this layout has been maintained to facilitate different types of use cases as noted above.
Figure 1 Schematic of high-level applications architecture. Either MVApp or SWCApp can be called directly as standalone programs. Alternatively, the web portal can be used.
‘MVApp’ contains all of the core algorithms including data ingestion, processing, calculating baselines, evaluating savings, and generating outputs. MVApp is a stand-alone command-line program that can be run in isolation and whose operation is controlled by a single json formatted configuration file. It outputs results in a range of formats including spreadsheets, reports, figures, and direct data output to the cloud. MVApp operates directly on either raw data (for example csv’s) or Senaps/Data Clearing House (DCH) data-streams and does not utilise semantic building models.
‘SWCApp’ is also a separate stand-alone program whose operation is controlled by json formatted configuration files and that can be run in isolation. The role of SWCApp is to reformat the configuration files output by the DCH app hosting layer, to process the Brick 2 semantic model query response, to set additional configuration options that are not normally exposed to the user, to call MVApp, and finally to reprocess MVApp outputs so that they can be passed to the DCH app hosting layer, for example, for use by the Web portal
The DCH app hosting layer performs several functions in the DCH backend. This includes installing the M&V application on new sites, setting up NMI meter data streams, creating the SWCApp
2 Brick is an open-source schema that provides a standardised set of naming conventions that can be used to describe the physical, logical and virtual assets in buildings and the relationships between them. See https://brickschema.org/
configuration files, managing execution of the SWCApp code and communicating with the Webportal via an API. M&V users do not directly interact with the app hosting layer.
Users can run an M&V analysis using three different methods.
1. By creating an MVApp configuration file and running MVApp directly. This option is best when the specific data-streams are known, when running from local data files, or when non-standard configuration options need to be set. This includes running an analysis on sub-metered data.
2. By creating SWApp configuration files and running SWCApp directly. This option is best for doing batch runs over many sites coupled with using the semantic building models to determine data streams.
3. Using the Web Portal. This option is best for non-expert users or where a fully automated analysis is required. The Streamlined White Certification process is based around use of this web portal.
2.2 DCH app hosting layer
The DCH app hosting layer consists of components that orchestrate and report on the following resources in both DCH and the backing analysis service, Senaps. These are resources the user does not interact with directly.
• Data ingestion: The first step to automated application of the M&V tools is orchestrating tooling that can acquire the various meter data (streams) from external sources. This involves: i) setting up a ‘bucket’ or ‘datapool’ in DCH for the data to land in, ii) configuring a workflow in the Senaps analysis service that goes off and acquires the data and places it in the datapool, and then iii) classifying, or adding semantic data to each data-stream (point) according to its type (i.e., this is 3-phase energy data, or this is single phase current etc.)
• Site generation: The second step is to create a site in the DCH, which involves generating a simple semantic model of the site and linking up the data from the datapool. For the M&V application the minimum information required in the semantic model includes that a site exists with a certain NMI meter, the points associated with the meter and whether or not the points exist in the datapool.
• App installation: The final step is installing the White certificate application (SWCApp) on the Site in DCH. This uses the inbuilt app capability of DCH. The DCH app hosting layer 1) asks DCH to install the M&V app on the Site, and 2) asks DCH to run it.
The Web Portal integrates with the DCH app hosting layer. It can ask the app hosting layer to set up a site, and, then ask for the results of the baseline/analysis.
2.3 Using the web portal (Figure 1 A→G→D→B→A)
2.3.1 Using the portal
The web-portal is accessed through a browser by navigating to: https://dataclearinghouse.org/whitecertificate/dashboard
The description below is based on the MVP product as of May 2024. Given the web-portal is undergoing continuous development It is expected that some of these details may change.
After logging in, the main dashboard appears (Figure 2). This allows users to select the Organisation from a drop-down list of those they have access to, to add a new site, and to see existing sites that M&V has previously been installed on.
Selecting the ‘Add a new site’ button prompts the user to enter a site name, postcode, electricity retailer and NMI (National Metering Identifier) number. Future versions may incorporate other options such as the ability to enter multiple NMI’s for a single site.
Accepting the terms and conditions, initiates a sequence of steps in the DCH app hosting layer to create the site, find and ingest the NMI meter data and to configure and run the SWCApp on the DCH platform to generate a baseline. This process can take several minutes depending on the system load and other factors. The user is returned to the main dashboard and the new site appears almost instantly. An indicator is provided next to each site showing the status of the baseline creation and, if ready, whether the baseline meets all the eligibility criteria (see §3.4).
2.3.2 Understanding baseline model outputs
Baseline model quality is primarily visualised through a graph of the daily energy ‘delta’ as shown in Figure 3. This is the difference between the model estimated daily energy consumption and the actual daily energy consumption for the baseline period. The blue line plots the daily values and the solid black line the monthly average of the daily values.
Figure 2 Screenshot of the main buildings dashboard in the MVP Web Portal.
Daily values above the upper horizontal black dashed line indicate days where energy use is underestimated by a large amount compared to other days (i.e. actual consumption is higher than expected based on the model). Likewise daily values below the lower horizontal black dashed line are significantly over-estimated. These two sets of days are referred to collectively as outlier days
Outlier days during the baseline period may be considered for possible exclusions from the model fitting process if it is suspected that non-routine events occurred on those days. For example, if the site was shutdown on a particular day this is likely to appear as a value below the lower horizontal dashed line. If this is not something that would normally occur, it may be eligible for exclusion (see §2.4.3.11).
The monthly average daily energy delta is shown by the black stepped line. Entire months that are under or over-estimated by the model are indicated by those months that are above/below the horizontal red dash-dotted lines. Exclusion of individual outlier days within a month may be sufficient to cause an entire month to fall back within these bounds when the model is re-run. However, in general, under and overestimated months indicate that the model is failing to capture a consistent trend in the data. If the amount under/over is relatively small, and only occurs for a few months, then this is unlikely to be significant problem. However, if there is a clear trend of under/over-estimated months at certain times of the year or over time, or if the values are far outside the bounds for a given month, then the model may fail to meet the eligibility criteria (see §3.4).
Figure 3 Screenshot of the baseline model eligibility window in the Web Portal.
Additional values provided on the dashboard are i) baseline energy use, ii) minimum claimable savings, and iii) model quality.
Baseline energy use is the total measured energy consumption over the baseline period but including extrapolations for any missing or excluded days (these default settings can be changed in MVApp; see §3.6.1).
Minimum claimable savings is the estimated lowest measured percentage savings amount that could be credited with a non-zero savings given the accuracy of the fitted baseline model. An example is shown in Figure 4. Here the minimum (non-zero) claimable saving is 3.9% and a measured energy saving of 4% would be eligible to claim as a saving of approximately 0.8% (black line). The actual energy saving (red circle) was 11.6% which can only be claimed as a saving of 4.6% due to the inaccuracy of the model. Note the step changes in the claimable savings line arise from the Accuracy Factor as specified in the NSW Energy Savings Scheme Rule [16] and given below in Table 16
Figure 4 Visualisation of the difference between measured (or targeted) energy savings and claimable energy savings. Minimum claimable savings is defined as the intercept of the claimable savings line (black line) with the vertical axis. Here minimum claimable savings is 3.9%, measured or estimated savings is 11.6% and actual claimable savings is 4.6%.
Model quality is a non-numerical field with values of ‘fail’, ‘poor’ and ‘good’. A value of ‘fail’ indicates that the algorithm has not successfully generated a baseline. A detailed error log is available for further diagnosis. A value of ‘good’ indicates that a baseline model has been created and has passed all of the model criteria. A value of ‘poor’ indicates that a baseline has been successfully generated but that one or more of the baseline model criteria (see §3.4) has not been met. No distinction is made between a model that fails many criteria or fails the criteria by a large amount, and a model that only just fails to meet one criterion. Hence, a model denoted as ‘poor’ could be, for the most part, actually quite a good model. Hence, it is important to view and consider the warning messages returned. The section below provides further guidance.
2.3.3 What to do when model quality = poor
As discussed in the introduction, the Web portal-based M&V use case scenario is designed to be as streamlined as possible. This means that there are minimum parameters exposed to the user and hence limited ability to modify the default modelling approach.
It is expected that the automated M&V analysis will lead to a good quality baseline for fewer than half of all commercial sites (see the Case Study section for further details). There are multiple reasons why the baseline model quality can be reported as poor. For example, the site energy consumption may be trending over time (referred to as non-stationary), there may be large gaps in the data, or there may be large step changes in consumption from one month to the next. In some cases, the variability of energy consumption is simply not sufficiently explainable by the variables included in the model (i.e. there may be key variables such as production or occupancy parameters the have a strong influence on energy use). In other cases, it may not be explainable by any plausible variables (i.e. is strongly influenced by behavioural factors). In some instances, there may be little option but to consult an M&V professional to perform a manual assessment
A model that is reported to be ‘poor’ has failed one or more of the eligibility criteria (see §3.4). No distinction is made between a model that just fails one criterion (for example there is too much missing data in 1 month) as compared to a model that fails multiple criteria by large amounts. This can be assessed by inspecting the information messages that are returned.
Inspection of the daily energy delta figure (Figure 3) together with the model information messages can provide further insight. A list of common messages and a description of the cause and potential solutions is provided in Table 1
Message
A trend of increase/ decreasing energy use with time is identified
A seasonal trend is identified
Characteristic & meaning Cause & possible solution
Energy consumption is trending over time.
Monthly mean delta is consistently above /below dashed lines towards the start or end of the baseline period
Energy consumption is higher in some seasons
Could indicate a site where occupancy/production are gradually changing or where new generation is gradually coming online. A baseline cannot be established if consumption is trending. The only option is to recalculate the baseline when the site operation stabilises.
Could indicate additional equipment being activated over specific periods and that is not accounted for by the ambient temperature independent variable. An automated baseline may not be feasible.
The model is a poor fit to the data (both CVRMSE and NMBE criteria fail).
The model is reported to not capture the trends in the data (R2 or CVRMSE criteria fail)
The model is very poor & is unlikely to be usable.
The model is poor & is unlikely to be usable.
Consumption is influenced by additional variables not considered by the automated algorithm. Consult with an M&V professional.
Consumption is influenced by additional variables not considered by the automated algorithm. Consult with an M&V professional.
Table 1 Summary of baseline model warnings displayed by the Web Portal along with potential user actions.*
The model is reported to capture (some) trends but consistently under/over predicts (NMBE fails)
Model residuals are strongly auto-correlated
Energy use is consistently lower/ higher than expected N months
Model errors are biased toward under/over prediction.
Consider re-baselining using a model with a coarser time resolution such as a daily model.
Excessive missing data
Excessive excluded data
2.3.4
The model error on any given day is similar to the error on the previous/next day which could cause uncertainty to be under-estimated.
There are whole months where consumption is above/below the prediction using the model fit to all months.
Too many days in the baseline period have failed the missing data criteria for 1 or more months.
The combination of missing and excluded data exceeds the allowed amount for 1 or more months.
Estimated savings may still be usable although associated uncertainty ranges may be too narrow. Re-running the analysis using a coarser time resolution (for example a daily analysis model) is recommended.
Consider whether there are any unaccounted for nonroutine events and if so, add the relevant dates to the excluded period & re-run the baseline. Larger exceedances or more exceeded months indicates that the model is not capturing the underlying variability in the data.
There is a problem with one or more of the input data sources. Inspect the baseline data to check which periods are missing. Consider modifying baseline start/end dates. Re-run the baseline once the missing data issue is resolved.
Reduce the number of days excluded from the model for the identified month(s) and/or check whether the missing data criteria is also failed for those months.
* CVRMSE - Coefficient of the Variation of the Root Mean Square Error, NMBE - normalized mean bias error, R2 – coefficient of determination
Post-retrofit analysis outputs
The Web Portal does not currently include a post-retrofit analysis dashboard This section will be updated at a later date.
2.4 Running MVApp directly (Figure 1 E→G)
As described in §2.1, an M&V analysis can also be conducted by calling MVApp directly, bypassing the web-portal and SWCapp. This is useful for implementing non-default settings, for performing batch analysis, or for running using local data.
2.4.1
Installation & versioning
To run MVApp directly it is necessary to install it on a local machine. A Windows self-installation file is supplied. A Linux installer is available upon request. The installer should be run with administrator privileges. Releases are numbered according to the pattern A.B.CCC. A is the major release number. B is the minor release number. CCC are updates incorporating minor bug fixes and improvements. It is recommended to remove previous versions prior to installing a newer version.
2.4.2 Running
MVApp is a command line program. Once installed It can be run from a command prompt by navigating to the folder with the executable (on Windows, default: C:\Program Files\CSIRO\MVapp\) and typing the name of the executable followed by the full path to the configuration file. For example: “mvapp.exe C:\myconfigurationfile.json”. If the configuration file argument is not supplied, MVapp will attempt to read the configuration file “default_config.json” from the executable directory.
Optional second and third input arguments can be supplied. The second argument is the path where outputs will be saved, the third argument is the filename that can be used to override the default logging file location (for example when MVApp is called from another program such as SWCApp.)
User adjustable parameters are supplied via a json encoded configuration file. This configuration file is discussed in the following section. A full listing of available parameters is given in Table 34 in the appendix.
The installer includes a file ‘batchrun.bat’ which, if called from the command line, will sequentially run an MV analysis using all of the config files found in the sub-directory “\configfiles” located in the installation directory. The resulting output files are saved to the standard output directory (see §2.4.5).
2.4.3 Setting up a basic configuration file
Upon execution, MVApp reads a json configuration file that must include certain required fields and can also include other optional fields. The name of the file is passed as an argument upon execution. However, if no filename is supplied, MVApp attempts to read the file named ‘default_config.json’ located in the application directory.
The json format is a convenient human readable text-based format that can be created in standard text editors, though specific json readers make viewing and editing easier. It consists of key – value pairs structured in a hierarchy 3. An example MVApp input file is shown in Figure 5.
3 https://www.json.org/json-en.html
The field casename specifies a name that is used for writing out data files and Senaps output streams. Note: if any Senaps streams or output files using this casename already exist, then they will be overwritten. Hence care must be taken when specifying casename
A Senaps API key parameters apikey must be provided (see §2.4.3.1) if data is to be sourced from Senaps. The parameter datasource = ‘senaps’ specifies Senaps as the source of data. Alternatively, the program can read raw data from a file (see below).
The field selected_streams is also required and is used to specify which Senaps data streams to use for specific ‘classes’ or input variables. Further details are given in §2.4.3.2.
In this example, two parameters ‘site_lat’, ‘site_lon’ as well as the analysis and baseline period start and end dates have also been specified
The code performs checks on the input parameter variable type and the parameter values. Values that are outside of the allowed bounds are adjusted and a warning message is printed in the log file. Default parameters are assumed for all fields that are not specified in the configuration file. A listing of all parameters and associated default values is given in Table 34
2.4.3.1
Data sources
MVApp can read data directly from Senaps (default) or from a local data file (csv or mat file).
Reading from Senaps uses the ‘SensorDataAPI’ to read data from streams and, if the optional parameter writedata_to_senaps = true, it also uses the ‘SensorDataAPI’ and the ‘AnalysisAPI’ to create and write output streams back to Senaps. Both API’s use an alpha-numeric key that must be specified in the configuration file via the field apikey. Apikey’s can be generated using the Senaps web portal. Vector streams for outputting processed data and document nodes for model data are created under the Senaps Organisation and Group specified via the fields senaps_organisation and senaps_group. Users must ensure that the Senaps account linked to the specified apikey has the required permissions to read/write and create streams. Note: MVApp overwrites data if there are existing data-streams with the same casename
MVapp can also be set to read raw data from a local .mat or .csv file, bypassing Senaps. This may be useful for testing purposes or to run off-line using previously downloaded data. To enable
Figure 5 Example MVApp configuration json file.
reading from a local file, set datasource equal to the file name. For example, an MVapp output result file from a previous run (see §2.4.5.6) also contains the raw data and can be used as input data by setting datasource = “results_uid.mat” where uid is the unique id of a previous running instance.
If using a .mat file, it must contain a Nx1 cell array variable named ‘rawdata’ with timetables corresponding to the N data-streams specified in the config file. It is important to note that all settings in the config file are retained so that the list of streams in the config file must be a one-toone (ordered) match to those in the data file. In addition, the time range of the raw data must include the baseline and analysis periods.
If using a .csv file, then the data must be formatted as pairs of columns containing Timestamp and Value in the order specified in the selected_streams parameter. That is, if 3 data-streams are specified, then the csv should contain 6 columns as shown in Figure 6. Note that the lengths (number of time-data values) for each stream can be different. Times should be specified in the format: dd/MM/yyyy HH:mm
Figure 6 Sample data formatting for csv file input
To switch back to reading data from Senaps, set datasource = “senaps".
2.4.3.2
Stream classes
Because MVApp was developed prior to the development and implementation of Brick Semantic models in the DCH, it defines its own set of stream ‘classes’ that correspond to specific quantities related to a site or building. For example, the stream class ‘tamb’ corresponds to the ambient air temperature at the site. This class has a specific set of default data process options, default and allowable units, and a specific use case in the application. When specifying input streams in the configuration file, the user must assign each stream as one of the available stream classes as listed in Table 28. The electricity metering stream classes are related via the configuration shown in Figure 7.
7 Schematic relationship between electricity metering streams. **Note: consumption is not a stream class.
Data streams are passed via the parameters structure array selected_streams in the configuration file. Each stream must have class and stream_id fields specified as a minimum. Class defines the specific quantity that the data-stream corresponds to and must be set as one of the available stream classes (see Table 28). Stream_id is the full stream name (id) of the data stream in Senaps. The optional fields type, units, primarystream and a structure options can be set. These fields relate to how the data stream is processed and are described further below.
The stream class net is the only stream class that must be supplied (note though that other streams may be required for certain parts of the code to function).
If there is no onsite generation, then the M&V analysis will be applied to this net stream (or the sum of net streams if multiple are supplied). Internally, the aggregate site consumption is stored in a variable called ‘consumption’. Note: because consumption is not a stream class, it cannot be directly input.
In the general case where onsite generation is present, site consumption is calculated using:
Here net is power/energy flowing via a connection from the site to the grid. A positive value indicates energy purchased from the grid, a negative value net export of power when onsite generation is greater than onsite consumption. Often there is only 1 net meter, though multiple net meters can be specified.
Note: NEM12/13 metering data often includes ‘consumption’ and ‘generation’ data variables. In general, these are not actually consumption and generation. Instead, they are import and export which can only be used to determine net (i.e. net = import – export). If onsite generation is present, then in the general case NEM12/13 derived data is not sufficient to perform an M&V analysis. Methods to handle PV generation in some cases are discussed below.
Here netG is the net power/energy generated or consumed through onsite generation or through controlled battery charging. A positive value indicates energy that is generated or supplied (such as output from photovoltaic panels). A negative value indicates energy that is consumed, for example during charging of batteries.
Depending on the aims of the analysis, it may be appropriate to treat EV charging as part of the underlying site consumption, for example if the charging is uncoordinated or there are no ‘smart’
Figure
control algorithms in place. In this instance, EV charging should be treated as part of consumption (i.e. included within net but not netG) as shown in Figure 7.
Net generation power/energy must be added to net grid power/energy to calculate the site underlying consumption. The site consumption is treated as missing (i.e. NaN) for any given data point if, and only if, all of the (processed) net stream values are missing for that data point (see §3.1 for more details on how the data is processed).
As indicated in the note above, for some sites only net data may be available. In general, an M&V analysis cannot be run for these sites if there is also onsite generation because consumption cannot be derived. However, for the special case where the only onsite generation present is photovoltaic generation, the optional flag use_net_data can be set. This enables additional terms in the modelling to attempt to fit a model for the pv generation so that consumption can still be evaluated. This flag is automatically set to true if the presence of PV generation is detected in the consumption data (§3.4).
The resulting analysis is likely to be less accurate than using supplied generation data, and may fail if the generation amount is greater than 20 to 30% of the consumption (see §3.8.1 for a case study example). Note, this does not apply to sites with no generation – for these sites set use_net_data=false.
2.4.3.3 Handling of multiple streams of the same class
Some stream classes permit multiple data streams to be assigned the same class. Different stream classes are combined in different ways as indicated in Table 28. For example, multiple netG data streams can be supplied (for example corresponding to individual pv arrays or generators), in which case they are summed together. Note that, by default, for any given data point the sum of netG streams is treated as missing only if all netG streams are missing for that data point. Multiple ambient temperature classes are averaged.
When combining multiple streams two different approaches for handling missing data are used. For a given stream, if the field primarystream = true, then for a given time interval, the aggregated value for that class is set equal to NaN (missing) if the value for that stream is missing. If primarystream = false (default) then the aggregated value for that class is only set to nan if the values for each stream of the same class are also nan.
2.4.3.4 Stream types, units & custom data processing option
The optional field type, tells the application to apply a specific set of data processing options to the stream as well as the default unit and the allowed units. Each class has a corresponding default type associated with it as given in Table 28 in the appendix. However, a user can choose to assign any type to a given input stream. The data processing options corresponding to specific types are listed in Table 30. In most cases, the default type will be appropriate.
Units are based on the QUDT ontology [https://www.qudt.org/2.1/catalog/qudt-catalog.html]. Limited unit conversions are available as indicated in Table 32
Individual data processing options can also be set for an individual input stream using the options structure. Available options are listed in Table 31 in the appendix.
2.4.3.5
Default inputs streams
The inputs stream classes tamb, irrad, spotprice and gridemissions have in-built default streams that are used if no user supplied stream is specified. The default stream id’s are listed in Table 29 in the appendix
The site location (site_lat, site_lon) is used to automatically determine the nearest Bureau of Meteorology weather location based on the lookup table (see the appendix). (These coordinates are also used to determine the time-zone if no time-zone is supplied.)
The nearest BOM station identifier is used to index the appropriate column in the BOM vector input streams for tamb and irrad. It is possible to manually override the chosen location by setting the value of bom_station_id equal to the ‘id’ of the chosen weather station.
The site location is also used to determine the appropriate network region (state) for setting the default spotprice and gridemissions data streams. For input coordinates outside the National Electricity Market states, MVApp will default to using the NSW regional reference electricity price and grid emissions intensity factor.
Note that the weather station is also used to select the fallback in-built temperature data to use in the unlikely event that both BOM temperature data and any user supplied temperature data streams are empty.
2.4.3.6
Electricity tariffs
Electricity tariffs are specified as an array of individual tariff structures tariff_data. The individual tariffs are aggregated and contribute towards the overall electricity cost (in addition to wholesale costs and carbon costs if applicable). Each tariff structure must have the parameter chargetype set as one of the values in Table 2. Optionally, time of use parameters daytype, time and month may be set for certain charge types. The parameter value specifies the numerical value of a specific charge. Optionally a description field can also be used to add a text-based description of the charge.
Electricity tariffs are included in cost calculations only if the flag network_price_calc= true
The overall electricity charge over any given data is the sum of all the individual tariffs that apply. Hence it is possible to specify charges that overlap in time. Because of this, is it suggested that monthly energy costs calculated by MVApp be cross-checked, for example with available electricity bills to ensure tariff values have been entered correctly.
Fixed (monthly) charges are applied to a given energy data timeseries as a fractional amount, proportional to the overall time span of the time-series. That is, missing energy data between the start and end of the timeseries does not affect the calculated fixed charges.
Electricity variable charges (Energy charge, Demand charge & Capacity charge and Feed in) are applied after the data has been averaged across half-hourly intervals. Since reactive power timeseries are not available, demand and capacity charges are based on real power import from the grid (i.e. assuming power factor of 1). Energy charges are based on real power net import and feed in charges real power net export. Remaining missing data (after processing and interpolation) within a given timeseries may lead to lower calculated energy charges (but not necessarily lower demand or capacity charges) depending on parameters extrapolate_missing_days and extrapolate_whole_months (see §3.6).
Demand and capacity charges are applied to the highest net import real power over half-hour periods within the specified demand charging window over the current month (demand charges) or the 12 months prior to the current day (capacity charges). Like the fixed charges, these charges are scaled in proportion to the overall time-span of the time series. The parameter mincapcharge can be used to specify the minimum capacity charge power level to apply.
Note: if the parameter extrapolate_whole_months = true (see Table 34), then reported monthly electricity charges for part-months (i.e. where the start/end of the baseline or analysis period falls part-way through the month) are estimated for the entire month.
Note: all tariff values (including feed in values) should be specified as positive values.
Table 2 Summary of allowed ‘chargetype’ values for tariff specification.
Chargetype
Fixed $/month Electricity charges that aren’t time-dependent
Energy charge c/kWh Electricity charges that are proportional to electricity net import either with/without a time of use dependence.
Demand charge $/kVA/month Electricity charge proportional to the highest power import over a 30 min period within the specified time window over the current month.
Capacity charge $/kVA/month Electricity charge proportional to the highest power import over a 30 min period within the specified time window in the 12 months prior to the current day.
Feed in c/kWh Electricity credit paid to net electricity export with/without a time of use dependence.
Daytype, month, time
Daytype, month, time
Charges that apply to certain day types, to certain months and/or to certain hours of the day can be defined using the daytype, time and month parameters. Daytype must be one of the values listed in Table 3. If daytype is omitted the charge is applied to all day types.
The month parameter must be entered as an integer array of months to which the charge should be applied (i.e. for January and March, set month = [1, 3].) Charges are applied to whole months. If month is omitted then the charge is applied to all months.
The time parameter must be entered as an integer array of hours (in 24 hr time) to which the charge should apply. Charges are applied to the whole hour proceeding the specified value. For example, to apply a charge from 2pm to 3pm, set hour = 14. To apply a charge from 2pm to 8pm, set hour = [14,15,16,17,18,19]. If time is omitted then the charge is applied to all hours of the day. All times are in the timezone specified by the timezone parameter (which defaults to the timezone corresponding to the nearest standard meridian if no time-zone is specified). Note, energy values are ‘right-edged’. That is, an energy value with a timestamp of 14:00, is assumed to correspond to the energy use from 13:30 until 14:00. Hence, this energy use would not be included in a tariff applied to hour =14.
Table 3 Summary of allowed ‘daytype’ values for tariff specification.
Daytype Description
all Charge applies to all days of the week (note this is equivalent to omitting the daytype parameter entirely).
workdays Charge applies M-F when it’s not a public holiday
nonworkdays Charge applies Saturday Sundays and public holidays
weekdays Charge applies only to Saturdays and Sundays
weekends Charge applies Monday to Friday
businessdays Charge applies to all days that aren’t public holidays
nonbusinessdays Charge applies only to public holidays
Note: Day types are case sensitive and must be specified as lower case.
2.4.3.7 Wholesale (spot price) costs
The cost of purchasing energy on the wholesale spot market can be included by setting the flag wholesale_price_calc = true. If the flag export_earn = true, then net power exported to the grid is assumed to earn at the same price (see §3.6.2 for further details).
2.4.3.8 Carbon cost & emissions
Carbon emissions savings are calculated if the flag emissions_calc = true. By default, timedependent emissions factors read from the gridemissions stream class are used. If the flag use_fixed_emissions_factor =true, then a fixed emissions factor specified via the parameter default_emissions is used instead.
Carbon costs are included along with emissions based on a fixed carbon price specified via the parameter carbon_price. If the parameter carbon_credit = true, then exported power earns at a rate equal to the avoided emissions multiplied by the carbon price. Carbon emissions calculations can be included while excluding carbon costs by setting a zero carbon price. For further details on the calculation of carbon related quantities see §3.6.2.
2.4.3.9 Occupancy status
The interval baseline model includes coefficients related to a binary occupancy independent variable. By default, a binary occupancy flag is estimated for each interval of the data using the algorithm described in §3.3.1.
An alternative to the automatic estimation model is to supply the binary occupancy status either as a separate timeseries via the occ stream class, or alternatively, to specify an explicit occupancy schedule. If occupancy data is supplied as a time-series, then it will automatically be used. To use an explicit occupancy schedule, set estimate_occ = false and provide the schedule(s) using the occupancy_schedule parameter as per below.
Specification of an explicit schedule follows the same approach as for electricity tariffs. That is, specify combinations of time periods using the daytype, time and month fields along with a value
of either 0 (un-occupied) or 1 (occupied). For example, to specify a 2pm-5pm M-F workday occupancy in March only, use:
Note, occupancy information is not used by the daily, weekly, or monthly models.
2.4.3.10 Public holidays
By default, MVApp reads Australian public holiday data from an outline resource: https://data.gov.au/data/dataset/australian-holidays-machine-readabledataset/resource/33673aca-0857-42e5-b8f0-9981b4755686
The online data is added to a static, internally stored Australian public holiday data-set covering the period 2010-2024 (inclusive).
Public holidays for a given site are selected based on the identified state (determined using site_lat and site_lon or postcode). To disable use of Australian public holidays, set use_aus_holiday = false User specified holidays can be added to the list of holiday dates by specifying a list of dates via the parameter extra_holiday_dates. All holidays are applied as whole days.
2.4.3.11 Excluded dates
Whole days of data can be entirely excluded from the baseline, analysis and/or forecast periods by specifying a list of dates via the parameter exclude_dates. These days are not used for fitting or evaluating the models, or for calculating savings. Note though that if the parameter extrapolate_missing_days = true, then calculated total quantities are scaled up to account for both missing data days and excluded days.
2.4.4 Advanced options
2.4.4.1 Forecasting future savings
Although primarily developed to estimate actual savings during the post-retrofit period using the model fit using the baseline period, MVApp can also be used to calculate normalised or forecast year savings. Note though that this capability is under development and may require inspection of the detailed results file to interpret outputs.
To enable calculation of normalised year savings, set the parameter forecast_period_savings= true. When enabled, an operating energy model is fit across the analysis period (in addition to the baseline model fitting over the baseline period) and both the baseline and operating energy models are used to estimate energy use over the forecast period. The difference between the
baseline model estimated energy use over the forecast period and the operating energy use over the forecast period is the normalised year savings.
The forecast period start and end dates are specified using the parameters forecast_start and forecast_end. Data for the forecast period must be contained within the same data-streams as for the baseline and operating periods with the exception of ambient temperature which may (optionally) be supplied using a separate stream class forecasttamb
2.4.4.2 Demand response calculation mode
MVApp can be set to run in a demand response analysis mode by setting analysis_mode = DR. (Set analysis_mode = MV for standard M&V analysis mode). In demand response mode the analysis period is entirely coincident with the baseline period (analysis period start and end dates are automatically set equal to baseline period start and end dates). The baseline model is then fit over the baseline period but only for intervals where the value of drflag stream class is equal to zero. The baseline model is then used to estimate energy use for intervals where the drflag is non-zero. Additional outputs are computed, details of which can be obtained by contacting the author. Note also, this capability is under development and may require inspection of the detailed results file to interpret outputs.
2.4.4.3 Simulating interventions
For testing purposes MVApp includes functions to simulate one or more energy saving interventions. These are used for development and research purposes only, for example to assess the uncertainty estimate algorithms. Simulated interventions are enabled using the flag enable_interventions. Multiple additional parameters (not described here) are used to specify details of the simulated interventions.
Simulated interventions apply weighting factors to scale the energy data over specified time ranges. Multiple types of weighting factors can be applied for any given intervention, and these can be applied using different methods such as time of year weighting, day of week, time of day and temperature. For example, a weighting factor could be applied to decrease energy more on workdays over summer and with a weighting inversely proportional to temperature. Multiple weighting factors are scaled such that they deliver a specified mean and standard deviation of percentage saving for the intervention over the specified date range. An additional function allows pre-scaling of the data-set to eliminate any pre-existing mean differences between the baseline and analysis period consumption by re-shuffling the data sets by shifting whole days of all data variables while preserving day of the week values. Further details on using simulated interventions can be obtained by contacting the author.
2.4.4.4 Adding additional independent variables
An arbitrary number of additional independent variables can be included in the baseline model formulation as given in §3.3. To include an additional independent variable, specify a data stream of class ‘indepvar’ and the associated stream properties, and set the parameter include_indepvar = true. Note: a given independent variable may or may not be retained in the final model formulation depending on its predictive power.
2.4.5 Outputs
On Windows output files are saved to: “C:\Users\userid\Documents\MVApp\” where “userid” is the windows user name under which the application is installed.
On Linux output files are saved to: “~\MVApp”.
2.4.5.1 Word® report
On Windows computers with Microsoft Word installed, MVapp can auto-generate a summary report in Word® format. Creation of this report is enabled by default but can be disabled by setting the parameter prepare_word_report = false. The file is saved as “MVreport_casename.doc” in the output directory. The report includes summary statistics, listing of parameters, selected data tables and the figures listed in Section 2.4.5.4 with text-based descriptions and limited automated interpretation of the results. A listing of the program log file is also included.
2.4.5.2 Json outputs
Selected outputs are saved to json format in 3 different files:
• “casename_baseline_summary.json” contains information related to the fitted baseline model.
• “casename_analysis_summary.json” contains information related to estimated savings from an model.
• “casename_detailed_results.json” contains selected daily and monthly results.
See Appendix A.9 for a listing of the parameters in these files.
2.4.5.3 Baseline model file
The fitted baseline model is saved to a “casename_model.mat”. This file is not intended to be used directly by a user but can be used to run an analysis using a previously computed baseline model by setting the parameter ‘baseline_filename’ to this filename and setting run_baseline = false
2.4.5.4 Figures
All figures included in the Word® report are also saved as individual .emf (Windows) or .jpg (Linux) files. Figure files are saved even if the prepare_word_report is set to false. Files are saved and prefixed with the casename. For example, “casename_consumption.emf”. A listing of these files is given in Table 35 in the appendix.
2.4.5.5 Excel®spreadsheet
On Windows computers with Microsoft Excel, selected daily and monthly quantities are output to an Excel® spreadsheet with filename “MVresults_casename.xlsx” where casename is set in the configuration file. Output quantities are listed in Table 37 in the appendix.
2.4.5.6 Matlab® raw data file
Raw and processed data timeseries, parameter values, logging data and results are saved to a Matlab® specific data file with mat format named “results_uid.mat” where uid is the unique identifier for the particular running instance of MVapp. This file is used internally by MVapp to
produce other outputs such as the Word report and Excel spreadsheet and is not intended to be used by most users. However, as noted above, it is possible to use this file to supply raw data to MVapp when a connection to Senaps is not available or to ensure exactly the same data is used.
Most variables stored in this file conform to the naming schema: “frequency_period_quantity_type” with values as listed in Table 4
Table 4 Variable naming convention
2.4.5.7
Log file
Logging information including information, warnings and error messages are written to the file “log_uid.txt”. All messages have the following format: “Timestamp flag message string” as described in Table 36. Logging information is also printed to the Word® report.
2.5 Running SWCApp directly (Figure 1 C→G→D)
As described in §2.1, an M&V analysis can also be conducted by calling SWCApp directly, bypassing the web-portal. SWCApp in turn calls MVApp. SWCApp is also a command line program. The main reason for running SWCApp as opposed to MVApp is to use a Brick query against a building semantic model to automatically identify and set data-streams, and associated parameters. SWCApp was developed primarily to interface between the Web portal and MVApp.
2.5.1 Installation & versioning
To run SWCApp directly it is necessary to install it on a local machine. SWCApp is supplied with a Windows self-installation file. A Linux installer is available upon request. The installer should be run with administrator privileges. Releases are numbered according to the pattern A.B.CCC. A is the major release number. B is the minor release number. CCC are updates incorporating minor bug fixes and improvements. It is recommended to remove previous versions prior to installing a newer version.
2.5.2 Running
SWCApp is a command line program. Once installed it can be run from a command prompt by navigating to the folder with the executable (on Windows, default: C:\Program Files\CSIRO\SWCapp\) and typing the name of the executable followed by a Senaps API key. For example: “swcapp.exe mysenapsapikey”. Optionally two arguments can be supplied. In this case, the first argument is the parameter senapsurl and the second argument is apikey.
SWCApp reads inputs from pre-configured json formatted ‘port’ files. On Windows these port files must be located in the executable directory. On Linux the port files must be located in a sub-folder ‘/runtime_input_ports’. Required files are listed in Table 5.
Table 5 Listing of required input port (json) files for SWCApp
Contains the Brick sematic model query response. This is processed by SWCApp to determine selected_streams fields. The query response is what is returned from running a BRIQL query against a target building model.
Any MVApp parameters not listed in the table above are fixed by SWCapp and not user configurable.
In addition to the above files, SWCApp optionally reads an additional file “baseline_model.port” that is also a file output by the application (see the following section). This file is read if run_baseline = false. This file is encoded in base64 format when output by SWCApp and so is not human-readable. This file is read directly from the output file path (see below).
2.5.3 Querying semantic models
BRIQL (BRIck Query Language) is a domain specific query language for querying semantic models of buildings that use the BRICK modelling ontology. The industry standard query language for querying models built against ontologies like BRICK is SPARQL. Writing SPARQL queries is a skill that relatively few software engineers possess and has a significant learning curve. BRIQL was created to reduce that learning curve. It is JSON based and includes a vocabulary familiar to
buildings domain software engineers. Writing and running generic BRIQL queries is beyond the scope of this guide. However, an example M&V query file is provided in the appendix. This query must be run and the resultant query response is what must be supplied to SWCApp as the ‘meter_points.port’ file. Separate code to read the example query file, submit the query and generate the ‘meter_points.port’ file is available upon request.
2.5.4 Output files
Six output port files and a further 7 figure files are output by SWCApp. These files are mapped from the MVApp outputs as described in Table 6.
On Windows output files are saved to: “C:\Users\userid\Documents\MVApp\” where “userid” is the windows user name under which the application is installed. On Linux output files are saved to: “~\MVApp\runtime_output_ports”.
baseline_model.port
baseline_result_summary.port
Analysis_result_summary.port
Detailed_results.port
Mvapp_log.port
Matresults.port
Figure_01.port -> Figure_07.port
Casename_baseline_model.mat
Renamed from casename_baseline_summary.json
Renamed from casename_analysis_summary.json
Renamed from casename_detailed_results.json
Renamed from log_uid.txt
Renamed from results_uid.mat
Renamed from hrlyfit, monthlyenergy, occupancy, consumption2, modeldiagnostics, hourtrend and temperaturetrend figures.
In addition, SWCApp writes its own logging file: log_uid.txt file to the output file directory. Here the uid is unique to the SWCApp and is a different uid to that which is used by MVApp.
Table 6 Summary of SWCApp output files
Filename
Mapping from MVApp output file
3 M&V algorithm overview
This section is intended to provide an overview of the key elements of the M&V algorithm rather than an exhaustive description of all aspects of the code. For further details on any aspects please contact the author.
An overview of the internal M&V algorithm stages is shown in Figure 8. A significant amount of code is dedicated to the initialisation and data-preparation stages prior to running the actual M&V analysis (i.e. fitting the baseline model and estimating analysis period consumption). This includes reading and cross-checking configuration parameters, setting and calculating default options and derived parameters, cleaning and processing individual data streams, calculating derived data streams, and performing checks on the processed data. The data cleaning steps are described in §3.1 Data checks are described below in §3.2. The core M&V algorithms are outlined in §3.3. Section 3.4 describes the baseline model criteria and §3.5 the tests conducted on computed savings. The calculation of cost, energy and emissions savings as well uncertainty is summarised in §3.6.
3.1 Data cleaning & processing
Data processing & cleaning functions are an important step. These are automatically applied to individual raw data streams in the following order:
1. Sort data in time order and add a data-point at the current time
2. Remove duplicate data points
3. Process upper and lower outliers
4. Apply scaling and offset factors & unit conversions
5. Synchronize points to the global timestep dt_global (if not cumulative data)
6. Interpolate gaps & forward fill
7. Differentiate cumulative data
8. Synchronize points to the global timestep dt_global (cumulative data)
Figure 8 Overview of MVApp algorithm key stages
Fetch
Different data processing operations are applied (by default) to different stream classes as described in §2.4.3.4. These defaults can also be overwritten.
In addition to the processing of individual raw data streams, a final data processing step is applied to the aggregated ‘net’ stream class (i.e. after all streams with the ‘net’ classes have been combined). This involves running an additional dynamic outlier detection algorithm to detect large outliers that are missed by the individual stream detection Outliers are removed, and because consumption is calculated from net, these values will also be removed from the consumption data.
3.2 Data checks
A series of data checks are applied to the processed data streams. Processed data streams have had cleaning, scaling, missing data and unit conversion processes already applied. Raw data statistics calculated prior to data processing and checking are also output (see Table 33 in the appendix).
Data checks (Table 7) are applied separately to the baseline and analysis data-sets. Each data check can trigger a warning or error. If ignore_dataerror = false, then data errors trigger the program execution to stop. Some checks are performed only on certain stream classes and for some checks threshold levels varying according to the stream class of the data being checked. Note that the PV check applied here is different to baseline model PV check (see §3.4).
Table 7 List of data checks and association threshold levels
1 Empty data true [error] Checks for completely empty data
2 Timezone >2 hr deviation [error] Compares set-timezone to an approximate value based on the nearest standard meridian
2b Timezone (alternative) Min >10 or <2 or max >16 or <11 [warn] (tamb class only)
Finds the approximate time-of day (24hr time) of the mean daily maximum & minute temperatures.
3 NaN count >10% [warn], >30% [error] Percentage of NaN values
4 Initial/ending missing period >1 day [warn] Number of continuous days of NaN values at start or end of the dataset
6 Outlier checks Dependent on the data-class
Data values above/below upper/lower soft limits trigger warnings. Exceeding hard limits triggers an error.
7 Periodicity >±0.3 days [error] Uses continuous wavelet transform (Morse wavelet in frequency domain) to find maximum of the spectrum at each time-step (within the cone of influence) for continuous non-Nan data segments. The median peak frequency is evaluated and compared to a daily frequency.
8 Zero night-time >0 [error] (pvgen class only)
Computes sunrise/sunset times given location. Non-zero values more than 20 minutes before sunrise or after sunset trigger are counted.
9 Range checks < 1/20th of the range [warn]
10 Non-zero integers >70% [warn]
11 Unique values <10% [warn]
12 Non-zero duplicate sequences >10% [warn] >20% [error] 4 or more
13 Zeroes >10% [warn], >60% [error] (excluding netG and pvgen)
14 Monotonicity True [error]
15 Linear trend >0.9 [error]
16 Auto-correlation <0.5 [warn]
17 PV generation in consumption data (simple)
<0.05 [warn]
Compares the interval containing 95% of values with the range defined by soft limits.
Percentage of non-zero values that are integers
Percentage of non-zero values that are unique to floating point tolerance
Compares the percentage of the data that consists of sequences of more than a certain number of non-zero duplicate values.
Percentage of zero values
Checks whether the data values are sorted (either in descending or ascending order)
Checks whether the data has a linear trend with time by fitting a linear regression model and comparing the R2 value of the timedependent coefficient.
Determines the absolute value of the lag-1 partial auto-correlation coefficient of the de-trended data.
Tests for the possible presence of PV generation in consumption data. Uses clear-sky irradiance estimates and compares the normalised daily consumption data profile with normalised PV generation profile using Procrustes analysis. Tolerance is the normalised RMSE deviation.
3.3 Base algorithms
3.3.1 Interval model
The interval M&V algorithm is based on a weighted piecewise continuous multi-linear regression for the overall site consumption �������� for a given time interval i as given by Eq. 1.
The baseline period is divided in into ���� periods each with a time duration (excluding missing data) of day_int (default 30) days. A separate model is fit for each period p and weighting factors ��������,���� used to combine the predictions from each model for the time interval i. The method used to determine the weighting factors is described below.
Continuous piecewise temperature levels are used to present the variation of consumption with temperature after the effect of time has been excluded. As shown in Figure 9, the piecewise regression component consists of �������� ,������������=1 + 1 coefficients ��������,������������=1 for occupied time intervals and �������� ,������������=0 + 1 coefficients ��������,������������=0 for unoccupied time intervals. Determination of temperature levels is described below.
Figure 9 Schematic showing piecewise continuous temperature changepoints and associated coefficients.
Coefficients �������� are determined for each half-hour interval of the week. Different options are available for the number of coefficients. The theoretical maximum number of coefficients that can be fit is 672 (48x7x2) (in the most general case each interval can be either occupied or unoccupied), however a more typical number is around 336 since each interval of the week is usually assigned as either occupied or un-occupied. Coefficients are not fit for intervals with insufficient data. In cases where an interval-occupancy combination has multiple data points but the same interval and opposite occupancy combination has zero or 1 data point, only the former coefficient is fit and used for predicting both cases.
An optional coefficient ����������������=1 can be fit for all occupied intervals that fall on public holidays. This applies an additional scaling factor that can be useful to account for sites with less than usual but above baseline consumption on public holidays. By default, this coefficient is not included, and is also disabled if there are insufficient holiday days Setting the parameter fit_holiday_coeff = true enables this coefficient.
The parameter int_mode can be used to modify the way certain intervals are treated from the default (int_mode=1). Setting int_mode =2 groups intervals on public holidays with Sundays. Other options may be added in the future.
In addition, ‘k’ independent variables ‘I’ can also be included with coefficients �������� (see §2.4.4.4 for further details on how to enable independent variables).
For the special case where generation data is unavailable and where the only onsite generation is from PV arrays, it is possible to fit the model directly to net metering data. In this case, it is strongly recommended to use the daily, weekly or monthly models as use of hourly irradiance data can cause issues with fitting of non-irradiance terms (see the Case Studies for further guidance). Setting use_net_data =true includes an additional coefficient ���� in the model which is proportional to an irradiance value �������� for the given interval. The particular irradiance data used (i.e. global horizontal, direct normal, etc.) is not defined, though MVapp automatically links in global horizontal irradiance data if no other irradiance data is supplied. In general for standard fixed mount PV panels it is sufficient to use global horizontal irradiance data even though panels may be
mounted at a significant incline. If there is no on-site generation at all, (or if generation data is supplied in the input stream netG) then set use_net_data=false (the default).
Note: MVApp also runs a check for the possible presence of PV generation. If this check returns positive, then use_net_data is automatically set equal to true, but the type of baseline model (analysis_mode) is not overridden.
3.3.1.1 Weighting factors
Weighting factors are used to combine the N forecasts from the N individual fitted models into a single overall forecast for the period of interest (i.e. the baseline, analysis or forecast periods).
Two different weighting factor methods (Table 8) and three different weighting functions (Table 9) are currently available.
‘Time of year’ weighting applies greater weighting factors to the predictions of models that were fit to data from a similar time of year. This is more appropriate for sites that have additional seasonal variation that is not accounted for by temperature dependent terms in the model. ‘Closer in time’ weighting applies greater weighting to the predictions of models that were fit to data closer in time (i.e. more recently). In both cases time differences are determined from the centre of a given period.
Three different weighting functions (quadratic, cubic and cosine) are available. Weighting functions define the variation of the weighting factor given the weight method. Care needs to be taken in applying the cosine weight function as weights can be zero for part or all of a given period.
Weight method is specified by setting the parameter weight_method to the appropriate id as given in Table 8. Weight function is specified by setting the parameter weight_function to the appropriate id as given in Table 9
Table 8 List of available weight methods
Table 9 List of available weighting functions
3.3.1.2
Temperature levels
In general, two sets of temperature coefficients are fit; one to occupied period data-points (i.e. ��������,������������=1 ) and one to unoccupied points (��������,������������=0 ). The minimum spacing between temperature levels is set using the parameter minDT (default = 1°C). The algorithm combines a given level with its neighbouring level if the number of non-missing data points in the level is fewer than 50 (fixed). This process is performed in ‘leap-frog’ style commencing from the lowest temperature level, then processing the highest temperature, then the 2nd lower temperature level and so on. This process is designed to favour retention of levels at the lower/upper bounds. The temperature levels for occupied and unoccupied periods can be different. There is no limit on the number of temperature levels, including zero levels which results in no temperature coefficients in the model.
3.3.1.3
Occupancy estimation model
Half-hour intervals of the week are classified with a binary ‘occupancy’ flag primarily to enable fitting of separate temperature coefficients in the model for times corresponding to when a site is primarily in-use or largely unoccupied. A typical example is to capture the different behaviour of a commercial office with air-conditioning systems set to maintain zone set-points during office hours and with temperature set-backs or other controls in use overnight and on weekends.
By default, the binary occupancy flag is determined automatically from the derived site consumption data. The primary approach that is used, if there are sufficient data points above and below pre-set temperature change points of 22°C and 17°C, is to fit a simple 2-term regression model to the data with coefficients for the slope of consumption with temperature above and below those two levels. Broadly speaking, data values where the fitted residual is large compared to the actual value are taken to correspond to occupied times. However, different options can be specified as described below. In the event that there are insufficient data points above or below the pre-set change points than only 1 term is fit. If there are insufficient points both above and below then a clustering approach is applied to the energy data using a Euclidian distance metric.
If the parameter ave_occ =true (default), then the occupancy flag is determined by averaging across all of the data values within the specified period for a given half-hour interval of the week. For the standard regression approach this is achieved by evaluating the percentage of positive value model residuals. For the clustering approach the percentage of values in each cluster for a given interval is used.
The occupancy determination processes is applied to segments of the baseline data set of length period weeks in whole weeks. The default is period=999 applies the occupancy algorithm to the entire period of the data. Setting a period of 1 would mean that a particular half-hour interval of the week (say 8:00 to 8:30am on Monday) could be determined to be ‘occupied’ in one week and ‘unoccupied’ the following week. It is recommended that the default period be used to avoid potential problems with over-fitting. This means that at most 336 �������� coefficients are fit. Setting the parameter estimate_occ = false disables the automatic occupancy estimation algorithm. If no occ stream class is supplied and no pre-specified occupancy schedules are defined (see §2.4.3.9), then the occupancy related terms will be excluded from the model.
3.3.2 Daily, weekly and monthly models
Daily, weekly and monthly models all apply the same basic formulation as given by Eq.2 (where the ‘d’ subscript refers to the day, week or month as appropriate). Average daily, weekly or monthly consumption is fit to, in the general case, a function of Cooling Degree days (CDD), Heating Degree days (HDD), average irradiance, binary holiday flag, binary weekend/weekday flag and an arbitrary number of independent variables.
Days, weeks or months with too much missing data are excluded from the fitting process. For the daily and weekly model daily_missing_pct_threshold is used as the threshold, while for the monthly model monthly_missing_pct_threshold is used.
This fitting process is performed using a stepwise model that allows for first-order interactions but only adds terms that are statistically significant. In addition, the upper and lower temperature thresholds �������� and �������� used to calculated CDD and HDD are varied within an allowed range allowing the model to choose values that lead to the best model fit. Bayes Information Criteria is used for final model selection.
Note that occupancy information is not used by the daily, weekly or monthly models.
Similarly to the interval model, Newy-West covariances are computed and standard statistical metrics (R2, RMSE, CVRMSE, and NMBE) are evaluated. Unless cross-validation is enabled (cross_validation =true), these metrics are based on the entire baseline data-set.
3.4 Baseline model criteria
Several tests and metrics are evaluated for the fitted baseline model as listed in Table 10 Each is compared to a criterion or threshold to determine a binary pass/fail. Collectively a given model must pass all the criteria that are included in the overall rating to achieve an overall ‘good’ rating. A failure of one or more included criteria results in a ‘poor’ rating. Models that are given a ‘fail’ rating are those where the baseline model isn’t fit at all (for example due to lack of data or a program error.)
Table 10 Baseline model criteria
R2 Coefficient determination (see Eq. 3). The proportion of variation in consumption explained by the model.
CVRMSE Coefficient of variation of the root mean square error (see Eq. 4). A normalised measure of the variability of the model
25 Y
residuals.
NMBE Normalised mean-bias error (see Eq. 5). A measure of whether the model tends to under or over-predict on average.
Residual autocorrelation
Seasonal trend
Time trend
Based on the Durbin-Watson test statistic for residual autocorrelation.
Determines whether there is a statistically meaningful trend for the model residuals based on season. Based on the method outlined in [17]
Determines whether there is a statistically meaningful trend for the model residuals over time based on the method outlined in [17]
>-0.5 and <0.5% N
>0.5 and <3.5 Y
All segments
P>0.01 or R2 < 0.65 Y
All segments
P>0.01 or R2 < 0.65 Y
Over & underestimated months
Missing data
No. of whole months where the difference between the mean daily model and actual consumption is greater than ±3 standard deviations from the mean.
Whole days are classed as missing when missing data exceeds a daily threshold. The missing data criteria is then calculated based on the percentage of days in the months classed as missing. Excluded days are not counted towards missing data.
Excluded data As per the missing data check. However, excluded days are counted towards missing data.
≤2 underestimated ≤2 overestimated Y
Daily missing threshold 20%.
Monthly threshold <30%. Y
Daily threshold 20%. Monthly threshold <25%. Y
Outliers Difference between model and actual daily energy consumption is greater than ±3 standard deviations from the mean <5 outlier days N
PV
Fits a regression model for daily normalised energy use as a function of ambient temperature, temperature squared and irradiance. Compares the coefficient of the irradiance term for magnitude and statistical significance.
Coefficient < -2.5E-4 & p < 1E-22 or Coefficient <-4E-4 N
The R2, CVRMSE and NMBE metrics are evaluated over the fitted data at the frequency corresponding to the fitting model. For example, for the interval model, hourly values are used. For the daily model, daily values are used and so on.
R2 gives a measure of the amount of variability in the data explained by the model with higher values corresponding to better performance. R2 is calculated by;
where �������� and �������� are the actual and predicted values evaluated over the baseline data-set.
CVRMSE gives a normalised measure of the variability of the error (the difference between the actual and predicted values. A smaller value indicates that the model is able to explain the trends in the data well. CVRMSE is defined as:
NMBE gives a normalised measure of the average error (difference between actual and predicted values) and can be positive or negative. A positive value indicates that the model consistently underpredicts the actual value. NMBE is defined as:
Because under and overprediction errors are cancelled out when calculating NMBE, it is necessary to compare both CVRMSE and NMBE. Table 11 gives an indication of how to interpret these values. For example, a model with a good (low) CVRMSE and a poor (high absolute value) NMBE captures the trends in the data well but consistently under/overpredicts. A model with a poor (high) CVRMSE and a good (lower absolute value) NMBE does not capture the trends in the data well but under/over prediction errors tend to cancel out.
Table 11 Summary of how to interpret model fitting performance statistics
Model captures trends ok & errors cancel out
Model does not capture trends in data well, but errors cancel out
Model captures trends ok but tends to under/over predicts
Model does not capture trends & errors only partially cancel out
3.5 Checks on analysis data & computed savings
Model captures trends ok but consistently under/over predicts
Poor fit of model to data
Tests performed on analysis data and computed savings are for information only. That is, they are not used to update the quality of the baseline model or to modify computed savings. The currently included tests are listed in Table 12
Table 12 Summary checks applied to analysis data or computed savings
Criteria Description
Missing data As per baseline check
Excluded data
Cumulative savings trend
Savings outlier days
Savings outlier periods
Assesses whether the cumulative daily estimated savings amount has a residual seasonal trend after removing any linear trend component.
Identifies individual days where the calculated savings are significantly different from the mean.
Identifies sequences (periods) of 5 or more days where the 30-day moving median savings is greater or less than the 90% and 10% percentiles (respectively)
Use
Can indicate interventions where the savings have a seasonal dependence
May indicate days where non-routine events occurred or alternatively where the baseline estimate is poor.
May indicate non-routine periods.
3.6 Output quantities & uncertainty estimation
3.6.1
Integrated quantities
Integrated output quantities other than those relating to costs and emissions (for example, daily values, monthly values and totals over the baseline or analysis period), are calculated from interval (notionally hourly) values regardless of the model employed. For example, if using a daily model, estimates from the daily model are returned as hourly values that are then processed in the same way as per estimates from the interval model.
Daily values are calculated from interval values by calculating daily means excluding missing data and then scaling for the number of hours in the day. If the percent of missing intervals on a given day is greater than the parameter daily_missing_pct_threshold, then the daily value for that quantity is set equal to missing. Monthly values are calculated from mean daily quantities. An example calculation is provided in Table 13 and Table 14
Missing days are excluded from the calculation of the mean value and, if the number of missing days is greater than the parameter monthly_missing_pct_threshold, then the monthly value for that quantity is set equal to missing. Monthly totals are calculated from the mean daily values with an additional parameter extrapolate_missing_days controlling the scaling. If this parameter is false, then the means are multiplied by the number of non-missing days within the period (i.e. missing and excluded days are not counted). If this parameter is true, then the total number of days within the period is used.
For monthly values an additional parameter extrapolate_whole_months controls whether monthly estimates are based on the number of calendar days in the month, or the number of days in the month within the period of analysis (i.e. within the baseline or analysis periods). When extrapolate_whole_months is enabled, monthly quantities are automatically scaled for missing and excluded data regardless of the extrapolate_missing_days parameter.
Total values over baseline and analysis periods are calculated from mean daily quantities. Totals are also scaled for missing data based on the value of the extrapolate_missing_days parameter. Total quantities are always over the defined baseline or analysis period and are never extrapolated for whole months.
Table 13 Parameters for monthly quantity calculation example
Table 14 Calculated monthly quantity example result
Costs & emissions
The code includes the option to calculate electricity costs and emissions. Electricity cost and greenhouse gas emissions quantities are calculated on a monthly basis as opposed to energy related quantities that are derived from interval values. Electricity costs are comprised of the sum of i) electricity bill tariffs referred to as ‘network’ costs; ii) wholesale (spot price) electricity costs; iii) carbon emission costs. That is, total cost is given by:
�������� = �������� + �������� + �������� (2)
Network costs (NC) include fixed charges, feed-in credits, time of use tariffs, demand charges and capacity charges, as specified in the input tariff data structure (see Table 2). Wholesale costs (WC) based on the Regional Reference Price (RRP) or spot price are included by setting wholesale_price_calc=true and are given by:
where
and
is a flag that is set to true when the parameter export_earn =true. This expression is also shown qualitatively in Table 15
If emissions_calc = true, then carbon costs (CC) are calculated based on the net emissions from grid consumed electricity and, if the parameter use_fixed_emissions_factor = false, using regional real time grid emissions intensity factors EF and the specified carbon price according to:
Here ������������������������������������������������ is a flag that is set to true when the parameter carbon_credit = true.
Table 15 Explanation of wholesale price calculation approach
To calculate cost and emissions savings the adjusted baseline net import energy over the interval i is calculated using the expression:
Monthly costs are evaluated by applying the cost function f (Eq. 2) separately to the adjusted baseline net import and the actual net import on a monthly basis and scaling to whole months (Note that the fixed network charges in Eq. 2 are calculated pro-rata, hence it is appropriate to scale all costs for missing data).
Greenhouse gas emissions are calculated in a similar way as by:
3.6.3 Estimated and claimable energy savings
Estimated or actual energy savings are calculated alongside the claimable energy savings for scheme purposes. Claimable savings follows the methodology outlined in [18] that includes an Effective Range adjustment factor (ERAF) and Accuracy Factor (AF) that lower savings to account for model uncertainties.
Daily actual energy savings (�������� )���� are calculated as described in §3.6.1 from the mean difference between the baseline estimated and actual energy consumption over the interval. (�������� )���� = 24〈(
)〉
The total actual energy saving �������� and percentage saving ������������ then follows as:
Claimable daily energy savings include an additional factor ERAF calculated as:
Where ERAF is the Effective Range Adjustment Factor defined below. �������������������� is the percentage of data points for independent variable v that are outside the effective range. The effective range is calculated based on the data values used to fit the baseline model according to: �������� = min(���� ) 0 05 × �������������������� �������� = max(���� ) + 0.05 × �������������������� �������������������� = max(���� ) min(���� ) �������������������� = max(max(0, �������� ���� ) , max(0, ���� �������� )) /�������������������� ���������������� = 1 |3 × max (�������������������� )|
Total claimable savings (����)���� and percentage savings ������������ follow as: (����)���� =
Where �������� is the estimated savings calculated as: �������� = max(0, (����)���� ) × ��������
AF is the accuracy factor determined via piecewise-linear interpolation of the data points in Table 16 and given the savings precision calculate as:
Calculation of the uncertainty in the baseline estimated total consumption ∆�������� is described in §3.6.4
Table 16 Relation between savings precision and accuracy factor ( [18])
3.6.4
Uncertainties
The primary means of estimating model uncertainty is based on the approach outlined in ASHRAE14 [6]. Assuming measurement uncertainties are small, the total absolute uncertainty in an integrated quantity Q (e.g. monthly or total energy, cost or emissions saving) is estimated as:
In this equation:
〈���� 〉���� is the model estimate of the mean quantity (for example the daily mean energy consumption) over the period of interest
〈���� 〉���� is the actual value of the mean quantity over the baseline period. ���������������� is the mean square error (residual) of the quantity over the baseline period.
����′ is the estimated number of interdependent observations over the baseline period and is calculated as ����′ =(1 ����ℎ����)/(1+ ����ℎ����) where rho is the first order autocorrelation of the baseline residuals.
�������� is the number of values in the period of interest.
m is the number of intervals over which the mean quantity is to be aggregated
��������,�������� is the inverse cumulative Student’s t distribution function for probably p and degrees of freedom nu
���� is the desired confidence level set using the parameter confidence (���� =1 −����������������������������������������/100 )
Monthly energy uncertainties are evaluated using the daily energy values while monthly cost and emissions uncertainty are evaluated using monthly total costs and emissions.
Note: internally MVApp is setup to compute heteroscedastic covariance estimates from the fitted regression models and use these to compute prediction intervals (via the parameter hac_on). This feature was used in earlier versions of the app. At present, heteroscedastic uncertainty intervals are overridden by the calculated ASHRAE prediction intervals regardless of the value of this parameter.
Part II Case studies
4.1 US commercial buildings (MVApp)
The International Efficiency Valuation Organisation (EVO) operates a portal that can be used to benchmark the performance of Measurement and Verification applications [19]. This contains whole of building half-hourly interval electrical power consumption data and ambient temperature data from 367 commercial buildings spread across the United States. No other information is known about the buildings or their locations.
For each building 1 year of data is available to construct a baseline model. An additional year of temperature only data is provided for blind validation. That is, users can upload the predictions output from their M&V model based on the additional year temperature data and the portal evaluates model performance against an unseen data-set that is not available to users. Benchmarking of MVApp using this portal is described in §3.7.1 below.
The EVO portal data is also used here to assess how many of the buildings pass the different baseline model criteria (see §3.7.2).
Finally for each building known energy saving interventions are simulated over the last month of the data and the ability of the MVApp to correctly identify these saving is evaluated (see §3.7.3).
4.1.1 Benchmarking performance
MVapp underwent a blind validation using the Efficiency Valuation Organisations online EVOportal [19] [20]. One year of energy and temperature data is available and the M&V algorithm was used to generate baseline models and predictions for the second year of data which only includes temperature data. The portal automatically calculates model perform metrics (CVRMSE and NMBE) based on a comparison of the model predictions for the second year of data.
At the time of running 93 models had been submitted to the portal with summary performance results as shown in Figure 10 The DCH M&V application achieved a median CVRMSE score of 33.75% and NMBE score of 0.04% placing it in the top 5% of all models and very close to the top performing model. Note that, unlike the requirements listed in M&V standards and guidelines these performance metrics are calculated on a separate testing data-set, and not on the data used to train the models. Hence, the reported CVRMSE values in particular are inherently higher than those calculated for the fitted models.
DCH M&V
CVRMSE=33.75%
NMBE=0.04%
Figure 10 Screenshot of all M&V applications with results shared on the EVO-portal (as of 26th July 2023). DCH-M&V application results as indicated.
4.1.2 Assessing baseline model criteria
Daily and interval baseline models were fit to the EVO portal buildings and the number of buildings where the models satisfied each of the baseline rules and checks was assessed. Results are summarised in Table 17. Just below half of the buildings met all of the baseline criteria with the number slightly higher (48.5%) for the interval model than for the daily model (42.8%). Similar percentages of buildings passed the CVRMSE and under-overestimated month criterion, but although there was a degree of overlap (approximately two-thirds of buildings either met or failed both criteria), a sizeable number passed one and failed the other criteria suggesting that these tests assess different aspects of the models. The rules/checks applied by MVApp lead to an approximate 20% increase in the percentage of buildings that fail the criteria as compared to only R2 and CVRMSE metrics alone.
Table 17 Summary of percentage of EVO building baseline models that passed each baseline rule/checks
data
applicable (no excluded data)
(no
Plots of the cumulative distributions of R2 values, CVRMSE values and total number of over or under-estimated months are shown in Figure 11 from the interval models. Also shown are boxplots CVRMSE for different numbers of estimated outlier days. Most buildings (>95%) had R2>50% or CVRMSE < 40 or the number of under and over-estimated months less than or equal to 3. CVRMSE is weakly correlated with the number of outlier days, though there is significant variability
Figure 11 Cumulative histograms of R2, CVRMSE and number of over or under-estimated months, and boxplots of CVRMSE for interval baseline models for EVO portal buildings.
4.1.3
Comparison of actual, estimated & credited savings for simulated interventions
Since no information on any interventions is available for EVO portal buildings, the intervention simulation test feature of MVApp was used to artificially simulate an energy saving. This was used to test the ability of the models to correctly identify an energy saving.
Because only 12 months of data was available, baseline models were fit to 11 months of the data, and an average 20% energy saving intervention simulated for the remaining one month of data The intervention was applied to workdays and office hours only using a ‘vshape’ temperature
dependent weighting factor of 0.2. That is, a higher energy saving was applied to higher and lower temperatures, but the saving was such that it averaged to precisely 20% over the month.
Overall results are summarised in Table 18 for the sites where the baseline model met all of the criteria (48.5% or 178 sites using the interval model and 43.3% or 159 sites using the daily model).
The estimated energy saving band contained the actual savings amount for only approximately one-quarter of sites. However, the majority of sites were credited with a savings amount between 10% and 20% and almost no sites were substantially over-credited with a savings >30%. This is evident in the distribution of credited savings percentages (Figure 13 left). The spread in estimated savings tends to increase as CVRMSE increases which is consistent with CVRMSE being an indicator of the quality of the model fit.
Figure 12 plots the magnitude of estimated and actual energy savings for each site with a valid building model from the daily and interval analysis. Lines of best fit are also shown. For the majority of sites, the magnitude of the estimated saving is predicted well, though there are some sites with a large error. These sites are not necessarily associated with a higher baseline model CVRMSE. They tend to be sites where the baseline (non-intervention-adjusted) energy use for the post-retrofit month deviated from the trend over the full baseline period.
Table 18 Summary aggregate energy savings statistics across buildings with valid baseline models
Figure 12 Magnitude of estimated and actual energy saving for sites with valid building models (interval model and daily model results shown). Interval model slope = 0.9798. Daily model slope = 0.9689.
In contrast to the variability present at a site-by-site level, the estimated, credited and actual aggregate absolute energy savings over all buildings are very close for both models. The magnitude for the daily model is less as fewer buildings had valid baseline models and hence the aggregation contains fewer buildings.
It is important to note that the above results are based on 1 month ‘post-retrofit’ periods. It is likely that the estimated percentage savings would be much closer to the simulated amount if analysing a full year of data as any skew present in the estimates for the selected month would tend to average out over the year. Hence, these results should be viewed as a worst-case scenario.
Figure 13 Distribution of credited percentage saving (left) and boxplots of estimated saving percentage as a function of CVRMSE (right) for sites with a valid baseline model based on the interval analysis model.
Figure 14 Distribution of credited percentage saving (left) and boxplots of emitting saving percentage as a function of CVRMSE (right) for sites with a valid baseline model based on the daily analysis model.
4.2 Australian commercial buildings (MVApp)
4.2.1 Use of net metering data in the presence of onsite generation
Interval and generation meter data for seven commercial sites spread across regional NSW was used to evaluate the sensitivity of the MVApp algorithms to the amount of PV generation present when using the ‘net-meter’ analysis model (i.e. use_net_data=true).
The sites all have onsite PV generation systems with annual generation ranging from 16% to just over 70% of annual consumption. M&V was run with 1 year of baseline data and 1 year of reporting data. The same baseline and reporting periods (baseline 01/03/2021 to 28/02/2022 and reporting 01/03/2022 to 28/02/2023) were selected for each site without any assessment of the presence of non-routine events or data quality. No information on the presence (or not) of energy interventions or non-routine events was available.
The presence of generation meters for each site enabled a comparison of two methods:
Method 1: MVapp applied to consumption data using only temperature and time related variables (i.e. Eq. 1 with ���� =0 and Eq. 2 with ����5 =0). Consumption was calculated from: consumption = net meter data + generation meter data, and
Method 2: MVapp applied to net metering data directly and with the radiation dependent model term activated.
Analysis was performed using both the interval and daily models (note: net meter analysis is only recommended to be used with the daily model). Additional analyses were also run with sub-hourly measured onsite generation values scaled up/down by fixed factors to simulate different sized PV generation systems. Model performance was assessed using CVRMSE and NMBE (normalised mean bias error) with CVRMSE used as the primary indicator of model acceptability.
4.2.1.1
Interval model results
Interval model results are summarised in Tables 19 and 20 with sites listed in order of increasing overall generation amount as a percentage of overall consumption. Five of the seven sites had
acceptable baseline models built using the consumption meter data. None of the sites had a valid model using the net meter data with additional irradiance model term. In terms of the calculated saving over the reporting period (noting that there were no known interventions applied), consumption and net meter model calculated savings overlapped at least partially for all but one site, although mean savings predictions varied considerably, and the uncertainty range from the net-meter based models was very wide for the sites with greater generation fraction. Given that none of the sites had acceptable models using the net-meter approach, it is recommended that the interval model not be used with net meter data.
Table 19 Summary of interval model performance metrics for consumption and net meter analysis models
Table 20 Summary of interval model calculated reporting period savings for consumption and net meter analysis models
4.2.1.2 Daily model results
Daily model performance results are summarised in Table 21. For the consumption-based model the same five sites had acceptable models as per the hourly model. However, in contrast to the hourly approach, all four of these sites still had acceptable net meter-based models, although the CVRMSE increased in line with the increasing generation (Figure 15). In terms of estimated savings,
the consumption and net based model estimates overlapped for the three sites with lowest generation percentage, although again there were differences in the mean estimated savings (Table 22).
Table 21 Summary of daily model performance metrics for consumption and net meter analysis models
Table 22 Summary of daily model calculated reporting period savings for
analysis models
For each site, the relative generation amount (ratio of annual generation to annual consumption) was also varied between 0% (no generation) and 90% by scaling the generation meter data accordingly Figure 15 compares the resultant increase in the CVRMSE of the net meter-based model (i.e., CVRMSE_model2 – CV_RMSE_model1) versus the generation as a percentage of consumption. Symbols indicate the result corresponding to the actual generation for each site while the shaded region shows the range of values for all sites.
Given that a CVRMSE below 15 is considered a very good model, and a minimum acceptable CVRMSE of 25 is cited in several sources [6] [2], the results suggest that generation values up to 20 to 30% of consumption may be tolerable using the daily model with an irradiance dependent term included. However, the consequence of this increase in CVRMSE is a reduction in the predictive
power of the model and hence an increase in the minimum saving that can be identified (without the additional solar PV metering) with a given confidence level. Hence, it is always better to perform an M&V analysis using consumption data (i.e. to supply both net and generation data) where available.
Figure 15 Increase in CVRMSE between baseline consumption daily model and net meter based daily model as a function of relative generation amount.
4.2.1.3 Choosing between interval and daily models for net- meter analysis
A comparison of the estimated savings using the hourly consumption, hourly net, daily consumption, and daily net models is given in Figure 16 Sites are ordered with increasing generation fraction from left to right. Note that statically significant changes (either savings or energy increases) are calculated for several of the sites. However, given that no information was available on the presence (or not) of any energy interventions for these sites it is not possible to state whether the estimated savings are the result of real measures, changes to site operation or non-routine events.
Results show similar differences between the hourly and hourly-net models and the daily and daily-net models with both showing much greater variation as the generation fraction increases (moving from left to right). Note the large deviations for the daily-net model for Griffith and Moree may be linked to the fact that even the consumption-based models for these sites did not meet the baseline criteria. Hence, it is difficult to draw further conclusions from this.
Figure 16 Comparison of estimated percentage savings and uncertainty ranges as computed using hourly, hourly net, daily and daily net models for seven sites.
Both the interval and daily models add a simple irradiance term to attempt to account for onsite generation as a function of local global irradiance. As shown in Figure 17 (left) for one particular location, the daily integral global horizontal irradiance is linearly correlated with the daily integral PV generation as the cumulative effects of localised shading, collector orientation, weather and other unknown effects on generation tends to be smoothed out. However, on an hourly scale (Figure 17 right) much more scatter is present. R-squared coefficients of a linear regression applied to hourly and daily data (Table 23) show this behaviour is consistent across the sites. The practical effect of this scatter is to introduce noise that makes it more difficult for the temperature and time-of-week terms in the model to be fit accurately.
Figure 17 Correlation between daily site generation and daily horizontal irradiance (left) and hourly site generation and hourly horizontal irradiance (right) for Site 6. Dashed lines show fitted linear regression model with zero intercept.
Inspection of hourly boxplots of the residuals of regression model fit to the hourly data (Figure 18) shows a trend of under-estimates in the morning and over-estimates in the middle of the day which is likely due to a combination of collector orientation and shading effects. Since the analysis has no access to such information for a given site (i.e., we assume it is not available) it is not possible to employ a detailed PV model. However, additional simulations were trialled
standard collector orientation and tilt to calculate irradiance in the plane of the collector. The resulting net meter-based model did not yield any improvement in performance (either model metrics or savings estimates).
Figure 18 Boxplots of hourly irradiance – generation linear regression residual as a function of hour of the day.
Table 23 R-squared values of linear regression models fitting hourly and daily site generation to hourly and daily horizontal irradiance.
Site 1 0.870 0.964
Site 2 0.860 0.900
Site 3 0.910 0.962
Site 4 0.830 0.950
Site 5 0.865 0.925
Site 6 0.872 0.938
Site 7 0.871 0.935
Although these results are based on analysis of a limited number of sites, they suggest that it is possible to perform M&V using net meter data only and achieve an acceptable baseline model. However, a model with daily or longer timescale should be used, and if the overall site (PV) generation is more than approximately 20-30% of consumption, then additional generation meters should be installed so that the analysis can be based on consumption data. The reduction in model predictive power that results from using net meter data and the impact this has on the creditable savings is also an important factor to consider If non-PV generation is present, then either additional independent variables appropriate to that generation type or additional sub-metering are likely to be required. Although not considered here, battery storage systems are a special case, and their treatment will depend on how the batteries are operated as well as the M&V project scope. For example, a small battery that never or rarely causes significant net export and that operates on a regular schedule of charging and
discharging may be able to be ignored (i.e., incorporated into the baseline site load). On the other hand, if for example, the battery operation is subject to significant variability or results in a large quantity of exported power at certain times then additional metering is likely required. Complications such as if the battery operation changes in response to the energy efficiency intervention will require the expertise of an M&V professional.
4.2.1.4 Assessing the PV baseline check
MVapp includes a test to identify the possible presence of PV generation data in (what is supposed to be) consumption data. When performing an analysis on net-meter data, these equates to testing for the presence of PV generation in the net data. As described in Table 10 the test attempts to fit the daily normalised energy to a function of ambient temperature, temperature squared and irradiance. The coefficient of the irradiance term is tested for magnitude and statistical significance.
This test was applied to each site with different levels of assumed generation implemented by scaling the actual generation up/down using a multiplying factor. The value of the annual percentage of generated energy compared to consumed energy at which the test detected the presence of generation is given in Table 24
For the sites that had a valid baseline model using the net meter data with the actual amount of generation (sites 1, 2, 3 and 4), the test was able to detect PV generation levels at least greater than 18% and, in some cases, less. For one site with a valid net-model and only a small generation percentage this was not detected, while for all other sites the generation was detected in the actual net meter data. Thus, a positive detection of PV generation does not automatically mean that the net-based baseline model will fail all other criteria. This check provides additional information that might help to diagnose why a particular baseline model fails.
Table 24 Summary of PV generation test results
4.3 Direct NMI meter data Australian office buildings (SWCApp)
4.3.1 Overview
As part of developing the Web-portal and SWCApp, net meter data from NMI meters across 13 demonstration sites was ingested into the DCH to replicate the information expected to be available from the Consumer Data Right based NMI meter ingestion process via the electricity market operator AEMO. These sites are a mixture of commercial office buildings and combined office and research facilities. This data was used to test the automated end-to-end process from adding a site using the Web portal through to viewing a baseline model.
From a user perspective, very few steps are involved, however as described in §2.1 there are several stages and interacting pieces of software. Once a web-portal user selects ‘add new site’ and enters the necessarily fields, the back-end initiates a series of processes that begins with finding and ingesting the data and setting up the building model. Once this and several other processes complete, SWCApp is called. The models are queried, data-stream and user parameters initialised, configuration parameters constructed, and MVApp is called to process the data and construct the baseline model. Once the baseline is obtained SWCApp manages the outputs, creating the necessary output port documents that the back-end exposes to the web-portal.
This case study was used both to test these processes, and also to evaluate how many of the sites resulted in successful creation of a model that passed all of the baseline criteria, without any specific parameter selected for individual sites (i.e. using a fully automated approach). For any sites where an eligible baseline model was not obtained the underlying cause of the failure was investigated and this case study focusses on these results.
For all cases 12 months of data (Nov 2022 – Nov 2023) was used to construct the baseline models. No days were excluded, and default values of all other settings were used. The post-code associated with each site was the CBD of the capital city in the corresponding state. Given no weather data (ambient temperature and irradiance) was specified in the models, MVApp linked data from the nearest BOM weather station to this postcode. Given some sites are located significant distances from these postcodes, differences between this weather data and the actual conditions at the sites are a potential source of significant model error as discussed below.
4.3.2 Understanding why baseline models can fail
Baseline model results for the 13 sites are summarised in Table 25 A baseline model was successfully created for 12 of the sites with the model failing to run on 1 site. A ‘good’ baseline models was obtained for 5 of the 12 sites.
Table 25 Summary of automated baseline model metrics for 13 test NMI meter sites.
202 poor 66.4 11.7 Time trend, PV check Clear trend of consistent increase in energy use with time.
203 poor 38.4 47.2 Autocorr, time trend, over/under estimated months
204 poor 50.3 36.9 CVRMSE
Clear trend of large decrease in energy use with time.
Data shows large increase in energy use over winter.
207 poor 45.3 5.7 R2, over/under estimated months, missing data
208 Poor 34.4 8.7 R2, autocorr, over/under estimated, missing data
310 fail - - N/A
Data has large spikes (both high and low energy use) consistent with non-routine events.
Highly variable energy use.
No valid data points found in Semantic model
R2,CVRMSE,autocorr, over/under estimated
Data quality issues. Data from multiple meters with covering different periods. Long periods with duplicate values on some meters.
For the 1 site where baseline model failed (310), inspection of the error logs reveals that the Brick query returned 1 energy meter but no associated ‘points’ (i.e. data streams). With no valid data streams MVApp was not called and SWCApp was aborted, as intended. Investigation of this software automation failure (rather than baselining algorithm failure) is ongoing as to why the building model was constructed without any points.
The seven sites that returned poor baseline models failed one or more criteria as listed in Table 25. Figure 19 shows plots of the daily actual energy use timeseries, daily outliers, monthly average, model estimated energy use and the expected allowable range of monthly estimated energy use for 6 sites. These sites were selected as representative of different characteristic behaviours.
The model for site 201 (top left) was close to passing all criteria, failing on the CVRMSE criteria by a relatively small amount. Inspection of the data reveals a clear daily energy use pattern but with significant amounts of variability from week to week. For example, some weeks have consistent daily energy use values twice as high as the previous week, and this variation is not explained by
temperature, irradiation or holiday variables used. This causes the CVRMSE, which is a measure of the amount of variability not explained by the model, to be larger. Inspection of the hourly data reveals that the site has PV generation which, when combined with the fact that the actual site is located approximately 550km inland from the location used for the irradiance data, may be a significant cause of the unexplained variations.
Figure 19 Daily energy use variation plots. Actual daily energy use during the baseline period (blue line), monthly mean estimated energy use (black line), expected range of monthly estimated (blue shaded region), daily outliers (symbols) and monthly outliers (pink and light green lines). Top left: case just failing CVRMSE criteria only (201). Top right: case failing time trend (203). Centre left: case failing CVRMSE and showing large seasonal variation (204). Centre right: large spikes in consumption coupled with periods of lower and higher consumption (207) Bottom left: data quality issues confounded by the presence of many meters (509). Bottom right: case passing all criteria (311).
The baseline model for site 203 (top right) failed the time trend criteria (and several others). The figure shows a very clear trend of decreasing energy use from the start to the end of the baseline period with under-estimated months at the start and over-estimated months at the end. Since creation of a baseline requires a stable use pattern, it is appropriate for the baseline to fail for this site and selected period.
The model for site 204 (centre left) also failed the CVRMSE. However, unlike site 201 which displayed large variability throughout the year, the model for site 204 is quite a good fit over summer, autumn and spring but under-estimates usage by a large amount in winter. This is typical of a site with high energy consuming equipment that is used seasonally. This may be, for example an electrical heating system that is either manually activated or activated based on the season as opposed to the actual daily temperature. Noting though that this site is also approximately 350km inland from Sydney the model may improve significantly if more accurate weather data is used. The use of non-linear temperature dependent terms in the model, or use of the interval model instead of the daily analysis model, may also improve the model but these options are currently only available by manual configuration (i.e. by using MVApp).
Site 207 (centre right) displays large daily outliers (mostly lower than expected energy use) as shown by the green symbols. Also present are longer periods of lower and higher energy use consistent with non-routine events and significant gaps in the data (missing data). This site houses a large astronomy telescope and associated equipment whose energy consumption patterns are unknown but potentially variable. This model fails the R2, over/under-estimated months and missing data criterion. Manual use of exclusion days, and rectification of missing data issues could result in a good baseline using this data, though this is not possible with an entirely automated process.
Site 509 (bottom left) had multiple NMI meters (6 data streams) and suffers from severe dataquality issues. Some of these streams had 1 or more periods of several weeks duration with identical (duplicate) non-zero values. The automated data processing rules would have removed these periods from the data streams in question but not excluded the time intervals entirely due to the presence of valid data from other streams. In addition, some of the data streams look to have started part-way through the period which may indicate changes to the site (for example addition of buildings, new PV generation or meter reconfiguration). These data issues are flagged by MVApp and reported in the logs but are otherwise not visible to the Web-portal users and only manifest in a failed baseline if severe enough to cause one or more of the baseline criteria to be violated. These periods appear on the daily energy use plot as periods of multiple outlier days and or over-estimated months.
Finally, 5 of the sites (close to 40%) resulted in a ‘good’ baseline passing all the criteria. Site 311 (bottom right) is an example of the daily energy use plot for such a site. Daily energy use varies over the year, but the trends are captured by the model. There are a small number of daily outliers and only 1 under-estimated month (incidentally the last month of the baseline period and the subsequent analysis period months show higher usage. Note there were no known interventions for any sites here)
All 5 of the sites with a ‘good’ baseline were CSRIO sites with largely office-based usage and only one other CSIRO office-based site (Pullenvale) did not achieve a good baseline. The other CSIRO sites that failed the baseline criteria included the Parkes observatory, Narrabri Telescope and Canberra Deep Space communications complex. The 3 Property NSW sites (201, 202 and 203) also failed the baseline for the reasons described above. The automated analysis is more likely to be suited to office-based applications as opposed to sites with potentially bespoke high energy consuming equipment.
4.4 Industrial site with additional independent variables (MVApp)
4.4.1
Overview
This case study consists of an industrial waste-water treatment site where an actual energy saving measure was implemented. The energy savings project involved replacing 1 of 4 aeration blowers with a high efficiency unit. The measurement boundary was the energy use from all 4 blowers. Energy savings credits were awarded under the NSW Energy savings Scheme based on a PIAM&V analysis. The consultant report [21], calculations and associated 15-minute datasets corresponding to pre- and post-intervention periods were provided.
The PIAM&V analysis was based on 1 year of pre-intervention data used to construct the baseline model and 16 months of post-intervention data used to construct an operating energy model. Energy savings were estimated for a normalised year constructed from approximately 4 years of pre-intervention data.
The data provided included the total air-flow rate for the 4 blowers, some additional process variables related to the plant operation, weather variables, the overall site energy consumption, and the total blower energy consumption. Following the consultant report the blower total airflow rate was included as a primary independent variable and the total blower energy consumption as the dependent variable. The default MVApp settings also include ambient temperature as an independent variable (or CDD & HDD for the daily models) if statistically significant (a weak dependence on ambient temperature was confirmed).
For the initial analysis (§4.4.2) and analysis of influence of excluded data (§4.4.3) weather variables were allowed to be retained and savings were computed for the actual post-retrofit year. In the analysis of the influence of normalised year selection (§4.4.4), temperature variables were forcibly excluded by modifying the MVApp code, and savings were calculated for a normalised year using both baseline and operating energy models following the method used in the consultant M&V report.
4.4.2 Analysis with default settings to estimate actual year savings
An initial analysis using MVApp was conducted with default settings which meant inclusion of ambient temperature as an independent variable in the model alongside time-related variables and the air-flow rate variable.
Because MVApp is primarily intended for computing savings directly over the post-intervention period as opposed to normalised year savings, initially actual year savings estimates from both the hourly and daily model were compared. Additionally, the PIAM&V report identified 26 days (7%) of the baseline data to be excluded from the model fitting process; stated to be due to non-normal plant operation on those days.
The influence of excluding these days was compared relative to the choice of model analysis type. Results are summarised in Table 26. The hourly and daily models give similar savings estimates, particularly when using all the data, but the uncertainty is much wider without the exclusions. Excluding the suggested days increases the savings estimate while substantially improving the uncertainty. Further analysis of the influence of excluding days is discussed below.
4.4.3 Influence of excluding data on actual year savings
In recognition of the fact that deviations from ‘normal’ operation of a site occur from time to time and that it may be appropriate to remove the effect of these deviations from the baseline operating model, the IPMVP and other guidelines describe methods of handling these ‘nonroutine events’. The simplest approach is to exclude the periods entirely. For example, under the NSW MBM up to 20% of days in the baseline period can be excluded from the fitting process if it is deemed the operation on these days was significantly different to normal operation [1] Justification for removal of these days may involve stating, for example, that the site under-went maintenance on those days, or that some equipment was not operational.
This case study was used to assess in more detail the influence of the number and choice of days to exclude from the baseline model fitting process. As above, the default settings were employed, and actual year savings were evaluated. The number of days excluded was varied and 5 different approaches to excluding days were used. These were:
Method 1: Preferentially removing days with the highest energy use
Method 2: Randomly removing days
Method 3: Preferentially removing days with the lowest energy use
Method 4: Preferentially removing days with the highest value of the independent variable
Method 5: Preferentially removing days with the highest baseline model error
Each of these methods is ‘blind’ to whether or not the data exclusions would be justified by real plant non-routine operational considerations. As such, these methods represent possible ways of automating exclusions (rather than applying manual exclusions by expert assessors)
For each approach and for each assumed number of excluded days, a new baseline energy use model was fit to the remaining data and used to estimated post-retrofit energy savings. For all cases the post-retrofit data was unchanged.
Results are summarised in Figure 20 which plots the estimated savings versus the percentage of baseline data excluded for the 5 different exclusions methods. Removing days randomly (pink line) leads to almost no change in the estimated saving which provides some assurance that the estimated savings are insensitive to the amount of data used to train the model. Removing either the highest or lower energy use days has the expected effect on savings (removing higher energy use days lowers the baseline estimate which in turn lowers savings) but the change is relatively
gradual with minimal effect below 5% of data excluded. The most surprising result is perhaps the effect of removing days that have the highest model error (i.e. the highest difference between the model estimates and the actual energy use). In the worst case, only a small percentage of days removed (4%) is enough to change the estimated savings by 60%. This suggests that automated processes should not be used to remove data solely on the basis of the quality of the model fit. This would effectively assume that the model was a perfect representation of the site operation and that any deviations were caused by non-normal operation – an invalid assumption. Hence, MVApp does not include any automated data exclusion on the basis of model fitting.
The results also show that removing more data does come with increased risk of influencing the estimated savings. For any given analysis, it is possible to assess the influence of any exclusions that an M&V practitioner may apply by separately fitting two models; one with the data excluded and one without. This could potentially be used to evaluate whether the specified date exclusions are allowable in addition to the fixed allowance for the amount of data that can be excluded. For example, if it were required that the influence of the data exclusion on estimated savings be <±10%, then the 26 days excluded here would just fall within a 10% deviation and hence would be allowed.
02468101214161820
Figure 20 Variation of estimated savings as a function of percentage of excluded data for different data exclusions strategies.
4.4.4
Influence of choice of normalised year
Some M&V applications such as forward creation of energy savings certificates involve estimation of ‘normalised year’ savings. These are savings for an ‘average’ or ‘normal’ year. This involves constructing two models; a baseline model trained on the pre-intervention data, and an operating model trained on the post-intervention data. Both models are then used to estimate energy use against a set of independent variables chosen to be representative of the normal year.
This section considers the influence of the selection of the values of normal year independent variables on the estimated savings. Currently MVApp does not automate the construction of normal year independent variables because calculation of normalised year savings was not a core development focus (this feature may be added in the future). Hence, users must separately construct the normal year data and supply it to MVApp using the standard stream classes structure.
To study the influence of choice of normal year variables, the MVApp daily analysis model was used, and the temperature related variables (CDD and HDD) were forcibly disabled to match the model used in the consultant PIAM&V report as closely as possible. Days were excluded from the baseline and analysis periods to match the report.
The NSW PIAM&V guide [2] gives limited information on how to choose normal year variables. For example, it suggests only that measured data less than 3 years old should be used and the data should be representative of ‘typical’ operating conditions. The IPMVP guidelines are also open to interpretation [5]. Here, five different approaches for choosing the normal year data for the single independent variable (airflow) were considered. These were:
Method 1: For each day of the year, calculate the absolute difference between the daily air-flow value and the average air-flow value from all years for the same day. Calculate the monthly sum of the absolute differences. For each month, choose data for that month from the year that has the smallest monthly deviation for that month. Note: this method was used in the consultant M&V report.
Method 2: For each day of the year, average the daily air-flow from all years.
Method 3: For each month of the year, calculate the monthly average airflow. Then calculate the difference between the monthly average air-flow for a given year and the monthly average across all years. For each month, choose data for that month from the year with the smallest absolute monthly difference.
Method 4: Select daily airflow values from the weighted cumulated distribution function of all daily values specifically constructed for that day of the year. The weighting factor is based on the day of the year and applies a quadratic decay.
Method 5: For each day of the year, select the value for that day from all years randomly. Accept the selected day with a probability based on the weighted probability distribution function of daily values from all data.
Baseline and operating energy models were fit to the pre and post intervention data-sets respectively. Then a normalised year of data for the independent variable was constructed using approximately 4 years of daily data and the 5 different normal year methods (i.e. 5 different ‘normal’ years were created). The same baseline and operating energy models were then used to estimate baseline and post-retrofit energy use for the normal year. Results are summarised in Table 27
The estimated energy saving ranges from 98.7MWh to 114.4MWh. A variation of 15%. The methods that result in higher annual mean airflow (Method 1 and 3) lead to higher savings because the difference between the power use estimates of the operating and baseline models increases for higher airflows.
Both of these methods are based on choosing whole months of data by comparing the absolute deviation of the monthly averages. This means that if the distribution of daily values is skewed (for example there are more values below the mean than above the mean), the overall average for the ’normal year’ can be significantly different from the data. This is the case here were the overall mean daily airflow from all the data is 1581L/s and the value for the Method 1 ‘normal’ year is 1648 L/s. Method 2 avoids this problem by calculating average values per day. However, this may introduce other problems. Most critically by averaging across years it reduces the variability in the data. It also doesn’t use the actual value from any specific day.
Methods 4 and 5 sample from the actual data distribution functions and hence they recover the true variability of the independent variable. Method 5 has the added benefit of using data from actual days to construct the normal year. Although it does not recover sequences of days (for example where a day with a low value of the independent variable may be more likely to follow a previously low day), this is not required for M&V applications where each day is treated as a separate independent occurrence. Hence, Method 5 is the suggested approach for constructing normal year data.
Finally, of note is that the PIAM&V reported savings was 125.2MWh ±4.9 MWh. This is 9% higher than the value calculated by MVApp (114.4MWh ±13.6MWh) using the same normalised year data. The difference stems from the baseline energy model developed here that included weekend/weekday as an additional independent variable found to be statistically significant (t-stat = 3.2) and which resulted in the baseline energy use being 11 MWh higher. Also of note is the relative uncertainty estimated by MVApp which is wider despite the models having similar fit statistics (R2 and CVRMSE). MVApp uses the ASHRAE equations to calculate uncertainty whereases the report uses the standard error estimate approach.
5 Works Cited
[1] IPART, “Metered Baseline Method Guide v2.6,” Independent Pricing and Regulatory Tribunal of NSW, Sydney, 2023.
[2] IPART, “Project Impact Assessment with Measurement and Verification Method Guide v5.0,” Independent Pricing and Regulatory Tribunal of NSW, Sydney, 2023.
[4] Government of South Australia, “Retailer Energy Productivity Scheme,” Essential Services Commission of South Australia, [Online]. Available: https://www.escosa.sa.gov.au/industry/reps/overview/reps. [Accessed 18 7 2023].
[6] ASHRAE, “Guideline 14-2014 Measurement of energy, demand and water savings,” ASHRAE, Atlanta, 2014.
[7] M. Goldsworthy, “Automated M&V2.0: Current status and challenges,” CSIRO, Newcastle, 2023.
[8] S. Fernandes, E. Crowe, S. Touzani and J. Granderson, “Detecting the undetected: Dealing with nonroutine events using advanced M&V meter-based savings approaches,” Lawrence Berkeley National Laboratory, 2020.
[9] S. Touzani, B. Ravache, E. Crowe and J. Granderson, “Statistical change detection of building energy consumption: applications to savings estimation,” Energy and Buildings, vol. 185, pp. 123-136, 2019.
[10] S. Touzani, J. Granderson, D. Jump and D. Rebello, “Evaluation of methods to assess the uncertainty in estimated energy savings,” Energy and buildings, vol. 193, pp. 216-225, 2019.
[11] J. Granderson and P. Price, “Development and application of a statistical methodology to evaluate the predictive accuracy of building energy baseline models,” Lawrence Berkerley Laborartory, 2013.
[12] B. Koran, L. West, E. Boyer, S. Khawaja, J. Rushton and J. Stewart, “A comparison of approaches to estimating the time-aggregated uncertainty of savings estimated from meter data,” in International Energy Program Evaluation Conference, Baltimore, 2017.
[13] W. Travis, P. Price and M. Sohn, “Uncertainty estimation improves energy measurement and verification procedures,” Lawrence Berkeley National Laboratory, 2014.
[14] C. Gallagher, K. Leahy, P. O'Donovan, K. Bruton and D. O-Sullivan, “Development and application of a machine learning supported methodology for measurement and verification (M&V) 2.0,” Energy and Buildings, vol. 167, pp. 8-22, 2018.
[15] M. Ke, C. Yeh and C. Su, “Cloud computing platform for real-time measurement and verification of energy performance,” Applied Energy, vol. 188, 2017.
[19] J. Granderson, S. Touzani, C. Custodio, M. Sohn, D. Jump and S. Fernandes, “Accuracy of automated measurement and verification (M&V) techniques for energy savings in commercial buildings,” Applied Energy, vol. 173, pp. 296-308, 2016.
[22] J. Granderson, S. Touzano, S. Fernandes and C. Taylor, “Application of automated measurement and verification to utility energy efficiency program data,” Energy and Buildings, vol. 142, no. 1, 2017.
[23] G. Ruiz and C. Bandera, “Validation of calibrated models: common errors,” Energies, 2017.
[24] NSW Government, “PIAM&V method application requirements for non-routine events and adjustments,” NSW Government, 2022.
Not allowed irradiance Irradiance used for fitting PV generation. For example can be total horizontal irradiance or tilted surface irradiance.
Not allowed electricityspotprice Historical regional reference price of electricity purchased from the grid
Not allowed emissionsfactor Historical emissions intensity of electricity purchased from the grid.
drflag DimensionlessUnit DimensionlessUnit Not allowed signedflag Flag indicating the prescence of a DR event (1 = increase event, -1 = decrease event, 0 = no event)
indepvar none N/A
Retained as separate streams none
Additional independent variables added to the model.
occ DimensionlessUnit DimensionlessUnit Not allowed binaryflag A binary occupancy flag that can be used by the interval model.
1Allowed units are those associated with the default ‘type’. Allowed units can be changed by changing the specified ‘type’.
2 If data is supplied as energy values instead of power values, change the ‘type’ to netenergymeter or cumulativeenergymeter. This will also change the default and allowed units.
Table 29 Default Senaps stream ids for selected stream classes
Input stream class Default stream id
tamb
irrrad
spotprice
gridemissions
dch.bom.grids.T_SFC_AUS_BOMSTATIONS
dch.bom.grids.AV_SWSFCDOWN_BOMSTATIONS
dch.aemo.public_prices.dregion. REGION1.RRP
aemo_gridemissions_REGION1
Source
Bureau of Meteorology
Bureau of Meteorology
AEMO NEMweb
CSIRO
Table 30 Listing of data processing options associated with default types. Note, empty cells are default value.
A.2 Custom data processing options
Table 31 Summary of custom data processing options that can be applied to individual input streams
Options field
Units/ available values
differentiate True/false
syncmethod Mean, previous, next, nearest, linear
removeduplicates True/False
duplicatecount integer > 1
zerothreshold float
interpolategaps True/False
interpolatewindow Hours
forwardfillgaps True/False
forwardfillwindow Hours
interpmethod Mean, previous, next, nearest, linear
Description
Evaluates the first difference between successive data points and assigns the value to the time corresponding to the second data point
Method use to synchronize data streams to the global data-timestep.
If set to TRUE, then treats sequences of more than duplicatecount repeating identical values as missing.
Number of repeating identical values to allow before duplicate data points are treated as missing points.
Data points with an absolute value less than this number are not treated as duplicates. Set to a negative number to exclude this test.
Flag to enable interpolation between fetched data points.
Maximum gap size (in hours) to interpolate between data points.
Flag to enable forward filling (interpolation) of gaps in the data up until the current time.
Maximum gap size (in hours) to forward fill up until the current time.
Interpolation method applied to interpolation and forward filling as follows: Previous/next: the interpolated value is the value at the previous/next data point. Nearest: the interpolated value is the value at the closest (in time) data point. Linear: linear interpolation between known data points.
upperlimitvalue NaN or numerical value Set to a non-nan value to apply an upper limit (maximum allowed value) to the data. If upperlimitvalue = NaN, then no limit is applied.
upperlimitremovevalue True/False
If set to TRUE, then data values greater than upperlimitvalue are removed (treated as missing).
lowerlimitvalue NaN or numerical value Set to a non-nan value to apply a lower limit (minimum allowed value) to the data. If lowerlimitvalue = NaN, then no limit is applied.
lowerlimitremovevalue True/False
If set to TRUE, then data values less than lowerlimitvalue are removed (treated as missing).
Invert True/False Flag to invert the data values.
scale float
offset Float
Multiplicative (scaling) factor applied to non-nan data values
Fixed offset factor applied to non-nan data values
A.3 Default & allowed units
Table 32 Default and allowed units for specified ‘types’
Type
signedflag - -
emissionsfactor
ambienttemp
thermalmeter
netpowermeter
electricityspotprice
powermeter
irradiance
energymeter
netenergymeter
cumulativeenergymeter
KiloGM-PER-KiloW-HR
DEG_C
KiloW
KiloW
CurrencyUnit-PER-MegaW-HR
KiloW
W-PER-M2
KiloW-HR
KiloGM-PER-KiloW-HR
GM-PER-KiloW-HR
DEG_C, DEG_F
KiloW,W,MegaW
KiloW,W,MegaW
CurrencyUnit-PER-MegaW-HR
KiloW,W,MegaW
W-PER-M2, KiloW-PER-M2
KiloW-HR, W-HR, MegaW-HR, J, KiloJ, MegaJ, GigaJ
A.4 Raw data statistics
Table 33 Listing of reported raw data statistics
Statistic
ndup
count
common_timestep
median_timestep
common_timestep_percentage
upper_prctile
lower_prctile
Description
number of non-zero sequential duplicate data points
total count of data points (including nans)
Most frequently encountered rounded time interval between data points (minutes)
Median of rounded time interval (nearest minute) between data points (minutes)
Percentage of sequential data pairs with timestep equal to the common timestep
99th percentile of non-nan data points
1st percentile of non-nan data points
max
min
zero_count
median
mean
stdev
nan_counts
est_missing_percent
Maximum data point value
Minimum data point value
Count of zero value data points
Median excluding nan data points
Mean excluding nan data points
Standard deviation excluding nan data points
Count of nan data points
Estimated percentage of missing data (both completely missing points and nan points) in the time interval bounding the data points.
A.5 MVApp input parameter listing
Table 34 Listing of all available MVApp parameters
Parameter
Overall
casename string mycase Casename for the M&V run. This is used for writing output files and senaps data streams
selected_streams See §2.4.3.2
datasource string ‘senaps’ Specifies the source of rawdata. Set to ‘senaps’ to using the SenapsAPI to fetch data from senaps. Set to a file name to read a local data file (see §2.4.3.1).
site_lat
Site latitude (degrees North) site_lon
timezone
Program execution
Site longitude (degrees West)
(Continent/City) - If empty, determined from site_lon
data_check_only True/false false If true, program execution ends after data checks are performed.
analysis_type interval, daily, weekly, interval The type of baseline model to use.
analysis_mode MV, DR MV M&V or Demand Response style analysis.
run_baseline
baseline_filename
run_analysis
True/false true If false, will attempt to read a previously computed baseline model from the file specified by baseline_filename
string - Name of file with existing baseline model.
True/false true If false program execution ends after the baseline model is fit.
forecast_period_savings True/false false If true, enables future year savings projections
analysis_start
analysis_end
baseline_start
baseline_end
Forecast_start
Forecast_end
exclude_dates
extra_holiday_dates
dd-MM-yyyy - First day (inclusive) of the analysis (i.e. post-retrofit) period
dd-MM-yyyy - Last day (inclusive) of the analysis (i.e. post-retrofit) period
dd-MM-yyyy - First day (inclusive) of the baseline period
dd-MM-yyyy - Last day (inclusive) of the baseline period
dd-MM-yyyy - First day (inclusive) of the forecast period
dd-MM-yyyy - Last day (inclusive) of the forecast period
List:
dd-MM-yyyy - List of days to exclude from any period
List: dd-MM-yyyy - List of days to treat as public holidays (in addition to Australian public holidays if enabled)
Senaps settings
apikey string - API key for reading data from, and (if enabled) writing output data streams to senaps.
senaps_organisation
senaps_group
senaps_url
Output processing
string - Senaps organisation (only required for writing outputs to senaps).
string - Senaps group (only required for writing outputs to senaps).
string senaps.io Primary link to the Senaps cloud service
writedata_to_senaps True/false False Specifies whether to write application
save_raw_data
True/false False
prepare_word_report
True/false True
prepare_figures
prepare_spreadshhet
True/false True
True/false True
outputs to Senaps data streams.
Specifies whether to write all data (excluding the baseline model) to a .mat output file.
Specifies whether to generate an output report in Word® format. This is only available when running on Windows.
Specifies whether to create & save output figure files.
Specifies whether to save output to an Excel® spreadsheet. This is only available when running on Windows.
prepare_json
Financial & emissions
carbon_credit
carbon_price
network_price_calc
wholesale_price_calc
True/false true
Specifies whether to save selected outputs to json file format.
True/false false If true, net power export to the grid receives a financial credit proportional to the carbon price
$/tonne 50 Price applied to carbon emissions
True/false true If true, activates the calculation of network TOU & demand/capacity power costs.
True/false false If true, activates the calculation of wholesale power costs based on the regional reference electricity price
emissions_calc
export_earn
True/false False If true, activates the calculation of carbon emissions and carbon costs
True/false true
Enables earning from export to the grid.
Export earnings under network tariffs also requires at least one feed-in tariff. Export earnings under wholesale tariffs occur with either net export or negative wholesale prices.
Default emissions factor to apply when real-time emissions data is missing.
use_fixed_emissions_factor
True/false false
Used for managing forecasting of emissions for future savings estimates. If true, the default emissions factor will be
assumed.
tariff_data See §2.4.3.6
tariff_name
Model – expert parameters
use_net_data
True/false false If true, the model attempts to fit directly to net metering data by fitting additional model terms to estimate PV generation.
dt_global minutes 60
estimate_occ
hac_on
True/false true
Overall model analysis time-step (do not change!)
If true, separate models terms are fit based on an inferred binary occupancy status. Applies to interval model only.
True/false false If true, heteroskedastic uncertainty bounds are used
Nboot 2 (currently not adjustable) 2
day_int No. of days 30
minDT DEG_C 1
int_mode 1 or 2 1
ave_occ
Number of bootstrapped forecast trajectories computed
Number of days per model fitting period.
Minimum spacing between piecewise temperature levels
Method of handling intervals of the week
True/false True If true, inferred occupancy pattern is averaged over occ_period weeks period Positive integer (>1) 999
weight_function 1, 2 or 3 1
weight_method 1 or 2 1
fit_holiday_coeff
use_aus_holidays
ignore_dataerrors
Number of whole weeks to average inferred occupancy pattern over
Weighting function for model fitting across periods. 1=quadratic. 2=cubic. 3= cosine.
Weighting function approach. 1=higher weights for similar time of year. 2=higher weights for points closer in time.
True/false false If true an additional additive coefficient is fit in the model for public holiday –occupied times.
True/false true If true includes Australian public holidays in the list of holidays.
True/false true If true, code execution continues ignoring all data errors. confidence percent 90
The desired confidence level applied to output results.
extrapolate_missing_days True/false true If true, aggregate totals are scaled up to account for missing days of data.
extrapolate_whole_months True/false true If true, monthly estimated quantities are extrapolated to whole months by factoring in missing data. This occurs regardless of whether baseline/ analysis periods stop/start part-way through the month.
daily_missing_pct_threshold percent 20
Percentage of intervals in a day (any required variables) that are missing before
monthly_missing_pct_threshold percent 30
nre_missing_pct_threshold percent 25
include_indepvar
occupancy_schedule
scale_factor
cross_validation
enable_interventions
intervention_prescale
interventions
True/false false
See §2.4.3.9
number 1
True/false false
True/false false
See §2.4.4.3
A.6 Output formatting
Table 35 Listing of output figure files
Filename
casename_consumption.emf
casename_consumption2.emf
casename_hourtrend.emf
casename_temperaturetrend.emf
the entire day is considered to be missing.
Percentage of whole days in a month that are missing before the entire month is considered to be missing. Excludes excluded days.
Percentage of whole days in a month (including any excluded days) that are missing before the entire month is considered to be missing.
Whether to include arbitrary independent variables
Allows scaling of netG data by a fixed factor (used for research purposes)
If True enables calculation of selected model metrics using cross-validation.
casename_monthlydifference.emf
Description
Left: timeseries plot of consumption data over the baseline and analysis periods. Right: Cumulative distribution function of consumption over baseline period.
Boxplots of consumption by hour of the day (left) and day of the week (right) for the baseline period.
Variation of actual and model predicted consumption as a function of hour of the day for the analysis period for occupied and un-occupied periods.
Variation of actual and model predicted consumption as a function of ambient temperature for the analysis period for occupied and unoccupied periods.
Scatter plot of the difference between actual and model predicted monthly consumption plotted against actual monthly consumption for baseline and analysis periods. Monthly with statistically significant higher (red) or lower (blue) consumption are marked with ‘’ and ‘’ symbols respectively and are located in correspondingly shaded regions. The significant level is set via the parameter alpha.
casename_monthlyenergy.emf
casename_occupancy.emf
casename_outliers.emf
casename_overallcost.emf
casename_tariff.emf
casename_rawdatastats.emf
casename_rawdatasample.emf
casename_modeldiagnostics.emf
Bar chart of actual and model predicted monthly energy consumption totals for the baseline and analysis periods with uncertainty ranges shown.
Plot of the mean inferred occupancy status for each half-hour interval of the week plotted vs hour of the week.
Timeseries of daily energy delta (actual – model estimated total daily consumption) for baseline and analysis periods. Monthly mean daily energy delta is also shown (solid black line). Dashed lines show +- 3 standard deviations ranges for the daily (black) and monthly (red) energy delta. Days outside these ranges are indicated by symbols.
Bar chart of total actual and predicted costs over the analysis period by category (fixed, feed in, energy, demand, capacity, wholesale and carbon).
Listing of all network tariffs successfully processed from the input configuration file.
Statistics of each raw data stream (prior to data processing operations). See Table 33 for descriptions.
Timeseries plots showing the last 7 days of raw and processed data in the baseline period for each data stream.
Model diagnostic plots. i) residual quantile-quantile plot, ii) residuals histogram, iii) residuals partial auto-correlation function, and iv) actual vs predicted consumption scatter plot.
Field
Description
Timestamp the date/time when the message was written
flag
1 = informational
20 = warning associated with input parameters
21 = warnings associated with senaps i/o
22 = warnings associated with data processing
23 = warning associated with calculation of derived data streams and quantities
24 = level 1 warnings associated with data checks
25 = level 2 warnings associated with data checks
26 = warnings associated with core analysis algorithm
27 = warnings associated with calculating derived results
28 = warnings associated with validity of model
29 = warnings associated with creating output visualisations
30 = warnings associated with writing outputs files
Table 36 Format of logging file messages
2 = other miscellaneous warnings
3 = errors causing program execution to stop or caught exceptions
Message string A character string containing message information. Messages may span multiple lines.
Table 37 List of output quantities in “MVresults_casename.xlsx” spreadsheet
Consumption predicted daily
Import actual
Import predicted
Export actual
Export predicted
Consumption actual Analysis daily
Consumption predicted
Import actual
Import predicted
Export actual
Export predicted
kWh
Model estimated daily total site consumption over the baseline period
Metered daily import energy from the grid over the baseline period
Model estimated daily import energy from the grid over the baseline period
Metered daily export energy to the grid over the baseline period
Model estimated daily export energy to the grid over the baseline period
Metered daily total site consumption over the baseline period
Model estimated daily total site consumption over the baseline period
Metered daily import energy from the grid over the baseline period
Model estimated daily import energy from the grid over the baseline period
Metered daily export energy to the grid over the baseline period
Model estimated daily export energy to the grid over the baseline period
$ Total actual cost of all fixed electricity tariffs for the month.
Feed in actual
Fixed actual Monthly network costs
Energy actual
Demand actual
Capacity actual
Fixed predicted
Feed in predicted
Energy predicted
Demand predicted
Capacity predicted
Wholesale actual Monthly wholesale costs
Wholesale predicted
Carbon cost actual Monthly carbon costs
Carbon cost predicted
Total actual negative cost (i.e. revenue) from energy exported to the grid for the month based on all feed in tariffs.
Total actual cost of all energy tariffs for the month (for example should, off-peak and peak charges)
Total actual cost of all demand charges for the month
Total actual cost of all capacity charges for the month
As above but model predictions instead of actuals.
$ Total actual (net) wholesale cost of electricity for the month based on the Regional Reference Price electricity spot price for the state in which the site is located. If export_earn =false then the cost is based on net import energy only.
As per above but model predicted instead of actual.
$ Total actual (net) cost of emissions based on grid consumed electricity, the specified carbon price and regional grid emissions intensity factor. If export_earn =false then the cost is based on net import energy only.
As per above but model predicted instead of actual.
Carbon emissions Monthly tCO2
Total actual carbon emissions based on grid imported electricity for the month.
(import) actual carbon emissions
Carbon emissions (import) predicted
Carbon emissions (export) actual
Carbon emissions (export) predicted
Total model predicted carbon emissions based on grid imported electricity for the month.
Total actual carbon emissions offset based on electricity exported to the grid for the month.
Total model predicted carbon emissions offset based on electricity exported to the grid for the month.
kWh Actual monthly energy imported from the grid.
Import predicted
Import actual Monthly import export
Export actual
Export predicted
Peak demand actual Monthly peak demand
Peak demand time actual
Peak demand predicted
Model predicted monthly energy imported from the grid.
Actual monthly energy exported to the grid.
Model predicted monthly energy exported to the grid.
kW Top 10 highest half-hourly demand values either during the demand charging window (if specified) or across all times.
- Correspond times of top 10 highest half hourly demand values
kW Predicted top 10 half-hourly demand values average values (note times do not necessarily coincide with actual peak demand times). Values are calculated from averaging multiple boot-strapped simulations.
A.9 Listing of quantities in output json files
coefficient of determination of the regression model evaluated over the testing data.
Coefficient of variation of the root mean square error.
- Normalised mean bias error
Qualitative model quality indicator checks various Results of model tests. See below checks.r2_met
checks.cvrmse_met
checks.nmbe_met
checks.alternate_model_pct_deviation
R2 test pass outcome
CVRMSE test pass outcome
NMBE test pass outcome
Difference in estimated baseline energy for alternative model fit with no excluded data periods.
boolean Outcome of test for presence of residual auto-correlation (true=autocorrelation present)
boolean Outcome of test for seasonal trend (true=trend present)
boolean Outcome of test for time trend (true=trend present)
Integer
Number of over and under-estimated months in the baseline period.
integer Number of over and under-estimated days in the baseline period
Vector of booleans
Flag indicated whether each month has violated the missing data criteria (true = violated)
boolean Flag indicating whether the non-routine events missing data criteria has been violated (true =violated)
%
Percentage of missing days in the baseline and analysis periods.
boolean Flag indicating whether a seasonal trend has been identified in the estimated savings
Vector of datetimes
List of identified days where estimated energy savings are much greater/less than surrounding days.
integer Count of number of savings outlier days
checks.nonroutineperiodssavings integer Count of number of separate periods where savings appear to be significantly different from the trend
checks.pvcheck
total_b_energy_pre
total_b_energy_act
total_b_deltaenergy_pre
Integer Result pv check for presence of pv generation in consumption data. 0 = no pv generation detected. 1= possible detection. 2= high probability of detection.
kWh
kWh
kWh
claimable_savings %
issues
Baseline model estimated total energy consumption over the baseline period.
Total actual energy consumption over the baseline period (may include estimates for missing periods, depending on settings)
Estimate uncertainty in model estimate total energy consumption based on specified confidence level.
Estimated minimum savings identifiable by the model.
string array List of potential issues with baseline model fitting process and input data
baseline_start datetime Start of baseline period
baseline_end datetime End of baseline period
Table 39 Listing of parameters in ‘casename_analysis_summary.json’
Estimated total energy weighted saving over the analysis period.
Eligible fuel (energy) savings over the analysis period.
Model uncertainty as a percentage of estimated savings.
Claimable energy savings over the analysis perido given the model precision.
Estimated percent energy saving over the analysis period.
Averages of various quantities as a function a grouping variable such as ambient temperature or hour of the day
Vector of degC Centre of each ambient temperature level
Matrix of kWh Mean of actual (measured) analysis or baseline period energy use for occupied and unoccupied periods and corresponding to ambient temperatures in each temperature level.
Matrix of kWh Mean of estimated analysis or baseline period energy use for occupied and unoccupied periods and corresponding to ambient temperatures in each temperature level.
Matrix of kWh Mean of estimated analysis or baseline period energy use uncertainty for occupied and unoccupied periods and corresponding to ambient temperatures in each temperature level.
group_means.hrofdaygroups Various Centre of each hour of the day level
group_means.hrofday_b_energy_act group_means.hrofday_a_energy_act Etc.
Various As per ambient temperature group means.
Table 40 Listing of parameters in ‘casename_detailed_results.json’
Variable name Units / type Description
daily_b_Time, daily_a_Time Vector of datetime
Date of each day in baseline or analysis periods.
daily_b_energy_act, daily_b_energy_pre, Vector Daily energy actual (measured) or predicted for
daily_a_energy_act, daily_a_energy_pre of kWh each day of the baseline or analysis periods.
Percentage of missing data over the month (including excluded date) (does not consider the whole month if the baseline/analysis period starts/ends part-way through the month).
Monthly CO2 equivalent emissions (actual and predicted) from imported energy over each month in the baseline and analysis periods.
Monthly CO2 equivalent avoided emissions (actual and predicted) from exported energy over each month in the baseline and analysis periods.
A.10 Weather station listing
As Australia’s national science agency and innovation catalyst, CSIRO is solving the greatest challenges through innovative science and technology.
