GridLearn: Multiagent reinforcement learning for grid-aware building energy management

Highlights

• Feasibility study of multiagent reinforcement learning for voltage regulation.

• Multiagent reinforcement learning for demand side management with power flow analysis.

• Open source building energy management simulation.

Abstract

Increasing amounts of distributed generation in distribution networks can provide both challenges and opportunities for voltage regulation across the network. Intelligent control of smart inverters and other smart building energy management systems can be leveraged to alleviate these issues. GridLearn is a multiagent reinforcement learning platform that incorporates both building energy models and power flow models to achieve grid level goals, by controlling behind-the-meter resources. This study demonstrates how multi-agent reinforcement learning can preserve building owner privacy and comfort while pursuing grid-level objectives. Building upon the CityLearn framework which considers RL for building-level goals, this work expands the framework to a network setting where grid-level goals are additionally considered. As a case study, we consider voltage regulation on the IEEE-33 bus network using controllable building loads, energy storage, and smart inverters. The results show that the RL agents nominally reduce instances of undervoltages and reduce instances of overvoltages by 34%.

Introduction

Towards pursuing a reliable and cleaner energy system, solar photovoltaic (PV) generation is being increasingly connected into electricity distribution networks. In 2019, there was estimated to be 23.2 GW of small-scale PV in the US [1], and this number is rapidly increasing each year. Distributed generation is an effective method for reducing carbon emissions from the electricity sector. However, the integration of PV into a distribution network can cause voltage issues [2], so there is a limit to the amount of solar that can be installed without intervention [3], [4].

The cost of upgrading distribution networks to increase PV penetrations is high, however the use of smart inverters and intelligently shifting building loads to overlap with solar production can avoid upgrades in many cases [5]. Smart inverters can vary the ratio of real and reactive power that is exported to the network, thus providing voltage support (known as volt-VAR control). Additionally, it has been suggested that energy storage and demand response could be used for voltage regulation [6]. Therefore, it is likely that a building with flexible loads and/or smart inverter-connected PV could be exploited, to lessen voltage issues in distribution networks with a high PV penetration.

Various methods have been proposed for controlling assets to achieve voltage regulation in distribution networks. In [7] two centralized control methods are proposed. However, centralized control algorithms typically require a full distribution network model, while utilities may not have accurate network models. Centralized control schemes also suffer from issues with privacy and response time [8].

Decentralized methods have also been investigated for use in voltage regulation, where the control algorithm is applied individually by each inverter/household. Standard volt-VAR controls determine a policy for reactive power injection based on the voltage at the point of connection. These do not require a full network model, however they can sometimes cause unintended oscillatory behavior [9]. More sophisticated decentralized methods, such as using particle swarm optimization [10], or a consensus negotiation [11] have also been proposed. However these, in addition to many decentralized methods that improve upon traditional volt-VAR, require an accurate model of the distribution network [12].

Reinforcement learning (RL) allows the control agents to learn from experience on the network, meaning a network model is not required, and agents can dynamically adapt to a changing environment. This means that, unlike previously discussed approaches, the algorithm will adapt when other buildings are taken on/offline or modified. Centralized RL has been previously investigated for voltage regulation [13], [14]. However, both only consider optimization of reactive power components; in [13] demand is considered to be constant, while in [14] load is inflexible. In reality, the flexibility provided by appliances or battery energy storage can also aid voltage regulation. It is also important to include varying electricity demand because voltage issues tend to occur at specific times of day, rather than constantly. Finally, these methods also suffer from the aforementioned drawbacks of centralized control algorithms.

Towards intelligent load shifting and demand response coupled with building-level energy goals, CityLearn is an OpenAI-Gym environment that allows for the implementation of RL strategies to achieve building and community level goals [15], [16]. However, objectives such as grid-level voltage regulation cannot currently be considered in CityLearn, as no power network model is included in the environment. Without the consideration of grid-level objectives and constraints, buildings are subsequently limited in the amount of demand response they can provide-constrained by grid-level voltage, current limits, and more. Shifting building loads across an entire distribution network can also have adverse effects on the grid, such as on voltages, as will be demonstrated in this paper.

To overcome issues that arise from grid unaware building energy management, we develop a model-free multi-agent reinforcement learning (MARL) framework that addresses both building-level and grid-level objectives, building upon the powerful CityLearn framework. MARL is preferential to a centralized reinforcement learning approach, because it considers privacy constraints of each building owner and can utilize distributed computing. Previous works have used MARL for other building energy management objectives [17], [18], [19] however these works do not include necessary power flow models. Here we expand environment model provided by the CityLearn framework to include PV inverters and a voltage regulation objective, which requires power flow analysis to be incorporated into the environment. Additionally in the agent models, unlike [18] we incorporate synchronous action selection, which limits the RL agents’ ability to identify the strategies of other agents [20]. Given that all buildings are working towards a communal goal, cooperation between buildings should be incentivized. A lack of coordination could result in over-correcting for whatever issue they may face (e.g. harmonic distortion in [21] and extraneous switching in [2]). However, we have chosen to analyze the worst case scenario, in which building operators cannot coordinate with other buildings in advance.

Therefore, the contributions of this paper can be summarized as follows. First, we develop a flexible framework (GridLearn) for training RL agents to satisfy grid-level goals while also considering building-level goals and demand response capabilities, building on the existing open-source CityLearn framework. Second, we investigate the use of MARL for voltage regulation, an important emerging issue in high-penetration PV networks, on the IEEE 33-bus network. Third, we validate the use of MARL for building energy management in the extreme case where actions are implemented simultaneously, which can potentially cause oscillatory behavior (e.g., the “rebound effect” [22]).

The rest of this paper is organized as follows: in Section 2 we provide context for the reinforcement learning algorithms used; in Section 3 we review the energy models from CityLearn and provide a new power flow model; in Section 5 we provide a brief study of MARL using a voltage deviation objective; finally, in Section 6 we consider other applications for the framework introduced in this paper.