In our work we investigate different Deep Reinforcement Learning (DRL) approaches to provide a local and proactive network control management to guarantee specific QoS requirements. These requirements are characterized by multiple aspects such as transmission delays, propagation delays, links utilization and failure events. Our idea aims to analyze a specific network scenario, focusing the attention on an aspect listed above, trying to maintain a stable network performance during critical situations. Such situations will be analyzed by a DRL-based framework, in which we will train an agent to react autonomously and proactively to avoid further deterioration in network performance. The agent will be a single node in the network or multi-node in the case of multi-agent, who will have the ability to modify the environment observed through the application of actions (e.g. re-routing operation) that will restore stability to the network, maintaining QoS requirements. Performance evaluations should consider the improvement achieved by the agent (or multi-agent) with respect to the observed critical parameter, and there should be a comparison of this Reinforcement Learning methodology against the heuristic solutions already available. Finally, the differences between single agent and multi-agent will be evaluated, to understand which network and agent location characteristics guarantee improvements in results, maximizing coordinated decision behavior.
1) Evaluation of different Reinforcement Learning models, assessing the best one with respect to the analyzed problem.
2) Advancement of the state of the art in DRL by proposing modifications to the currently most used models: Value Based, Policy Based and Actor Critic. Starting with the theory by Sutton and Barto[1], and studying the works developed by OpenAI team.
3) A first study of multi-agent with respect to the analyzed problem. In the literature a much debated topic is the realization of models with multi-agents, where one of the main problems turns out to be the management and coordination of the latter. One of our developments will be to evaluate different methods for characterizing the reward (factor that rewards or penalizes the agent) in the case of coordination. We will consider a reward function that is able to capture globally the changes made by actors acting in the environment, but at the same time is able to evaluate agents based on possible local improvements. The idea will be to find a common thread between Reinforcement Learning and Games and Equilibria.
4) Analysis of the project will lead us to evaluate possible alternatives to the proposed methodology, investigating viable alternatives. Certainly one area we will be repeatedly confronted with is heuristics field. The comparison will have to take into account the different factors according to the problem considered, first and foremost the time evaluations and the feasibility of the proposed methods. Then the different considerations will have to focus on the comparison of performance in the different situations analyzed. The idea will be to emphasize advantages and disadvantages of the compared methodologies.
5) Creation of a real testbed, replicating what is made available on the OpenIA GYM library. To concretely evaluate our proposal and its feasibility. This last evaluation represents one of the most important steps of the project, in which we will challenge our skills in harnessing the power of neural networks on hardware that allows a realistic implementation of the project.
[1] A. G. Sutton, Richard S.; Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.