Deep reinforcement learning (DRL) integrates the reinforcement learning's decision-making ability and deep learning's feature representation ability to attain robust end-to-end learning control capabilities.
Image Credit: chaylek/Shutterstock
In the last decade, DRL has achieved significant advances in multiple tasks that require making near-optimal/optimal decisions and perceiving high-dimensional input. This article deliberates on the major applications of DRL in diverse fields and future directions of this technology.
DRL in Economics
In economics, the popularity of DRL has increased substantially as this technology is characterized by scalability and offers significant opportunities for sophisticated dynamic economic systems' handling. DRL can be applied to high-dimensional problems along with nonlinear and noisy economic data patterns.
The technology provides higher efficiency and better performance compared to conventional algorithms while facing genuine economic problems in the presence of ever-increasing uncertainties and risk parameters. DRL possesses the ability to capture substantial market conditions for providing the best economic strategy.
DRL in Stock Trading
DRL methods like deep deterministic policy gradient (DDPG), adaptive DDPG, recurrent convolutional neural network (RCNN), and deep Q-networks (DQN) have been utilized for different stock market applications. For instance, the DDPG algorithm was used to identify the optimal strategy in dynamic stock markets.
The components of this algorithm handle large action-state space, enhancing data utilization, removing sample correlation, and ensuring stability. Results showed the effectiveness of this model in equilibrating risk and performing better compared to conventional approaches with a higher return guarantee.
Similarly, better investment strategies can be attained using DQN models to optimize the return in stock trading. A new adaptive DDPG was designed to detect optimal strategies in complicated and dynamic stock markets. This model combined a pessimistic and optimistic DRL that depends on both positive and negative forecasting errors.
The model possesses the ability to gain better portfolio profit under complicated market situations based on the Dow Jones stocks. An RCNN approach was employed to forecast stock value from economic news, while a deterministic DRL method based on cryptocurrency was used to identify the optimal strategy in financial problem settings. In a recent study, a novel model-based DRL scheme was designed for automated trading to take action and sequentially make decisions associated with global goals.
This model architecture consists of a generative adversarial data augmentation module (DAM), an infused prediction module (IPM), and a behavior cloning module (BCM) dealing with designed back-testing. Empirical results obtained using historical market data showed the stability of the model. The model also gained more return compared to baseline approaches and other model-free methods. Portfolio optimization is a difficult task during stock trading in the market.
A novel RL architecture associated with a risk-sensitive portfolio combining IPM has been used to predict the stock trend with historical asset prices to improve the RL agent performance, while DAM and BCM have been utilized to address the over-fitting problem and to retain the portfolio with low volatility and handle sudden movement in portfolio weights, respectively. Results displayed that this complex model is more profitable and robust compared to previous approaches.
A study focusing on real-time bidding (RTB) applied to sponsored search (SS) auction in a complex stochastic environment related to user action and bidding policies developed the SS-RTB model that engages RL concepts to adjust an effective Markov Decision Process (MDP) model based on a proper aggregation level of datasets from the auction market in a changing environment.
The feasibility of this method has been validated in both online and offline evaluation based on the Alibaba auction platform. In online advertising, addressing the bidding optimization problem is a significant practical challenge. The SS-RTB approach can be used to effectively handle sophisticated changing environments related to bidding policies.
DRL in Computer Vision
DQN, DDQN, duel DQN, duel DDQN, and actor-critic-based partial-policy RL are employed for single landmark detection, while DQN and collab DQN are used for multiple landmark detection. Visual object detection can be achieved using policy sampling and state transition algorithms.
Active object localization, hierarchical object detection, tree-structured sequential object localization, multi-task object localization, bounding box automated refinement, efficient object detection in large images, organ localization in CT, and monocular three-dimensional (3D) object detection are realizable using DQN.
The optimal search strategies to find anatomical structures can be learned based on the image data at several scales using the capabilities of scale-space theory and DRL. In this approach, the search begins at the coarsest scale level to capture the global context and then continues to finer scales to capture more local information.
Many DQN RL models have been utilized to train agents that accurately localize target landmarks in medical scans. Active object localization was achieved using DRL by considering MDP as the problem-solving framework. Eight separate actions, including taller, fatter, bigger, smaller, right, left, down, and up, were considered to improve the bounding box's fit around the object and an additional action for triggering the goal state.
An object detection method was developed in a study based on a sequential search strategy using DRL. Active lesion detection in the breast can be achieved using a DRL method by formulating the detection problem as an MDP. In the formulation, nine actions, including one trigger action, two scaling actions, and six translation actions, were utilized.
In object tracking applications, the actor-critic method is suitable for end-to-end active object tracking, tracking with iterative shift, and visual tracking, while DQN is effective in dual-agent deformable face tracking, collaborative multi-object tracking, multi-object tracking in video, and multi-agent multi-object tracking.
DRL in Production Systems
In production systems, DRL is applied to a variety of tasks, including process control, production scheduling and dispatching, intralogistics, assembly, robotics, maintenance, energy management, process design, and quality control.
In process control, DRL algorithms like DDPG, actor-critic, and DQN are used in batch process, brine injection process, liquid molding process, chemical microdroplet reactions, color fading, continuously stirred tank reactor, interacting tank liquid level control, double dome draping, general discrete-time processes, goethite iron removal process, hematite iron ore processing, laser welding, one-stage mineral grinding, propylene oxide batch polymerization, single-cell flotation process, tempered glass manufacturing, and well surveillance.
Similarly, in production scheduling and dispatching and intralogistics, algorithms like DQN, DDPG, double DQN, and dueling DQN are used in cloud manufacturing, dynamic scheduling, job-shop scheduling, mold scheduling, multi-chip production, packaging line scheduling, paint job scheduling, parallel and reentrant production, rescheduling, single machine scheduling, general job-shop, wafer fabrication, WIP bounding, AGV scheduling, QoS service composition model, syringe filling process, and three-grid sorting system.
Moreover, in assembly, DRL algorithms are used for sequence planning, high-precision insertion, insertion tasks, plug insertion tasks, and shoe tongue assembly, while in robotics, DRL algorithms are employed for intelligent gripping, motion planning, and visual control. In maintenance, DRL is utilized in condition-based maintenance, machine fault diagnosis, opportunistic maintenance, selective maintenance, self-diagnosis and self-repair, and sensor-driven maintenance.
Energy system balancing, multi-agent energy optimization, network resource management, PCB order acceptance, sustainable joint energy control, clamping position optimization, computer-aided process planning, integrated circuit design, rectangular item placement, and SaaS remote training are the major DRL applications in energy management.
Challenges and Future Directions
Despite the extensive application of DRL in diverse fields, many challenges still exist while using this method, including reward specification, generalization, model-based learning, sample complexity, hyperparameter tuning, scalability, efficiency, and stability. Thus, future studies must focus on performing systematic, comparative studies of DRL algorithms, developing groundbreaking applications for DRL, and ensuring that the method learns from both entities and raw inputs.
The goal is to increase the efficiency of the learning system concerning space, time, and sample to attain interpretability and prevent obvious mistakes. If the raw data could be processed with some knowledge or principle, the resulting representation would be more convenient for the learning system to make further decisions or predictions.
Overall, DRL is revolutionizing various fields like economics, computer vision, and production systems by enabling solutions to complex tasks. However, it faces challenges like reward specification and needs further research for better efficiency and interpretability.
References and Further Reading
Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., Dai, B., Miao, Q. (2022). Deep reinforcement learning: A survey. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/TNNLS.2022.3207346
Mosavi, A., Faghan, Y., Ghamisi, P., Duan, P., Ardabili, S. F., Salwana, E., & Band, S. S. (2020). Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics. Mathematics, 8(10), 1640. https://doi.org/10.3390/math8101640
Li, Y. (2018). Deep Reinforcement Learning. ArXiv. https://doi.org/10.48550/arXiv.1810.06339
Le, N., Rathour, V. S., Yamazaki, K., Luu, K., Savvides, M. (2022). Deep reinforcement learning in computer vision: a comprehensive survey. Artificial Intelligence Review, 1-87. https://doi.org/10.1007/s10462-021-10061-9
Panzer, M., Bender, B. (2022). Deep reinforcement learning in production systems: a systematic literature review. International Journal of Production Research, 60(13), 4316-4341. https://doi.org/10.1080/00207543.2021.1973138