Autonomous Agentic AI for Legacy-to-Cloud ETL Migration
Keywords:
agentic AI, ETL migration, cloud-native patterns, swarm intelligence, reinforcement learning, orchestration optimizationAbstract
The objective of this research is to presents an autonomous agentic AI framework for legacy-to-cloud ETL migrations. A distributed network of smart agents’ inventory ETL pipelines, finds schema and execution dependencies, and builds cloud-native data integration patterns. The architecture improves workload-based orchestration, data partitioning, and storage tier selection using reinforcement learning (RL).
Downloads
References
G. H. K. Lee, A. Y. Zomaya, and P. S. Yu, "Cloud computing: Principles and paradigms," Wiley, 2014.
F. T. Chong, J. Kephart, D. A. Patterson, et al., "Software-defined infrastructure for big data analytics," IEEE Computer, vol. 51, no. 7, pp. 14-23, Jul. 2018.
S. Ghemawat, H. Gobioff, and S.-T. Leung, "The Google file system," ACM SIGOPS Operating Systems Review, vol. 37, no. 5, pp. 29-43, 2003.
R. Ranjan, "Architecting big data applications and platforms," IEEE Cloud Computing, vol. 2, no. 2, pp. 22-27, Apr.-Jun. 2015.
A. Mariani and S. Ceri, "Modern data integration," in Encyclopedia of Database Systems, Springer, 2018.
J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S. Bae, J. Qiu, and G. Fox, "Twister: a runtime for iterative mapreduce," in Proc. 19th ACM Int. Symposium on High Performance Distributed Computing, 2010, pp. 810–818.
T. Chen, K. Liao, and P. Liu, "Reinforcement learning based dynamic resource allocation in cloud computing," in Proc. IEEE International Conference on Cloud Computing and Intelligence Systems, 2016, pp. 538-542.
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 3rd ed., Pearson, 2010.
J. Leskovec, A. Rajaraman, and J. D. Ullman, Mining of Massive Datasets, Cambridge University Press, 2014.
S. Dasgupta, D. Rafiei, and K. Kambatla, "Data migration techniques and tools for data warehousing," Journal of Information Systems, vol. 31, no. 2, pp. 1-12, 2017.
X. Liu, Y. Cui, C. Wang, and B. Zhang, "A survey on cloud computing migration," Journal of Computer and System Sciences, vol. 111, pp. 1-13, 2020.
K. B. Laskey, "Semantic web services and ontology-based data integration," IEEE Internet Computing, vol. 12, no. 6, pp. 68-72, 2008.
M. Wooldridge, An Introduction to MultiAgent Systems, 2nd ed., Wiley, 2009.
J. Kennedy and R. Eberhart, "Particle swarm optimization," in Proc. IEEE International Conference on Neural Networks, 1995, pp. 1942-1948.
C. Watkins and P. Dayan, "Q-learning," Machine Learning, vol. 8, no. 3-4, pp. 279-292, 1992.
V. Mnih, K. Kavukcuoglu, D. Silver, et al., "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540, pp. 529-533, 2015.
D. Silver, A. Huang, C. J. Maddison, et al., "Mastering the game of Go with deep neural networks and tree search," Nature, vol. 529, no. 7587, pp. 484-489, 2016.
M. H. Nguyen and N. M. Kim, "Data lineage in big data environments: State of the art and research challenges," IEEE Access, vol. 8, pp. 55345-55367, 2020.
A. V. Dastjerdi and R. Buyya, "Fog computing: Helping the Internet of Things realize its potential," Computer, vol. 49, no. 8, pp. 112-116, 2016.
D. G. Andersen, "The case for cooperative OS-level caching," in Proc. 2002 USENIX Annual Technical Conference, 2002, pp. 15-28.
