DATA ARCHITECTURES FOR ANALYTICS: INTEGRATION PATTERNS, ETL/ELT, AND PIPELINE SUPPORT
DOI:
https://doi.org/10.56238/levv13n31-081Keywords:
Data Architectures, Analytics, Data Integration, ETL, ELT, Analytical PipelinesAbstract
This article analyzes data architectures oriented toward analytics, with emphasis on integration patterns, ETL and ELT approaches, and mechanisms for sustaining analytical pipelines, considering the growing organizational need to structure information flows capable of integrating multiple sources, sustaining continuous processing, and making data available with greater consistency for analytical use. The research was developed through a qualitative approach, with descriptive and explanatory character, supported by bibliographic research, through which theoretical contributions related to data warehouse, data lake, lakehouse, data integration, governance, and the operation of analytical flows were examined. The results indicate that the evolution of data architectures is associated with the search for greater structural flexibility, greater integration capacity, and better coordination between storage, processing, and analytical consumption, revealing that the efficiency of analytics environments depends on the alignment between architectural model, processing strategy, and mechanisms for sustaining pipelines. It was also found that ETL and ELT correspond to distinct operational logics, whose adoption varies according to volume, diversity, update frequency, and analytical purpose of the data, while pipeline continuity requires monitoring, traceability, metadata, and articulated governance. It is concluded that data architectures oriented toward analytics need to be understood as integrated ecosystems, in which structure, flow, and operational control act interdependently in the production of reliable and scalable analytical environments.
Downloads
References
DHAOUADI, Asma; BOUSSELMI, Khadija; GAMMOUDI, Mohamed Mohsen; MONNET, Sébastien; HAMMOUDI, Slimane. Data warehousing process modeling from classical approaches to big data and ELT: state of the art and future trends. Data, v. 7, n. 8, 2022.
FIKRI, Noussair; RIDA, Mohamed; ABGHOUR, Noureddine; MOUSSAID, Khalid; EL OMRI, Amina. An adaptive and real-time based architecture for financial data warehouse. Journal of Big Data, v. 6, 2019.
GIL, Antonio Carlos. Métodos e técnicas de pesquisa social. 6. ed. São Paulo: Atlas, 2008.
HARBY, Ahmed A.; ZULKERNINE, Farhana. From data warehouse to lakehouse: a comparative review. In: IEEE INTERNATIONAL CONFERENCE ON BIG DATA, 2022. Anais [...]. IEEE, 2022.
LAKATOS, Eva Maria; MARCONI, Marina de Andrade. Fundamentos de metodologia científica. 5. ed. São Paulo: Atlas, 2003.
MACHADO, Gustavo V.; CUNHA, Ítalo; PEREIRA, Adriano C. M.; OLIVEIRA, Leonardo B. DOD-ETL: distributed on-demand ETL for near real-time business intelligence. Journal of Internet Services and Applications, v. 10, 2019.
NADAL, Sergi; JOVANOVIC, Petar; BILALLI, Besim; ROMERO, Oscar. Operationalizing and automating data governance. Journal of Big Data, v. 9, 2022.
NAMBIAR, Athira; MUNDRA, Divyansh. An overview of data warehouse and data lake in modern enterprise data management. Big Data and Cognitive Computing, v. 6, n. 4, 2022.
SAWADOGO, Pegdwendé; DARMONT, Jérôme. On data lake architectures and metadata management. arXiv, 2021.
WIEDER, Philipp; NOLTE, Hendrik. Toward data lakes as central building blocks for data management and analysis. Frontiers in Big Data, v. 5, 2022.