GCP allows Data Engineers to effectively manage the data pipeline with tools like Facets for analysis. It does not replace experts, but offers a centralized environment. Aqsone is exploring its use to assess its performance and architectural benefits in its internal projects.
Each month, during a KLS (Knowledge Lunch Session), an Aqsone collaborator presents to all collaborators a technical subject on which he has worked or is in the process of being trained.
For the month of July, Matthieu Vinette, Data Engineer, presented us with the advantages that Aqsone could get from working with the Google GCP platform.
Indeed, this platform offers the possibility for Data Engineers to be able to create and manage the entire Data Pipeline, from the ingestion of raw data to the provision of cleaned data to Data Scientists.
For example, GCP provides advanced and interactive data and model analysis tools such as Facets (open source). Facets allows you to visualize your data, characteristic by characteristic, and even allows you to work with images intuitively.
Such tools are a real plus in accelerating the data cleaning and de-biasing process.
Absolutely not!
GCP does not spell the end of Data Engineers. On the contrary, it offers them a new environment to understand that centralizes all the tools essential to their work.
We intend to use it as a data science platform in the work of the Aqsone Lab in order to be able to concretely measure, on internal projects, its possible contributions, whether in terms of performance gains or in facilitating the architecture of our projects.
