Freelance DATA : Distinguez les savoir-faire nécessaires à votre projet

Aperçu des missions de Mehdi,
freelance DATA habitant Paris (75)

• Société générale – DSI CFTParis
  Jan 2021 - aujourd'hui

  Project « Finance District » :
  
  Mission: Finance District supports the business areas of the CFT IT department in the design and
  implementation of solutions related to data
  Position: Technical Lead Big Data Engineer.
  
  Lead tasks:
  ❖ Daily technical support for data engineers.
  ❖ Implementation of a monitoring module for the execution plan of Quartier Finance.
  ❖ Design of a data quality management module.
  ❖ Design of a production job monitoring module.
  ❖ Supervision of the technical and functional migration from Talend Big data to Scala/Spark.
  ❖ Ensuring the migration of Spark 2.1 projects to 2.4.
  ❖ Implementation of a data anonymization solution.
  ❖ Implementation of the Scala/Spark CI/CD pipeline.
  ❖ Ensuring the migration from Hortonworks to CDP.
  Technical tasks:
  • Analyzing and implementing "hot fixes" in production.
  • Implementation of a Scala/Spark framework to facilitate and standardize Scala/Spark
  developments.
  • Developing a solution for sending files via WebHdfs.
  • Tuning the performance of Scala/Spark applications for batch interval, parallelism, and memory.
  • Optimization of existing algorithms in Hadoop using SparkSession, Spark-SQL, Data Frames,
  and Pair RDDs.
  • Manipulating large datasets using Partitions, Spark memory capabilities, Spark Broadcasts, and
  efficient Joins.
  • Developing audit logic to optimize the load in append mode.
  • Developing solutions for pre-processing large sets of structured and semi-structured data with
  different formats (text files, Avro, sequence files, JSON records).
  • Use of Parquet, ORC, Avro files according to the technical need.
  • Study of the choice of Spark partitions for "write HDFS" and the calculation of the coalesce to
  avoid the shuffle
  • Implementation of performance improvements based on job monitoring via SparkUI
  • Adding "persist" via Spark and "serialization" of data as needed
  • Use of RDD, DataFrame, DataSet according to the technical need.
  • Development of some UDF on Spark 2.4
  • Deployment and Orchestration of the project via ControlM
  • Use of scalaTest to ensure unit tests and code coverage at the SonarQube level
  • Development of test cases, test scenarios with TDD logic using Cucumber.
  • Log recovery via Scala/Kafka & Spark Streaming from YARN for analysis and monitoring of
  the application.

• aujourd'hui

  Project « PRORATA VAT »:
  Mission: Equip accounting and fiscal business with a digital solution for calculating the "PRORATA
  TVA" tax base
  Position: Technical Lead Big Data Engineer.
  Lead tasks:
  ❖ Facilitate or lead agility ceremonies.
  ❖ Implementation of the Data solution architecture using "Scala/Spark" and establishing a link
  with visualization tools.
  ❖ Optimization of the performance and scalability of "Scala/Spark" data systems for jobs.
  ❖ Ensuring the quality and reliability of data.
  ❖ Synchronize with web restitution and PowerBI teams.
  ❖ Collaborate with security teams to ensure the security of data.
  ❖ Definition of standards and best practices for data projects.
  Technical tasks:
  • Writing technical specifications.
  • Definition of big data architecture.
  • Scala/Spark development of calculations for the "prorata tva" bases.
  • Implementation of Scala/Spark jobs.
  • Development of data extraction jobs from APIs.
  • Analyze, design and build Modern data solutions using Azure PaaS service to support visualization
  of data. Understand current Production state of application and determine the impact of new
  implementation on existing business processes.
  • Extract Transform and Load data from Sources Systems to Azure Data Storage services using a
  combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake
  Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage,
  Azure SQL, Azure DW) and processing the data in In Azure Databricks.
  • Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and
  load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, writeback tool and backwards.
  • Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation
  and aggregation from multiple file formats for analyzing & transforming the data to uncover
  insights into the customer usage patterns.
  • Responsible for estimating the cluster size, monitoring and troubleshooting of the Spark
  databricks cluster.
  • Experienced in performance tuning of Spark Applications for setting right Batch Interval time,
  correct level of Parallelism and memory tuning

• aujourd'hui

  Project « C3S TAX »:
  Mission: Proposal for a technical data/big data and Azure Cloud architecture to meet the need for
  industrialization of the C3S tax calculation.
  Position: Data Architect
  Architect tasks:
  ❖ Define an architectur...