Big data processing and use cases

Abstract

Hi all, We’re happy to have Gauthier Lyan as our next DiverSE Coffee speaker. He’ll be introducing a couple of frameworks for big data processing and use cases. In the Agenda: (i) Apache NiFi for Data flow integration (with a few examples and use cases). (ii) Apache Spark & its architecture (clustering, scalability, limitations). (iii) How to use it locally (because it is a genius tool for data engineering, even for small datasets)?.(iii) How to simulate a cluster on a computer that has numerous cores and a lot of memory to bypass some local limitations and enhance CPU intensive tasks performance?. (iv) How to test you data engineering with a framework I made for my work? The presentation is held next Thursday, somewhere in the dark side of the lab (Room Bréhat!) at 1 pm.