Alibaba's big data platform evolved to handle massive transaction volumes from events like the Double 11 festival, processing petabytes of data daily.
The platform is structured around a typical data flow, including tools for data ingestion, storage, processing, visualization, and a dedicated AI platform called PAI.
A look at key processing tools including MaxCompute for massive-scale SQL, E-MapReduce for Hadoop migrations, and Flink for real-time stream processing.
DataWorks provides a single environment for data development and operations, integrating various engines like MaxCompute and Hadoop and including data governance tools.
The PAI platform supports both visual drag-and-drop model building with PAI Studio and code-based development using the Data Science Workshop (DSW) with JupyterLab.
An overview of the AI Champion 2020 competition, which offers cash prizes and cloud credits for innovative projects built on the PAI platform.
A walkthrough of the DataWorks interface, demonstrating how to write and execute SQL queries, visualize results, and manage data integration jobs.
A demonstration of building a recommendation model using the drag-and-drop PAI Studio and running a deep learning example in the PAI-DSW JupyterLab environment.
43:26Ayon Roy
46:43Bobur Umurzokov
29:37Stefan Donsa & Lukas Alber
45:58Andreas Christian
1:09:49Simi Olabisi
41:13Cheng Zhang
58:21Julian Joseph
26:12Ayoub Alouane
Jobs that call for the skills explored in this talk.
