Implementing continuous delivery in a data processing pipeline
What if you could roll back a data pipeline instantly, with zero state migration? See how treating data as an immutable, versioned artifact makes this a reality.
#1about 4 minutes
From research concepts to production-ready data products
The Volkswagen Data Lab shifted its focus from demonstrating proof-of-concepts to building and deploying real-world data solutions for its clients.
#2about 7 minutes
Core concepts of continuous delivery for data
Continuous delivery for data pipelines requires adapting standard CI/CD principles, where data is the deliverable, by progressing through version control, integration, and deployment stages.
#3about 11 minutes
Implementing a pipeline with immutable, versioned data
The five-step pipeline relies on treating data as immutable, creating a new versioned output for each run to enable simple rollbacks and reproducibility.
#4about 6 minutes
The challenge of orchestrating chained data jobs
Managing dependencies between jobs becomes complex when each job consumes versioned, immutable data inputs from upstream processes.
#5about 5 minutes
Pros and cons of the immutable data approach
While this method offers powerful benefits like reproducibility and instant rollbacks, it introduces challenges in orchestration complexity and increased storage costs.
Related jobs
Jobs that call for the skills explored in this talk.
Matching moments
00:36 MIN
The distinct roles of CI and CD pipelines
#90DaysOfDevOps - The DevOps Learning Journey
06:04 MIN
Using continuous delivery to enable business agility
The Affordances of Quality
01:29 MIN
Overview of the data and machine learning tech stack
Empowering Retail Through Applied Machine Learning
09:45 MIN
Implementing an AI-in-the-loop continuous learning cycle
A solution to embed container technologies into automotive environments
04:09 MIN
Automating the data pipeline with multi-cloud services
Leverage Cloud Computing Benefits with Serverless Multi-Cloud ML
00:51 MIN
Defining continuous integration, delivery, and deployment
CI/CD with Github Actions
02:13 MIN
Using AI to optimize CI/CD pipelines
Navigating the AI Wave in DevOps
03:47 MIN
Applying software engineering practices to data pipelines
Dev Digest 134 - Where pixels sing?News and ArticlesWeAreDevelopers LIVE Data and Security Day is on Wednesday, 25/09/2024. Learn about OPC UA Updates, Best Practices for Using GitHub Secrets, Passwordless Web 1.5, Emerging AI Security Risks, Data Privacy in LLMs and get a chance to t...
Dilek Demir
Data Science & more: The Lopez dilemmaCatwalk, Data Science, Hollywood, Google Images, Haute Couture, StackOverflow, Comfort Zone, Dota 2 and Versace – all these topics are connected and influenced by each other. Read here how and why!In 2000 Jennifer Lopez's green Versace dress went vi...
Daniel Cranney
Panel Discussion: Responsible AI in Practice - Real-World Examples and ChallengesIntroductionIn the ever-evolving landscape of artificial intelligence, the concept of "responsible AI" has emerged as a cornerstone for ethical and practical AI implementation. During the WWC24 Panel discussion, three eminent experts—Mina, Bjorn Brin...
From learning to earning
Jobs that call for the skills explored in this talk.
30:46
43:47
56:19
44:40
25:24
41:06
54:46
46:06
