Beam and Spark with Holden Karau

Holden Karau is on the podcast this week to talk all about Spark and Beam, two open source tools that helps process data at scale, with Mark and Melanie. Holden Karau Holden Karau is a transgender Canadian open source developer advocate @ Google with a focus on Apache Spark, BEAM, and related “big data” tools. She is the co-author of Learning Spark, High Performance Spark, and another Spark book that’s a bit more out of date. She is a commiter on and PMC on Apache Spark and committer on SystemML & Mahout projects. She was tricked into the world of big data while trying to improve search and recommendation systems and has long since forgotten her original goal. Cool things of the week Twitter’s collaboration with Google Cloud blog & tweet Kaggle CERN TrackML Particle Tracking Challenge Competition site Open-sourcing gVisor, a sandboxed container runtime blog & repo Announcing Stackdriver Kubernetes Monitoring blog MLPerf: collaborative effort to standardize ML benchmarks site Interview Spark site & community site Beam site Cloud Dataflow site & docs Cloud Dataproc site & docs Using Spark on Kubernetes Engine blog Testing future Apache Spark releases and changes on Google Kubernetes Engine and Cloud Dataproc blog Spark Packages site Spark testing base repo Flink site Arrow site Upcoming Talks: PyCon 2018 & Debugging PySpark talk Scala Days & Keeping the “fun” in Spark talk Strata London & Understanding Spark tuning with auto-tuning talk J on the Beach & General Purpose Big Data Systems are eating the world talk Spark Summit 2018 & Accelerating TF with Apache Arrow on Spark talk Question of the week I have a continuous integration build process setup with Container Builder, but it’s all sequential. I want to speed things up by processing parts of it in parallel. How do I do that? Configure Build Step Order docs Where can you find us next? Mark can be found streaming Agones development on Twitch. Melanie is speaking at the internet2 Global Summit, May 9th in San Diego, and will also be talking at the Understand Risk Forum on May 17th, in Mexico City. Special shout out: Google I/O and PyCon are both happening this week