Applying Machine Learning To The Problem Of Bad Data At Anomalo

SummaryAll data systems are subject to the "garbage in, garbage out" problem. For machine learning applications bad data can lead to unreliable models and unpredictable results. Anomalo is a product designed to alert on bad data by applying machine learning models to various storage and processing systems. In this episode Jeremy Stanley discusses the various challenges that are involved in building useful and reliable machine learning models with unreliable data and the interesting problems that they are solving in the process.AnnouncementsHello and welcome to the Machine Learning Podcast, the podcast about machine learning and how to bring it from idea to delivery.Your host is Tobias Macey and today I'm interviewing Jeremy Stanley about his work at Anomalo, applying ML to the problem of data quality monitoringInterviewIntroductionHow did you get involved in machine learning?Can you describe what Anomalo is and the story behind it?What are some of the ML approaches that you are using to address challenges with data quality/observability?What are some of the difficulties posed by your application of ML technologies on data sets that you don't control? How does the scale and quality of data that you are working with influence/constrain the algorithmic approaches that you are using to build and train your models?How have you implemented the infrastructure and workflows that you are using to support your ML applications?What are some of the ways that you are addressing data quality challenges in your own platform? What are the opportunities that you have for dogfooding your product?What are the most interesting, innovative, or unexpected ways that you have seen Anomalo used?What are the most interesting, unexpected, or challenging lessons that you have learned while working on Anomalo?When is Anomalo the wrong choice?What do you have planned for the future of Anomalo?Contact Info@jeremystan on TwitterLinkedInParting QuestionFrom your perspective, what is the biggest barrier to adoption of machine learning today?Closing AnnouncementsThank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.If you've learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.To help other people find the show please leave a review on iTunes and tell your friends and co-workersLinksAnomaloData Engineering Podcast EpisodePartial Differential EquationsNeural NetworkNeural Networks For Pattern Recognition by Christopher M. Bishop (affiliate link)Gradient Boosted Decision TreesShapley ValuesSentrydbtAltairThe intro and outro music is from Hitman's Lovesong feat. Paola Graziano by The Freak Fandango Orchestra/CC BY-SA 3.0