Unlock the complexities of machine learning algorithms in Spark to generate useful data insights through this data analysis tutorialAbout This BookProcess and analyze big data in a distributed and scalable wayWrite sophisticated Spark pipelines that incorporate elaborate extractionBuild and use regression models to predict flight delaysWho This Book Is ForAre you a developer with a background in machine learning and statistics who is feeling limited by the current slow and “small data” machine learning tools? Then this is the book for you! In this book, you will create scalable machine learning applications to power a modern data-driven business using Spark. We assume that you already know the machine learning concepts and algorithms and have Spark up and running (whether on a cluster or locally) and have a basic knowledge of the various libraries contained in Spark.What You Will LearnUse Spark streams to cluster tweets onlineRun the PageRank algorithm to compute user influencePerform complex manipulation of DataFrames using SparkDefine Spark pipelines to compose individual data transformationsUtilize generated models for off-line/on-line predictionTransfer the learning from an ensemble to a simpler Neural NetworkUnderstand basic graph properties and important graph operationsUse GraphFrames, an extension of DataFrames to graphs, to study graphs using an elegant query languageUse K-means algorithm to cluster movie reviews datasetIn DetailThe purpose of machine