فهرست مطالب

• Data Analysis with Python and PySpark
• contents
• preface
• acknowledgments
• about this book
• about the author
• about the cover illustration
• 1 Introduction
• 1.1 What is PySpark?
• 1.2 Your very own factory: How PySpark works
• 1.3 What will you learn in this book?
• 1.4 What do I need to get started?
• Summary
• Part 1—Get acquainted: First steps in PySpark
• 2 Your first data program in PySpark
• 2.1 Setting up the PySpark shell
• 2.2 Mapping our program
• 2.3 Ingest and explore: Setting the stage for data transformation
• 2.4 Simple column transformations: Moving from a sentence to a list of words
• 2.5 Filtering rows
• Summary
• Additional exercises
• 3 Submitting and scaling your first PySpark program
• 3.1 Grouping records: Counting word frequencies
• 3.2 Ordering the results on the screen using orderBy
• 3.3 Writing data from a data frame
• 3.4 Putting it all together: Counting
• 3.5 Using spark-submit to launch your program in batch mode
• 3.6 What didn’t happen in this chapter
• 3.7 Scaling up our word frequency program
• Summary
• Additional Exercises
• 4 Analyzing tabular data with pyspark.sql
• 4.1 What is tabular data?
• 4.2 PySpark for analyzing and processing tabular data
• 4.3 Reading and assessing delimited data in PySpark
• 4.4 The basics of data manipulation: Selecting, dropping, renaming, ordering, diagnosing
• Summary
• Additional exercises
• 5 Data frame gymnastics: Joining and grouping
• 5.1 From many to one: Joining data
• 5.2 Summarizing the data via groupby and GroupedData
• 5.3 Taking care of null values: Drop and fill
• 5.4 What was our question again? Our end-to-end program
• Summary
• Additional exercises
• Part 2—Get proficient: Translate your ideas into code
• 6 Multidimensional data frames: Using PySpark with JSON data
• 6.1 Reading JSON data: Getting ready for the schemapocalypse
• 6.2 Breaking the second dimension with complex data types
• 6.3 The struct: Nesting columns within columns
• 6.4 Building and using the data frame schema
• 6.5 Putting it all together: Reducing duplicate data with complex data types
• Summary
• Additional exercises
• 7 Bilingual PySpark: Blending Python and SQL code
• 7.1 Banking on what we know: pyspark.sql vs. plain SQL
• 7.2 Preparing a data frame for SQL
• 7.3 SQL and PySpark
• 7.4 Using SQL-like syntax within data frame methods
• 7.5 Simplifying our code: Blending SQL and Python
• 7.6 Conclusion
• Summary
• Additional exercises
• 8 Extending PySpark with Python: RDD and UDFs
• 8.1 PySpark, freestyle: The RDD
• 8.2 Using Python to extend PySpark via UDFs
• Summary
• Additional exercises
• 9 Big data is just a lot of small data: Using pandas UDFs
• 9.1 Column transformations with pandas: Using Series UDF
• 9.2 UDFs on grouped data: Aggregate and apply
• 9.3 What to use, when
• Summary
• Additional exercises
• 10 Your data under a different lens: Window functions
• 10.1 Growing and using a simple window function
• 10.2 Beyond summarizing: Using ranking and analytical functions
• 10.3 Flex those windows! Using row and range boundaries
• 10.4 Going full circle: Using UDFs within windows
• 10.5 Look in the window: The main steps to a successful window function
• Summary
• Additional Exercises
• 11 Faster PySpark: Understanding Spark’s query planning
• 11.1 Open sesame: Navigating the Spark UI to understand the environment
• 11.2 Thinking about performance: Operations and memory
• Summary
• Part 3—Get confident: Using machine learning with PySpark
• 12 Setting the stage: Preparing features for machine learning
• 12.1 Reading, exploring, and preparing our machine learning data set
• 12.2 Feature creation and refinement
• 12.3 Feature preparation with transformers and estimators
• Summary
• 13 Robust machine learning with ML Pipelines
• 13.1 Transformers and estimators: The building blocks of ML in Spark
• 13.2 Building a (complete) machine learning pipeline
• 13.3 Evaluating and optimizing our model
• 13.4 Getting the biggest drivers from our model: Extracting the coefficients
• Summary
• 14 Building custom ML transformers and estimators
• 14.1 Creating your own transformer
• 14.2 Creating your own estimator
• 14.3 Using our transformer and estimator in an ML pipeline
• Summary
• Conclusion: Have data, am happy!
• Appendix A—Solutions to the exercises
• Appendix B—Installing PySpark
• Appendix C—Some useful Python concepts
• index