This is a quick reference Apache Spark cheat sheet to assist developers already familiar with Java, Scala, Python, or SQL. Spark is an open-source engine for processing big data using cluster computing for fast, efficient analysis and performance.
While you're here, learn more about Zuar's data and analytics services. From raw data through to dashboard creation, we've got you covered!
Benefits of Using Apache Spark
Apache Spark is an open-source, Hadoop-compatible, cluster-computing platform that processes 'big data' with built-in modules for SQL, machine learning, streaming, and graph processing. The main benefits of using Apache Spark with your preferred API are...
- Computes data at blazing speeds by loading it across the distributed memory of a group of machines.
- Leveraging standard APIs like Java, Python, Scala or SQL to be more accessible.
- Allowing enterprises to leverage their existing infrastructures by being compatible with Hadoop v1 and 2.x.
- Easy to install and provides a convenient shell for learning the APIs.
- Improves productivity by focusing on content computation.
Commands for Initializing Apache Spark
These are the most common commands for initiating Apache Spark shell in either Scala or Python. These are essential commands you need when setting up the platform:
Initializing Spark Shell Using Scala
$ ./bin/spark-shell --master local
Initializing SparkContext Using Scala
val conf = new SparkConf().setAppName(appName).setMaster(master)
Initializing Spark Shell Using Python
$ ./bin/pyspark --master local
Initializing SparkContext Using Python
from pyspark import SparkContext
sc = SparkContext (master = ‘local’)
Configuring Spark Using Python
from pyspark import SparkConf, Spark Context
conf = (SparkConf()
sc = SparkContext(conf = conf)
Apache Spark Set Operations
Here is a list of the most common set operations to generate a new Resilient Distributed Dataset (RDD). An RDD is a fault-tolerant collection of data elements that can be operated on in parallel. You can create an RDD by referencing a dataset in an external storage system, or by parallelizing a collection in your driver program.
Joining All Elements From the Argument and Source
Intersecting All Elements From the Argument and Source
Creating a Cross Product From the Argument and Source
Removing Data Elements in the Source
Joining Data Elements to Create a New RDD
Converting to an Iterable
Piping Each Partition of an RDD Through a Shell Command
Apache Spark Action & Transformation Commands
Most RDD operations are either:
- Transformations: creating a new dataset from an existing dataset
- Actions: returning a value to the driver program from computing on the dataset
We’ll cover the most common actions and transformation commands below. Although, you should note that syntax can vary depending on the API you are using, such as Python, Scala, or Java.
Common Apache Spark Actions
Here are the bread and butter actions when calling an RDD to retrieve specific data elements.
Counting Number of Data Elements in the RDD
Collecting an Array of All the Data Elements
Aggregating the Data Elements
Getting the First N Data Elements
Executing the Function for Each Data Element
Retrieving the First Data Element
Common Apache Spark Transformations
Here are the main operations when you’re calling a new RDD by applying a transformation function to the data elements.
Selecting Data Elements Based on a Function
Applying a Function to Each Data Element
Applying a Function to Each Data Element to Return a Sequence
Applying a Function to Each Data Element Running Separately on Each Partition
Applying a Function to Each Data Element Running on an Indexed Partition
Sampling a Fraction of the Data
Sample(withReplacement, fraction, seed)
Applying a Function to Aggregate Values
RDD Persistence Commands
One of the best features of Apache Spark is its ability to cache an RDD in cluster memory, speeding up the iterative computation. Here are the most commonly used commands for RDD persistence.
Storing RDD in Cluster Memory as Deserialized Java Objects at a Default Level
Storing RDD in Cluster Memory or on the Disk as Deserialized Java Objects
Storing RDD As Serialized Java Objects One Byte Array per Partition
Storing RDD Only on the Disk
Spark SQL & Dataframe Commands
These are common integrated commands for using SQL with Apache Spark for working with structured data:
Integrating SQL queries with Apache Spark
Results = spark.sql(“SELECT * FROM tbl_name)
data_name = results.map(lambda p: col_name)
Connecting to Any Data Source
results = spark.sql (“””SELECT * FROM tbl_name JOIN json …”””)
Beyond the Cheat Sheet
The more you understand Apache Spark’s cluster computing technology, the better the performance and results you'll enjoy. For more in-depth tutorials and examples, check out the official Apache Spark Programming Guides.
Zuar provides products and services that pave a path towards a successful data strategy, from reducing the time and cost of implementation to ensuring that the ongoing maintaining of your systems is pain free. You can learn more here.