: Zubair Nabi
: Pro Spark Streaming The Zen of Real-Time Analytics Using Apache Spark
: Apress
: 9781484214794
: 1
: CHF 31.20
:
: Anwendungs-Software
: English
: 243
: Wasserzeichen/DRM
: PC/MAC/eReader/Tablet
: PDF

Learn the right cutting-edge skills and knowledge to leverage Spark Streaming to implement a wide array of real-time, streaming applications.Pro Spark Streaming walks you through end-to-end real-time application development using real-world applications, data, and code. Taking an application-first approach, each chapter introduces use cases from a specific industry and uses publicly available datasets from that domain to unravel the intricacies of production-grade design and implementation. The domains covered in the book include social media, the sharing economy, finance, online advertising, telecommunication, and IoT.

n the last few years, Spark has become synonymous with big data processing. DStreams enhance the underlying Spark processing engine to support streaming analysis with a novel micro-batch processing model.Pro Spark Streamingby Zubair Nabi will enable you to become a specialist of latency sensitive applications by leveraging the key features of DStreams, micro-batch processing, and functional programming. To this end, the book includes ready-to-deploy examples and actual code.Pro Spark Streaming will act as the bible of Spark Streaming.

What You'll Learn:

  • Spark Streaming application development and best practices
    • Low-level details of discretized streams
  • The application and vitality of streaming analytics to a number of industries and domains
  • Optimization of production-grade deployments of Spark Streaming via configuration recipes and instrumentation using Graphite, collectd, and Nagios
    • Ingestion of data from disparate sources including MQTT, Flume, Kafka, Twitter, and a custom HTTP receiver
  • Integration and coupling with HBase, Cassandra, and Redis
  • Design patterns for side-effects and maintaining state across the Spark Streaming micro-batch model
    • Real-time and scalable ETL using data frames, SparkSQL, Hive, and SparkR
  • Streaming machine learning, predictive analytics, and recommendations
  • Meshi g batch processing with stream processing via the Lambda architecture
Who This Book Is For:

The audience includes data scientists, big data experts, BI analysts, and data architects.



Zubair Nabi is one of the very few computer scientists who have solved Big Data problems in all three domains: academia, research, and industry. He currently works at Qubit, a London-based start up backed by Goldman Sachs, Accel Partners, Salesforce Ventures, and Balderton Capital, which helps retailers understand their customers and provide personalized customer experience, and which has a rapidly growing client base that includes Staples, Emirates, Thomas Cook, and Topshop. Prior to Qubit, he was a researcher at IBM Research, where he worked at the intersection of Big Data systems and analytics to solve real-world problems in the telecommunication, electricity, and urban dynamics space.

Zubair's work has been featured in MIT Technology Review, SciDev, CNET, and Asian Scientist, and on Swedish National Radio, among others. He has authored more than 20 research papers, published by some of the top publication venues in computer science including USENIX Middleware, ECML PKDD, and IEEE BigData; and he also has a number of patents to his credit.

< iv>Zubair has an MPhil in computer science with distinction from Cambridge.
Contents at a Glance6
Contents8
About the Author14
About the Technical Reviewer16
Acknowledgments18
Introduction20
Chapter 1: The Hitchhiker s Guide to Big Data21
Before Spark21
The Era of Web 2.022
From SQL to NoSQL22
MapReduce: The Swiss Army Knife of Distributed Data Processing23
Word Count a la MapReduce24
Hadoop: An Elephant with Big Dreams25
Sensors, Sensors Everywhere26
Spark Streaming: At the Intersection of MapReduce and CEP28
Chapter 2: Introduction to Spark29
Installation30
Execution31
Standalone Cluster31
Master31
Workers31
UI31
YARN32
First Application32
Build34
Execution35
Local Execution35
Standalone Cluster35
YARN37
SparkContext37
Creation of RDDs37
Handling Dependencies38
Creating Shared Variables39
Job execution40
RDD40
Persistence41
Transformations42
Actions46
Summary47
Chapter 3: DStreams: Real-Time RDDs48
From Continuous to Discretized Streams48
First Streaming Application49
Build and Execution51
StreamingContext51
Creating DStreams52
DStream Consolidation53
Job Execution53
DStreams53
The Anatomy of a Spark Streaming Application55
Transformations59
Mapping59
map[U](function): DStream[U]59
mapPartitions[U](function): DStream[U]59
flatMap[U](function): DStream[U]60
filter(function): DStream[T]60
transform[U](function): DStream[U]60
Variation61
union(that: DStream[T]): DStream[T]61
repartition(numPartitions: Int): DStream[T]61
glom(): DStream[Array[T]]61
Aggregation61
count(): DStream[Long]61
countByValue(): DStream[(T, Long)]61
reduce(reduceFunc: (T, T) ? T): DStream[T]62
Key-value62
groupByKey(): DStream[(K, Iterable[V])]62
reduceByKey(reduceFunc: (V, V) ? V): DStream[(K, V)]62
combineByKey[C](createCombiner: (V) ? C, mergeValue: (C, V) ? C, mergeCombiner: (C, C) ? C, partitioner: Partitioner): DStream[(K, C)]63
join[W](other: DStream[(K, W)]): DStream[(K, (V, W))]63
cogroup[W](other: DStream[(K, W)]): DStream[(K, (Iterable[V], Iterable[W]))]64
updateStateByKey[S](updateFunc: (Seq[V], Option[S]) ? Option[S]): DStream[(K, S)]64
Windowing65
window(windowDuration: Duration, slideDuration: Duration): DStream[T]65
Actions68
print(num: Int): Unit68
print(): Unit68
saveAsObjectFiles(prefix: String): Unit68
saveAsTextFiles(prefix: String): Unit68
saveAsHadoopFiles[F68
6868
saveAsNewAPIHadoopFiles[F68
6868
foreachRDD(foreachFunc: (RDD[T]) ? Unit): Unit69
Summary69
Chapter 4: High-Velocity Streams: Parallelism and Other Stories70
One Giant Leap for Streaming Data70
Parallelism72
Worker72
Executor73
Choosing the Number of Executors74
Dynamic Executor Allocation74
Task75
Parallelism, Partitions, and Tasks75
Task Parallelism76
Batch Intervals78
Scheduling79
Inter-app