Apache Spark 1.3 architecture – module spark-core

This blog has been moved to new address: http://www.trongkhoanguyen.com.

After spending a significant time in reading the source code in spark-core project, I can briefly draw the architecture showing the relationships and the flow (messages passed) between important components in this module:Spark-core   See you in my next posts for more details on them. I believe that it’s extremely important to understand following components: schedule, shuffle and storage.

Update:
Understand the storage module in spark-core
Understand the scheduler component in spark-core
Understand the shuffle component in spark-core

This entry was posted in Architecture, Spark and tagged , . Bookmark the permalink.

2 Responses to Apache Spark 1.3 architecture – module spark-core

  1. Pingback: Spark deployment in cluster | Khoa's IT blog

  2. Pingback: Understand component scheduler in spark-core | Khoa's IT blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s