{"id":26669,"date":"2022-02-12T07:35:03","date_gmt":"2022-02-12T07:35:03","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=26669"},"modified":"2022-04-13T16:57:11","modified_gmt":"2022-04-13T16:57:11","slug":"top-50-apache-spark-interview-questions-and-answers","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/top-50-apache-spark-interview-questions-and-answers\/","title":{"rendered":"Top 50 Apache spark interview questions and answers"},"content":{"rendered":"\n
\"\"
Apache Spark<\/strong><\/em><\/figcaption><\/figure><\/div>\n\n\n\n

Q1. What is RDD?<\/h2>\n\n\n\n

Ans. RDD (Resilient Distribution Datasets) is a fault-tolerant collection of operational elements that run parallel. The partitioned data in RDD is immutable and distributed.<\/p>\n\n\n\n

Q2. Name the different types of RDD.<\/h2>\n\n\n\n

Ans. There are primarily two types of RDD \u2013 parallelized collection and Hadoop datasets.<\/p>\n\n\n\n

Q3. What are the methods of creating RDDs in Spark?<\/h2>\n\n\n\n

Ans. There are two methods \u2013<\/p>\n\n\n

By parallelizing a collection in<\/span> your Driver program.\nBy loading an external dataset from<\/span> external storage like HDFS, HBase, shared file system.<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Q4. What is a Sparse Vector?<\/h2>\n\n\n\n
Ans. A sparse vector has two parallel arrays \u2013one for indices and the other for values.<\/p>\n\n\n\n
Q5. Mention some of the areas where Spark outperforms Hadoop in processing.<\/h2>\n\n\n\n
Ans. Sensor data processing, real-time querying of data, and stream processing.<\/p>\n\n\n\n
Q7. What is Yarn?<\/h2>\n\n\n\n
Ans. Yarn is one of the key features in Spark, providing a central and resource management platform to deliver scalable operations across the cluster.<\/p>\n\n\n\n
Q8. Do you need to install Spark on all nodes of the Yarn cluster? Why?<\/h2>\n\n\n\n
Ans. No, because Spark runs on top of Yarn.<\/p>\n\n\n\n
Q9. Is it possible to run Apache Spark on Apache Mesos?<\/h2>\n\n\n\n
Ans. Yes.<\/p>\n\n\n\n
Q10. What is the lineage graph?<\/h2>\n\n\n\n
Ans. The RDDs in Spark, depend on one or more other RDDs. The representation of dependencies in between RDDs is known as the lineage graph.<\/p>\n\n\n\n
Q11. Define Partitions in Apache Spark<\/h2>\n\n\n\n
Ans. Partition is a smaller and logical division of data similar to \u2018split\u2019 in MapReduce. It is a logical chunk of a large distributed data set. Partitioning is the process to derive logical units of data to speed up the processing process.<\/p>\n\n\n\n
Q12. What is a DStream?<\/h2>\n\n\n\n
Ans. Discretized Stream (DStream) is a sequence of Resilient Distributed Databases that represent a stream of data.<\/p>\n\n\n\n
Q13. What is a Catalyst framework?<\/h2>\n\n\n\n
Ans. Catalyst framework is an optimization framework present in Spark SQL. It allows Spark to automatically transform SQL queries by adding new optimizations to build a faster processing system.<\/p>\n\n\n\n
Q14. What are the Actions in Spark?<\/h2>\n\n\n\n
Ans. An action helps in bringing back the data from RDD to the local machine. An action\u2019s execution is the result of all previously created transformations.<\/p>\n\n\n\n
Q15. What is a Parquet file?<\/h2>\n\n\n\n
Ans. Parquet is a columnar format file supported by many other data processing systems.<\/p>\n\n\n\n
Q16. What is GraphX?<\/h2>\n\n\n\n
Ans. Spark uses GraphX for graph processing to build and transform interactive graphs.<\/p>\n\n\n\n
Q17. What file systems does Spark support?<\/h2>\n\n\n\n
Ans. Spark supports the following systems:<\/p>\n\n\n
Hadoop Distributed File System (HDFS).\nLocal File system.\nAmazon S3<\/code><\/span><\/pre>\n\n\n
Q18. What is the difference between persist () and cache ()?<\/h2>\n\n\n\n
Ans. Persist () allows the user to specify the storage level whereas cache () uses the default storage level.<\/p>\n\n\n\n
Q19. What do you understand by SchemaRDD?<\/h2>\n\n\n\n
Ans. SchemaRDD is an RDD that consists of row objects (wrappers around the basic string or integer arrays) with schema information about the type of data in each column.<\/p>\n\n\n\n
Q20. What is a lazy evaluation in Spark?<\/h2>\n\n\n\n
Ans. For large data in Spark, multiple operations take place even for the execution of a basic transformation. When a transformation is called on an RDD, the operation does not occur immediately. Transformations in Spark are not evaluated until you trigger an action. This is known as lazy evaluation. It avoids unnecessary memory and CPU usage that could take place due to certain mistakes.<\/p>\n\n\n\n
Q21. What is Apache Spark?<\/h2>\n\n\n\n
Ans. Apache Spark is easy to use and flexible data processing framework. Spark can round on Hadoop, standalone, or in the cloud. It is capable of assessing diverse data source, which includes HDFS, Cassandra, and others.<\/p>\n\n\n\n
Q22. Explain Dsstream with reference to Apache Spark<\/h2>\n\n\n\n
Ans. Dstream is a sequence of resilient distributed database which represent a stream of data. You can create Dstream from various source like HDFS, Apache Flume, Apache Kafka, etc.<\/p>\n\n\n\n
Q23. Name three data source available in SparkSQL<\/h2>\n\n\n\n
Ans. There data source available in SparkSQL are:<\/p>\n\n\n
JSON<\/span> Datasets\nHive tables\nParquet file<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Q24. Name some internal daemons used in spark?<\/h2>\n\n\n\n
Ans. Important daemon used in spark are Blockmanager, Memestore, DAGscheduler, Driver, Worker, Executor, Tasks,etc.<\/p>\n\n\n\n
Q25. Define the term \u2018Sparse Vector.\u2019<\/h2>\n\n\n\n
Ans. Sparse vector is a vector which has two parallel arrays, one for indices, one for values, use for storing non-zero entities to save space.<\/p>\n\n\n\n
Q26. Name the language supported by Apache Spark for developing big data applications<\/h2>\n\n\n\n
Ans. Important language use for developing big data application are:<\/p>\n\n\n
Java\nPython\nR\nClojure\nScala<\/code><\/span><\/pre>\n\n\n
Q27. What is the method to create a Data frame?<\/h2>\n\n\n\n
Ans. In Apache Spark, a Data frame can be created using Tables in Hive and Structured data files.<\/p>\n\n\n\n
Q28. Explain SchemaRDD<\/h2>\n\n\n\n
Ans. RDD which consists of row object with schema information about the type of data in each column is called SchemaRDD.<\/p>\n\n\n\n
Q29. What are accumulators?<\/h2>\n\n\n\n
Ans. Accumulators are the write-only variables. They are initialized once and sent to the workers. These workers will update based on the logic written, which will send back to the driver.<\/p>\n\n\n\n
Q30. What are the components of Spark Ecosystem?<\/h2>\n\n\n\n
Ans. An important component of Spark are:<\/p>\n\n\n
Spark Core: It is a base engine for large-scale parallel and distributed data processing\nSpark Streaming: This component used for real-time data streaming.\nSpark SQL: Integrates relational processing by using Spark\u2019s functional programming API\nGraphX: Allows graphs and graph-parallel computation\nMLlib: Allows you to perform machine learning in Apache Spark<\/code><\/span><\/pre>\n\n\n
Q31. Name three features of using Apache Spark<\/h2>\n\n\n\n
Ans. Three most important feature of using Apache Spark are:<\/p>\n\n\n
Support for<\/span> Sophisticated Analytics\nHelps you to Integrate with<\/span> Hadoop and Existing Hadoop Data\nIt allows you to run an application in<\/span> Hadoop cluster, up to 100<\/span> times faster in<\/span> memory, and ten times faster on disk.<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Q32. Explain the default level of parallelism in Apache Spark<\/h2>\n\n\n\n
Ans. If the user isn\u2019t able to specify, then the number of partitions are considered as default level of parallelism in Apache Spark.<\/p>\n\n\n\n
Q33. Name three companies which is used Spark Streaming services<\/h2>\n\n\n\n
Ans. Three known companies using Spark Streaming services are:<\/p>\n\n\n
Uber\nNetflix\nPinterest<\/code><\/span><\/pre>\n\n\n
Q34. What is Spark SQL?<\/h2>\n\n\n\n
Ans. Spark SQL is a module for structured data processing where we take advantage of SQL queries running on that database.<\/p>\n\n\n\n
Q35. Explain Parquet file<\/h2>\n\n\n\n
Ans. Paraquet is a columnar format file support by many other data processing systems. Spark SQL allows you to performs both read and write operations with Parquet file.<\/p>\n\n\n\n
Q36. Explain Spark Driver?<\/h2>\n\n\n\n
Ans. Spark Driver is the program which runs on the master node of the machine and declares transformations and actions on data RDDs.<\/p>\n\n\n\n
Q37) How can you store the data in spark?<\/h2>\n\n\n\n
Ans. Spark is a processing engine which doesn\u2019t have any storage engine. It can retrieve data from another storage engine like HDFS, S3.<\/p>\n\n\n\n
Q38. Explain the use of File system API in Apache Spark<\/h2>\n\n\n\n
Ans. File system API allows you to read data from various storage devices like HDFS, S3 or local Filesystem.<\/p>\n\n\n\n
Q39. What is the task of Spark Engine<\/h2>\n\n\n\n
Ans. Spark Engine is helpful for scheduling, distributing and monitoring the data application across the cluster.<\/p>\n\n\n\n
Q40. What is the user of spark context?<\/h2>\n\n\n\n
Ans. SparkContent is the entry point to spark. SparkContext allows you to create RDDs which provided various way of churning data.<\/p>\n\n\n\n
Q41. Explain about transformations and actions in the context of RDDs.<\/h2>\n\n\n\n
Ans. Transformations are functions executed on demand, to produce a new RDD. All transformations are followed by actions. Some examples of transformations include map, filter and reduceByKey.<\/p>\n\n\n\n
Actions are the results of RDD computations or transformations. After an action is performed, the data from RDD moves back to the local machine. Some examples of actions include reduce, collect, first, and take.<\/p>\n\n\n\n
Q42. Can you use Spark to access and analyse data stored in Cassandra databases?<\/h2>\n\n\n\n
Ans. Yes, it is possible if you use Spark Cassandra Connector.<\/p>\n\n\n\n
Q43. Is it possible to run Apache Spark on Apache Mesos?<\/h2>\n\n\n\n
Ans. Yes, Apache Spark can be run on the hardware clusters managed by Mesos.<\/p>\n\n\n\n
Q44. Explain about the different cluster managers in Apache Spark<\/h2>\n\n\n\n
Ans. The 3 different clusters managers supported in Apache Spark are:<\/p>\n\n\n
YARN\nApache Mesos -Has rich resource scheduling capabilities and is well suited to run Spark along with<\/span> other applications. It is advantageous when several users run interactive shells because it scales down the CPU allocation between commands.\nStandalone deployments \u2013 Well suited for<\/span> new<\/span> deployments which only run and are easy to set<\/span> up.<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Q45. How can Spark be connected to Apache Mesos?<\/h2>\n\n\n\n
Ans. To connect Spark with Mesos-<\/p>\n\n\n
Configure<\/span> the<\/span> spark<\/span> driver<\/span> program<\/span> to<\/span> connect<\/span> to<\/span> Mesos<\/span>. Spark<\/span> binary<\/span> package<\/span> should<\/span> be<\/span> in<\/span> a<\/span> location<\/span> accessible<\/span> by<\/span> Mesos<\/span>. (or<\/span>)\nInstall<\/span> Apache<\/span> Spark<\/span> in<\/span> the<\/span> same<\/span> location<\/span> as<\/span> that<\/span> of<\/span> Apache<\/span> Mesos<\/span> and<\/span> configure<\/span> the<\/span> property<\/span> \u2018spark<\/span>.mesos<\/span>.executor<\/span>.home<\/span>\u2019 to<\/span> point<\/span> to<\/span> the<\/span> location<\/span> where<\/span> it<\/span> is<\/span> installed<\/span>.<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Q46. Why is there a need for broadcast variables when working with Apache Spark?<\/h2>\n\n\n\n
Ans. These are read only variables, present in-memory cache on every machine. When working with Spark, usage of broadcast variables eliminates the necessity to ship copies of a variable for every task, so data can be processed faster. Broadcast variables help in storing a lookup table inside the memory which enhances the retrieval efficiency when compared to an RDD lookup ().<\/p>\n\n\n\n
Q47. Is it possible to run Spark and Mesos along with Hadoop?<\/h2>\n\n\n\n
Ans. Yes, it is possible to run Spark and Mesos with Hadoop by launching each of these as a separate service on the machines. Mesos acts as a unified scheduler that assigns tasks to either Spark or Hadoop.<\/p>\n\n\n\n
Q48. What are the benefits of using Spark with Apache Mesos?<\/h2>\n\n\n\n
Ans. It renders scalable partitioning among various Spark instances and dynamic partitioning between Spark and other big data frameworks<\/p>\n\n\n\n
Q49. What is a DStream?<\/h2>\n\n\n\n
Ans. Discretized Stream is a sequence of Resilient Distributed Databases that represent a stream of data. DStreams can be created from various sources like Apache Kafka, HDFS, and Apache Flume. DStreams have two operations \u2013<\/p>\n\n\n
Transformations that produce a new<\/span> DStream.\nOutput operations that write data to an external system.<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Q50. When running Spark applications, is it necessary to install Spark on all the nodes of YARN cluster?<\/h2>\n\n\n\n
Ans. Spark need not be installed when running a job under YARN or Mesos because Spark can execute on top of YARN or Mesos clusters without affecting any change to the cluster.<\/p>\n\n\n\n
Related video:<\/h4>\n\n\n\n
\n