Private corporate batches, live online cohorts and 1-on-1 mentoring in distributed processing with DataFrames and Structured Streaming, and the tuning that decides job cost — taught by a practitioner who runs it in production.
20 years across DevOps, SRE and Security · 10,000+ engineers trained · Trained teams at JPMorgan Chase, Verizon, Nokia and the World Bank
Principal DevOps Engineer & Architect
Rajesh teaches Spark around its execution model rather than its API surface: lazy evaluation and the DAG, jobs to stages to tasks, and the shuffle as the operation that determines cost. Sessions cover partitioning and skew, broadcast versus sort-merge joins, the unified memory model, caching decisions, Adaptive Query Execution and reading the Spark UI to find the real bottleneck — then move on to Structured Streaming with watermarks, state and checkpointed exactly-once delivery, and to running Spark on YARN and Kubernetes in production.
Twenty years across DevOps, SRE and Security, in principal and architect roles at PayPay, SoftwareAG, ServiceNow, JDA Software, Intuit, Adobe and others. He has trained engineers at JPMorgan Chase, Verizon, Nokia, the World Bank, VMware, Oracle, Mercedes-Benz and Airbus — more than 10,000 people personally. He teaches what he runs, not what he reads.
You are booked with a named engineer, and that is who turns up. Marketplaces and larger providers rotate whoever is free, so the person who sold you the agenda is rarely the person teaching it.
The same trainer is available for the next engagement, which matters when a team builds on what it learned last time.
Your batch is assigned a named trainer before it starts, and that is who teaches it. See the full faculty.
Engagements are matched on the tool, not the calendar. For Spark that means a trainer who has run it in production — distributed processing with DataFrames and Structured Streaming, and the tuning that decides job cost — rather than whoever is free that week. You are told who is teaching before you commit, and that person is on the discovery call that shapes the agenda.
Where a batch is large enough to need a second trainer, the pairing is declared up front. The lead trainer stays accountable for the syllabus and the assessment either way.
Principal DevOps Engineer & Architect
Twenty years across DevOps, SRE and Security in principal and architect roles at PayPay, SoftwareAG, ServiceNow, JDA Software, Intuit, Adobe, IBM/Emptoris, Ness, MindTree and Accenture. He has trained more than 10,000 engineers personally, at organisations including JPMorgan Chase, Verizon, Nokia, the World Bank, VMware, Oracle, Mercedes-Benz and Airbus. He teaches what he runs, not what he reads.
Teams of 8–30
Custom agenda, your timezone, onsite or online, NDA-friendly.
Request a quoteIndividual engineers
A private instructor and a curriculum built around your goal.
₹99,999Individuals who want peers
Scheduled batch, max 8 to 10 hours of live instruction.
₹34,999A private batch starts with a discovery call. We look at the stack you actually run — the CI system, the cloud, the constraints — and map the agenda onto it, so examples use your topology rather than a generic one.
Delivery is onsite at your premises, live online, or hybrid, scheduled around your release calendar rather than ours. Batches run 8 to 30 engineers.
Every attendee leaves with recordings, slides, lab repositories and a completion certificate. You receive an attendance and assessment report. Invoicing supports PO and GST.
Talk to us about a private Spark batchApache Spark is a distributed processing engine for large-scale data. A Spark application is a driver program that builds a plan and a set of executors that run the work; between them sit a cluster manager — YARN, Kubernetes, standalone or a managed service — and a scheduler that turns the plan into stages and tasks. Nothing executes when you write a transformation. Spark records it lazily, and only an action forces the accumulated graph to run, which is what lets the engine optimise across the whole chain rather than statement by statement.
The API has two layers. RDDs are the original abstraction: a partitioned, immutable collection with a lineage that allows lost partitions to be recomputed after a failure. DataFrames and Datasets sit above them and carry a schema, which is what makes the Catalyst optimiser and the Tungsten execution engine possible — predicate pushdown, column pruning, join reordering and generated code that avoids the overhead of interpreting each row. In practice almost all new Spark code should be DataFrames or Spark SQL, with RDDs reserved for the cases the structured API cannot express.
Performance in Spark is dominated by one operation: the shuffle. Any transformation that redistributes data across partitions — a wide dependency such as a join, a group-by or a repartition — writes to disk and moves data over the network, and almost every slow job traces back to too much shuffle, badly sized partitions or skew that leaves one task doing most of the work. Understanding the memory model, join strategies, Adaptive Query Execution and how to read the Spark UI is therefore the difference between a job that costs a few dollars and the same job costing hundreds. Structured Streaming extends the same engine and the same DataFrame API to unbounded data, adding event-time watermarks, stateful operations and checkpointed exactly-once processing.
Spark is the default processing engine for large data. It runs the batch ETL behind most warehouses and lakehouses, the feature pipelines behind most production models, and a large share of streaming workloads, on every major cloud and inside Databricks, EMR, Dataproc and Synapse.
What makes the skill valuable is that Spark is easy to write and hard to run well. Anyone can produce a PySpark script that returns the right answer; the same script may take four hours and forty executors when a correctly partitioned version with a broadcast join takes eight minutes on four. In cloud environments that difference is billed directly, monthly, which is why Spark tuning is one of the few engineering skills with an obvious and immediate financial return.
The second driver is convergence. Batch and streaming now share one API, lakehouse table formats are read and written through Spark, and feature engineering for machine learning increasingly runs on the same engine. An engineer who understands Spark's execution model can work across data engineering, analytics and ML infrastructure rather than being confined to one of them.
The engine before the API. Where Spark came from and what it changed relative to MapReduce, then the execution model in detail: driver and executors, the cluster managers it runs under, lazy evaluation, and the translation from your code into a DAG of jobs, stages and tasks.
Topics: From MapReduce to a unified engine · Driver, executors and the cluster manager · Standalone, YARN, Kubernetes and managed platforms · Client vs cluster deploy mode · Lazy evaluation and the DAG · Jobs, stages, tasks and the scheduler · Narrow vs wide dependencies · Language bindings: Scala, PySpark, SQL and R · Setting up a working Spark environment
The original abstraction, taught because it makes the engine's behaviour visible. Creating RDDs and loading data, transformations against actions, closures and serialisation, key-value operations and the difference between reduceByKey and groupByKey, shared variables, and persistence with lineage-based recovery.
Topics: What an RDD is: partitions, lineage, immutability · Loading data and creating RDDs · Transformations vs actions · Lambdas, closures and serialisation pitfalls · Key-value pair operations · reduceByKey vs groupByKey and why it matters · Accumulators and broadcast variables · Persistence and storage levels · Fault recovery through lineage · When to still reach for an RDD
The API almost all production Spark should use, and the optimiser it enables. Schemas and types, reading and writing every common format and source, SQL against temporary views, user-defined functions and why they are expensive in PySpark, and Catalyst and Tungsten made visible through explain plans.
Topics: DataFrames, Datasets and typed vs untyped APIs · Schemas, inference and explicit definition · Reading and writing Parquet, ORC, JSON, CSV and JDBC · Temporary views and Spark SQL · Column expressions, functions and window functions · UDFs, pandas UDFs and the serialisation cost · The Catalyst optimiser: rules and stages · Tungsten and whole-stage code generation · Reading and acting on explain plans · Predicate pushdown and column pruning
The module that most job cost traces back to. Partition count and parallelism, repartition against coalesce, what a shuffle physically does, then join strategies — broadcast hash, sort-merge and shuffle hash — and skew, which is the single most common cause of a job whose last task runs for an hour.
Topics: Partitions, parallelism and default partition counts · repartition vs coalesce and when each is correct · The shuffle mechanism: write, fetch, sort · Shuffle partitions and how to size them · Broadcast hash join and the broadcast threshold · Sort-merge join and shuffle hash join · Detecting skew from stage task distribution · Skew mitigation: salting, AQE skew join, filtering hot keys · Bucketing to avoid repeated shuffles · Spill to disk and its symptoms
Making the job fit and go fast. The unified memory model and how execution and storage compete, sizing executors, cores and overhead, dynamic allocation, when caching helps and when it wastes memory, serialisation choices, Adaptive Query Execution, and the Spark UI as the primary diagnostic tool.
Topics: The unified memory model: execution vs storage · Executor memory, overhead and off-heap · Cores per executor and the parallelism trade-off · Dynamic allocation and the external shuffle service · Caching and persistence decisions · Kryo serialisation and object overhead · Adaptive Query Execution: coalescing, skew and join switching · Reading the Spark UI: stages, tasks, shuffle, spill, GC · Stragglers, speculative execution and data locality · Diagnosing OutOfMemory in driver and executor
The same engine applied to unbounded data. The micro-batch model and how a streaming query is just an incrementally executed DataFrame query; sources and sinks including Kafka; event time, watermarks and late data; stateful aggregations and joins; and checkpointing for exactly-once processing across restarts.
Topics: The Structured Streaming model and incremental execution · Sources and sinks, including Kafka and file sources · Output modes: append, update, complete · Triggers, micro-batch and continuous processing · Event time vs processing time · Watermarks and handling late data · Windowed aggregations: tumbling, sliding, session · Stateful operations and the state store · Stream-static and stream-stream joins · Checkpointing, recovery and exactly-once semantics · Monitoring streaming queries and backpressure
Where Spark sits among everything else. Open table formats used through Spark, ACID operations and time travel, MERGE for change data capture and deletions, table maintenance, catalogues and metastores, an overview of MLlib, and an honest comparison against warehouses and dedicated streaming engines.
Topics: Delta Lake and Apache Iceberg through Spark · ACID transactions, snapshots and time travel · MERGE, upserts and compliance deletes · Compaction, clustering and file-size maintenance · Hive metastore and modern catalogues · Partition strategy for lakehouse tables · MLlib: pipelines, feature transformers, model training · Spark for feature engineering in ML workflows · Spark against cloud warehouses and against Flink
Everything between a working script and a reliable scheduled job. Submitting and configuring applications, packaging dependencies for Scala and Python, running on Kubernetes or YARN, capacity and cost control, log aggregation and the history server, testing, and CI for Spark applications.
Topics: spark-submit and configuration precedence · Packaging: assembly JARs, Python dependencies, containers · Spark on Kubernetes vs Spark on YARN · Managed platforms: EMR, Dataproc, Databricks, Synapse · Dynamic allocation, autoscaling and spot instances · Cost per job and cluster right-sizing · History server, event logs and log aggregation · Retries, idempotency and safe reruns · Unit and integration testing of Spark applications · CI/CD for Spark jobs · Common production failures and their signatures
We rebuild the agenda around the tools you actually run.
Request a custom agendaRun a multi-stage job and map every line of code to the jobs, stages, tasks and shuffles the Spark UI reports, then predict the plan before running a variant.
Diagnose a skewed join from stage task distribution, fix it with broadcast, salting and AQE in turn, and compare all three outcomes.
Take a job that occupies forty executors for hours, then fix partitioning, join strategy, caching and executor sizing until it fits a small cluster.
Replace Python UDFs with native expressions and window functions, compare explain plans, and measure the serialisation cost you removed.
Build a Kafka to lakehouse streaming job with watermarks and stateful aggregation, kill it mid-batch, and prove the output has no gaps or duplicates.
Deliver a packaged, tested, tuned job running on Kubernetes or YARN with idempotent reruns, log aggregation, monitoring and a per-run cost figure.
Billed yearly at ₹9,996
Max 8 to 10 hours of live instruction
Full program, private instructor
8–30 engineers · custom agenda · onsite or online · PO and GST invoicing
Refunds. If we cancel or postpone a cohort, you get a full refund within 15 days. There is no money-back guarantee otherwise.
Terms. Course material remains licensed to the attendee. Read the terms.
Your data. We don't share it with third parties. Privacy policy.
Certificate of completion
Rajesh is a very good trainer I have experienced in DevSecOps training. The number of contents in different topics he has posted on the DevOpsSchool public website are amazing and user friendly for beginners and experienced professionals.
The trainer (Rajesh) provided very good sessions on SRE profession. Not only hands-on learning on the tools but also SRE mindset.
Very good training session. Well explained from the basics to the complex concepts. Also tried to cover practicals and demos within the 3 hour sessions. The learning content and videos are of a great deal of help.
Basics explanation was exemplary from Rajesh where he dealt with complicated topics to be simple. Great learning stuff personally for me.
Very detailed explanation and has lots of patience in attending the questionnaire. Thanks again for your wonderful sessions.
Good discussion, helped us to understand different tools in SRE.
| What matters | YouTube + blogs | Generic online course | Freelance marketplace | DevOpsSchool |
|---|---|---|---|---|
| Named practitioner | No | Rarely | Varies per booking | Yes — same trainer each time |
| Production experience | Unknown | Unknown | Unverified | 20 years, named employers |
| Custom agenda | No | No | Sometimes | Built from your stack |
| Onsite delivery | No | No | Sometimes | Yes |
| Lab environment | None | Sandbox that expires | Varies | Your own cloud — skill goes with you |
| Assessment | None | Quiz | Rarely | Assignments + capstone per module |
| Per-attendee certificates | No | Sometimes | Rarely | Yes |
| Corporate invoicing | No | Limited | Varies | PO and GST |
| Post-training support | None | Forum, time-limited | None | Lifetime forum access |
Tell us the team, the stack and the timeline. You'll get a straight answer, not a sales sequence.
Talk to an advisorPrefer to call or email?
More ways to reach us on the contact page.