Scala Trainer

Getting productive before getting theoretical. Where Scala sits relative to Java and Python for data work, installing a toolchain, the sbt build model, the REPL and worksheets, and the Scala 2 versus Scala 3 question that every team has to answer.

Topics: Scala's position in the data ecosystem: Spark, Kafka Streams, Flink · JVM interoperability and reusing existing Java libraries · Installing Scala, JDK selection and version management · sbt project layout, build definition and dependency syntax · The REPL, scala-cli and worksheets for exploration · Scala 2 versus Scala 3: syntax, migration and Spark support · IDE setup and the compiler feedback loop

• Assignments: (1) Create an sbt project with a dependency and a runnable main class; (2) Call a Java library from Scala and handle the type mismatches it produces
• Capstone: Deliver a reproducible project skeleton your team could start every new job from

The core language, taught around immutability. val versus var and why the default matters in a distributed system, everything-is-an-expression, the type system and inference, methods versus functions, and higher-order functions as the basic unit of composition.

Topics: val, var, lazy val and immutability by default · Expressions rather than statements: if, match and blocks return values · The type hierarchy: Any, AnyVal, AnyRef, Nothing, Unit · Type inference and when to annotate anyway · Methods, functions, and function literals · Higher-order functions, currying and partial application · Default and named arguments, varargs · String interpolation and formatting

• Assignments: (1) Rewrite an imperative loop-and-mutate routine as a pure expression; (2) Write a higher-order function that composes two transformations and prove it type-checks
• Capstone: Convert a mutable data-processing routine into an immutable, expression-based implementation with identical output

How a Scala engineer describes a schema. Classes, objects and companions, case classes with their generated equality and copy, sealed traits as closed hierarchies, and pattern matching the compiler can prove exhaustive — the combination that stops an unhandled record type reaching production.

Topics: Classes, constructors, objects and companion objects · Case classes: equality, copy, apply and unapply · Traits, mixins and the linearisation rules · Sealed traits and algebraic data types · Pattern matching on values, types, constructors and guards · Exhaustivity checking and why it prevents real defects · Extractors and custom unapply · Enums and opaque types in Scala 3

• Assignments: (1) Model a real event schema as a sealed trait hierarchy of case classes; (2) Write a match that the compiler proves exhaustive, then break it by adding a subtype
• Capstone: Deliver a fully typed domain model for a real dataset, with exhaustive handling of every variant

The vocabulary that Spark transformations mirror. Immutable collections and their operations, strict versus lazy views, and then the functional error model — Option, Either and Try — composed with map, flatMap and for-comprehensions so failure paths are explicit rather than implicit.

Topics: List, Vector, Map, Set and choosing between them · map, flatMap, filter, fold, reduce, groupBy, zip · Views, streams and lazy evaluation · Option: eliminating null from the domain · Either for typed errors and Try for exception capture · for-comprehensions as flatMap chains · Traversing collections of effects · Recursion, tail recursion and stack safety

• Assignments: (1) Parse a dirty CSV into typed records with every failure captured in an Either; (2) Rewrite nested null checks as a for-comprehension over Options
• Capstone: Build a validation pipeline that accumulates every error in a record rather than failing on the first

The mechanism behind Spark encoders, JSON codecs and most library ergonomics in Scala. Implicit parameters and conversions in Scala 2, given and using in Scala 3, context bounds, type classes, and how to use all of it without producing code nobody else can read.

Topics: Implicit parameters and implicit resolution scope · given and using in Scala 3 · Extension methods and enriching existing types · Type classes: definition, instances and derivation · Context bounds and Ordering, Numeric, Encoder · Implicit conversions and why they are dangerous · How Spark encoders are resolved · Debugging an implicit-not-found error

• Assignments: (1) Define a type class with instances for three types and dispatch on it generically; (2) Diagnose and fix a missing-encoder error in a Spark Dataset job
• Capstone: Build a small type-class-based serialisation layer and explain how Spark's encoder resolution mirrors it

The main reason most teams book this course. RDDs, DataFrames and typed Datasets and when each is correct; Catalyst and Tungsten; reading a physical plan; and the performance work that separates a job that runs from a job that finishes — partitioning, shuffle, skew, broadcast joins and caching.

Topics: SparkSession, RDDs, DataFrames and typed Datasets compared · Schemas, encoders and compile-time column safety · Transformations, actions and lazy evaluation · Catalyst optimiser and reading explain output · Partitioning, repartition versus coalesce · Shuffles, skew and salting strategies · Broadcast joins and join strategy selection · UDFs, typed aggregators and when to avoid them · Caching, persistence levels and memory management · Structured Streaming: sources, sinks, watermarks and output modes

• Assignments: (1) Convert an untyped DataFrame job to typed Datasets and catch a schema defect at compile time; (2) Take a job with a skewed join and reduce its runtime with a documented change
• Capstone: Deliver a tuned Spark job with an explained physical plan and a measured before-and-after runtime

Beyond batch. Futures and execution contexts, the actor model with Akka or Pekko typed actors, Kafka producers, consumers and Streams topologies with custom serdes and state stores, and Flink for genuinely low-latency processing.

Topics: Futures, execution contexts and composing asynchronous work · Blocking, thread pools and the failure modes of getting it wrong · Typed actors, supervision and message protocols · Kafka producers and consumers from Scala · Kafka Streams topologies, custom serdes and state stores · Exactly-once semantics and what it does and does not guarantee · Flink DataStream basics: windows, state and checkpointing · Choosing between Structured Streaming, Kafka Streams and Flink

• Assignments: (1) Build a Kafka Streams topology with a custom serde over a case class schema; (2) Compose three asynchronous calls with Futures and handle partial failure correctly
• Capstone: Deliver a stateful streaming application with windowing, custom serialisation and recovery after restart

Making it shippable. ScalaTest and property-based testing, testing Spark jobs without a cluster, sbt multi-project builds, assembly and shading, dependency and Scala version conflicts, and deploying a job so it runs the same from a laptop and from a scheduler.

Topics: ScalaTest styles, matchers and structuring a suite · Property-based testing with ScalaCheck · Testing Spark transformations with a local session · Test fixtures for streaming and stateful code · sbt multi-project builds and shared settings · Assembly, shading and dependency conflicts · Scala binary version compatibility and Spark's constraints · Packaging, spark-submit, cluster modes and configuration · Logging, metrics and diagnosing a failed job from its history server

• Assignments: (1) Write property-based tests for a transformation and find an edge case the example tests missed; (2) Resolve a real dependency conflict with shading and prove the artefact runs
• Capstone: Deliver a tested, assembled Spark or streaming job with CI, configuration management and a documented deployment