Spark 3化解性能差距 95%数据工程职位倒向Python
课代表立正
由于行业大量投资完善Python转换层，Spark 3已抹平与Scala的性能差距，导致除少数科技巨头外的高达95%的市场职位全面转向Python。
AI-assisted summary of Scala is dead? @EcZachly_ — original source. Check the original for full context.
Source published:
Detailed notes
## Background 该视频由“课代表立正”于2026年5月13日发布在YouTube平台上。视频的描述部分非常简短，主要提供了该频道的会员加入链接。
## TL;DR 演讲者分享了自己为了Apache Spark而学习Scala的经历，指出早期版本的Spark因为原生支持而更倾向于Scala。然而，随着业界对Python转换层的大量投入，Spark 3中的Python性能已与Scala基本持平。尽管Netflix和Airbnb等科技巨头仍坚持使用Scala编写数据管道，但整个行业市场中Python的份额已高达95%左右。因此，演讲者建议初学者优先选择Python和PySpark，以便获得更多的就业机会和更平缓的职业晋升路径。
## Atmosphere & style 视频整体呈现出一种务实、真诚且带有行业洞察力的语调。演讲者以个人经历切入，语气坦诚而接地气，没有高高在上的说教，而是像一位资深同行在分享行业大实话。整个叙述充满理性和职场战略感，让听众在面对技术选型时感到脚踏实地，并在职业规划上产生共鸣与启发。
## Key takeaways - The speaker originally learned Scala specifically for Apache Spark to use the framework's native language. - Older versions of Apache Spark heavily favored Scala because it is the native language of the framework. - A massive amount of industry investment has been poured into perfecting the translation layer for Python. - The technology adapted because there is a widespread industry preference to avoid using Scala. - With Spark 3, the performance gap between Python and Scala has largely closed, making them functionally identical for most use cases. - Big tech giants like Netflix and Airbnb still stubbornly stick to writing their data pipelines almost exclusively in Scala. - The reliance on Scala by major tech companies is not a true reflection of the broader industry. - In the wider job market, the language split for data engineering is closer to 95% Python. - Prioritizing Python over Scala is the strategic move for anyone trying to break into the field to maximize job opportunities. - Learning PySpark provides a friendlier, gradual career increase and serves as an accessible gateway to other tools.
## Detailed walkthrough ### The Reality of Scala vs. Python in Apache Spark
The speaker shares that they originally learned Scala specifically for Apache Spark, driven by a personal preference to use the "native ways of doing things." Historically, while data engineers could choose between Python and Scala, older versions of Spark heavily favored Scala because it was the native language of the framework. However, a massive amount of industry investment has since been poured into perfecting the translation layer for Python. The reality is that "people don't want to use Scala," which forced the technology to adapt. With the release of Spark 3, the performance gap has largely closed, making Python functionally the same as Scala for the vast majority of use cases.
While big tech giants like Netflix and Airbnb still stubbornly stick to writing their data pipelines almost exclusively in Scala, the speaker emphasizes that this is "not a reflection of Industry." In the broader job market, the split is closer to 95% Python. For anyone trying to break into the field and looking for an abundance of job opportunities, prioritizing Python over Scala is the strategic move. Learning PySpark provides a much friendlier, "gradual increase in your career" and serves as an accessible gateway; once you master PySpark, transitioning to other tools over time becomes incredibly easy.
- **The Native Advantage:** Older versions of Apache Spark performed significantly better in Scala because the framework is natively written in it. - **The Shift in Spark 3:** Massive investment in Python translation layers has made Python performance mostly identical to Scala in Spark 3. - **Market Share:** Big tech companies like Netflix and Airbnb rely heavily on Scala, but the broader industry is roughly 95% Python. - **Career Strategy:** Beginners should focus on Python and PySpark to maximize job prospects and build a scalable skill set.
## Notable quotes > native ways of doing things
> people don't want to use Scala
> not a reflection of Industry
> gradual increase in your career
## Mentioned Scala, Python, Apache Spark, Spark 3, Netflix, Airbnb, PySpark
[Source](https://www.youtube.com/watch?v=v7ttBGoJSU4)