90a3ff8194
- Simplify flatten_vault API to return usize instead of MigrationResult struct - Add KEEP_FOLDERS: attachments/ and _themes/ alongside type/, config/, theme/ - Use HashSet for collision tracking in unique_filename - Update wikilinks from path-based [[folder/slug]] to title-based [[slug]] - Clean up empty directories after flattening - Flatten demo-vault-v2: move all notes from type-based subfolders to root - Update smoke tests for flat vault structure - Remove migrate_to_flat_vault from repair_vault (one-time migration only) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
28 lines
2.3 KiB
Markdown
28 lines
2.3 KiB
Markdown
---
|
|
aliases: ["Data Engineering"]
|
|
Is A: Topic
|
|
---
|
|
# Data Engineering
|
|
|
|
Data engineering covers the infrastructure and practices behind moving, transforming, and serving data at scale — data pipelines, the modern data stack, analytics engineering, and the organizational challenges of making data useful. It remains a core area of technical knowledge from earlier career experience and a frequent topic in Refactoring content.
|
|
|
|
## Why this matters
|
|
|
|
Data engineering was a formative part of the professional journey before Refactoring, and the audience for the newsletter includes a large contingent of data and platform engineers. Writing credibly about engineering leadership requires staying current on how data teams work, what tools they use, and what problems they face. It also informs the broader perspective on [[topic-developer-tools]] and [[topic-saas-business]], since much of the modern data stack is built by venture-backed SaaS companies with interesting business dynamics.
|
|
|
|
## Key resources
|
|
|
|
- "Fundamentals of Data Engineering" by Joe Reis and Matt Housley — the best modern textbook on the field
|
|
- The Data Engineering Podcast — consistent coverage of tools, practices, and industry trends
|
|
- dbt (data build tool) documentation and community — the most influential tool in modern analytics engineering
|
|
- [[topic-developer-tools]] — the overlapping topic on how tools for engineers are built and sold
|
|
- Maxime Beauchemin's blog posts — foundational writing on data pipeline architecture from the creator of Airflow
|
|
|
|
## Notes
|
|
|
|
- The modern data stack hype cycle has peaked, and the industry is consolidating around fewer, more integrated tools — this is healthy
|
|
- Data engineering is one of the few engineering disciplines where the organizational problem (who owns the data, who defines metrics) is harder than the technical problem
|
|
- The best data teams treat data pipelines as software engineering problems, not as ETL scripts — testing, version control, and CI/CD apply
|
|
- Analytics engineering as a discipline (bridging data engineering and analytics) has been one of the most important organizational innovations in tech in recent years
|
|
- Many Refactoring readers work at the intersection of data and platform engineering, making this a consistently high-engagement topic
|