Engenheiro de Dados & Engenheiro de Plataforma de Dados
I build data platforms that decisions can rest on — from ingestion to the dimensional model.
At Minerva Foods, I cut the cost of the platform's largest data operation by 73%
while the volume served kept growing.
Work and curiosity converge on one engineering question: where did this come from? — answered fast enough to still change the decision.
protein
Physical chain
At Minerva the trail is protein: farm to container, port, and on to the destination. Traceability here isn't a report — it's an export requirement.
farm → packing plant → container → port → destination
pixel
Spatial chain
At MyCarbon the trail was pixel and polygon: CAR, PRODES and soil carbon, farm by farm. Cloud-native formats changed what you can ask — and at what cost. See the case →
Inside the platform the trail is lineage: from source to gold table, every number knows where it came from and what it cost to produce. The same question, turned inward.
source → bronze → silver → gold → decision
Three distinct domains, one engineering problem: ingest a lot, model it well, and reconstruct provenance on demand. Dimensional modeling, partitioning and lineage don't change in kind when the object goes from cattle to pixel to the table itself — the cardinality does. The same holds for transaction graphs, which I follow separately.
02 — Stack
What I use, by stage
End-to-end data architecture: medallion on a lakehouse, orchestrated and versioned. What's in production day to day.
01
Ingest
Python
REST / APIs
Batch & CDC
02
Orchestration
Apache Airflow
Databricks Jobs
Asset Bundles
03
Processing
Apache Spark
PySpark
Databricks
04
Storage
Delta Lake
Azure Data Lake
Unity Catalog
05
Modeling
Kimball / star
SCD Type 2
SQL
Swipe sideways to see the whole diagram →
data leaves one origin and lands in 100+ countriesvolume, velocity and variety — scale is the problem, not the detail
03 — Path
Where I've been
Sep 2023 — present
Data / Platform Engineer
Minerva Foods · remote
Data platform at one of the world's largest protein exporters — present in 100+ countries. I started in geospatial — the socio-environmental base for MyCarbon, the group's carbon subsidiary (see the case) — and then took on the platform. In August 2026 I presented a year of platform work to leadership as four deliberately ordered fronts, under one thesis: savings are a consequence, not the goal. What we built made the legacy unnecessary — shutting it down came last, not first.
02Optimizethe largest replication process, redesigned
03Separatea dedicated Data Science workspace
04Retireretire the legacy with evidence
In 2026 I design and lead the rebuild of orchestration: an unsupported Airflow 2.5.3 — 2,171 DAGs and about 47,000 tasks per business day — is being replaced by an Airflow 3 platform in 17 incremental steps, each with an automatic gate that proves its own result. A platform is not the server that runs the DAGs. It is the ability to prove, at any moment, that it is doing what it should.
Five-month intensive program in Big Data, Analytics Engineering and Data Engineering.
2022 — 2026
Information Systems (BTech)
PUC-Campinas
Bachelor of Technology. Graduating December 2026.
75 / 70,0000.1% acceptance
Selected for one of 75 seats on the Data Engineering track of Santander Coders 2023, out of 70,000+ applicants.
04 — Work
What I've built
Platform work inside a global protein operation, and an earlier geospatial case. The cost and legacy-replacement figures were formally presented to leadership in August 2026; the Airflow 3 orchestration rebuild figures are from 2026, an ongoing project, measured by the project's own suites and gates; the geospatial case is from 2023–24 and uses public sources only.
−73%cost of the largest data operation, from peak
I cut the cost of the platform's largest data operation by 73% from its peak — −42% within the first two months, before retiring any legacy system. Over the same period, the tables served grew about 10% (≈390 → ≈430) while compute consumption (DBU) fell 59%. Same delivery, same frequency, about one third of the cost per table.
100peak
58after 2 months −42%
27Aug 2026 −73%
Over the same period
+10%tables served ≈390 → ≈430
−59%compute consumption DBU
Cost of the largest data operation, indexed to peak = 100. Figures presented to leadership in August 2026; no currency amounts.
Platform cost as an engineering problem
FinOps
I took platform cost as an engineering responsibility, not a spreadsheet exercise. I redesigned the execution model so each process costs what it actually consumes, and built the governance layer that answers what each data product costs and who owns it — a question that previously had no reliable answer.
Show details
The outcome backed a formal presentation to leadership. More important than the figures: cutting cost stopped being guesswork. It now starts from cross-referenced independent signals, and keeps measured cost separate from apportioned cost instead of merging the two into one number.
−42% in 2 months−73% from peakcost per ownercross-referenced evidence
Socio-environmental geospatial base at scale
PySpark · Mosaic · GDAL
A carbon-credit company has to answer, for every farm, questions that no internal system holds: does it overlap indigenous land? a protected area? was there deforestation? what is the soil carbon? The answers are scattered across more than twenty public datasets, each with its own format, projection and update cadence. I carried out the ingestion of those datasets at MyCarbon, Minerva's carbon subsidiary — my first project there (Oct 2023 to Jan 2024).
Show details
Each source became a pipeline with its own cadence — daily to yearly —, pulled over WFS or HTTP, normalized to SIRGAS 2000, geometry validated before writing, and read at scale with Mosaic on Spark. Raster taught more than vector: 45 GB of elevation data does not belong in a database; it belongs in object storage, read on demand in a cloud-native format. That is why format comes before tooling.
Where the data lives
Lines: the 27 states covered by CAR, pulled over WFS. Colors: the 6 biomes covered by PRODES.
Amazon
Cerrado
Atlantic Forest
Caatinga
Pampa
Pantanal
Source coverage, not an analysis result. IBGE boundaries (states and biomes, 1:250k scale), simplified for the drawing.
From farm to answer
IndexEach geometry becomes cells of a hexagonal grid (H3).
FilterOnly cells touching the farm are candidates; the rest never enter the computation.
ConfirmThe exact geometry test runs only on candidates — the expensive part.
AnswerIndigenous land, protected area, deforestation: one row per farm.
farm
candidate cell
indigenous land
protected area
deforestation
Illustrative scheme, no real data. Grid indexing (H3 or BNG) is Mosaic's own design; the gain is running the expensive test only where an overlap is possible.
See inside Mosaic
Inside Mosaic
Mosaic was an Apache Spark extension for geospatial at scale, now archived. The project's own design has three layers:
Presentationgeometry display in the notebook and maps with Kepler.gl
ProcessingST_ geometry functions (JTS), Spark expressions with code generation, and grid indexing (H3 or BNG): polygons and lines are chipped into grid cells, and the spatial join becomes a key join
Datareading through GDAL/OGR — vector in parallel with multi_read_ogr, raster with gdal — and writing to Delta Lake
The core is Scala; Python, R and SQL are thin layers on top. Where I came in: I used the data layer and the raster functions, sent a fix to the documentation (PR #444) and opened a question about merging raster tiles at scale (issue #489). A maintainer explained the tile behavior I reported and placed it in version 0.4.1.
That is where the design showed the limit I hit: merging tiles in memory does not scale, and raster belongs in object storage, with a VRT index, read on demand. The successor, GeoBrix, documents that path today (COG and VRT mosaics).
The three-layer design is the one in the project README, redrawn here.
20+ public sourcesCAR across 27 statesPRODES across 6 biomesSIRGAS 2000
Data Science platform delivered end to end
Azure · Databricks
An entire Data Science team needed to move off a shared environment and start operating autonomously — over cost, configuration and delivery. I delivered the full platform — isolated networking, compute provisioned as code, and the same standard mirrored across development and production. What used to depend on a request and a wait became something the team provisions itself.
Show details
A platform nobody adopts is just cost. What drives adoption came with it: a CI/CD pipeline, handoff material and ready-made examples, so the team could work to the standard from day one — most had never worked this way. I'm the technical reference for that environment today: I hold the standard, keep track of what lands, and make sure the platform stays coherent as the team grows.
full platformdev and productionCI/CDtechnical reference
Replacing and retiring a legacy environment
Python · C#
historical load retired~89%
audited objects with a live dependency~7%
The area depended on over a thousand pipelines locked inside a visual tool — no versioning, no review, no traceable cost. I built the engine that replaced them and led the entire migration without a single data incident.
Show details
Only once the legacy had been replaced could it be switched off — the order matters, and it's what turned the shutdown into a consequence rather than a risk. I audited about 1,100 objects with a decision method grounded in cross-referenced evidence from four independent sources rather than declared inventory — only about 7% had a live dependency. I ran the retirement in waves with automated verification before each batch. Around 89% of the old environment's historical load went offline without breaking a consumer.
+1,000 pipelines migrated−89% of legacy loadzero incidents
Orchestration as an internal product
Apache Airflow
Orchestration lived inside a cloud visual tool: every pipeline was a clicked configuration — no versioning, no review, nothing reusable from one source to the next. I built the layer that replaced it — orchestration on Airflow where the pipeline comes from configuration, not from a screen.
Show details
Describing the source became enough: the pipeline is generated, lands in Git and is reviewed like any other software change. Adding a source stopped costing engineering time, and the knowledge stopped living with whoever clicked it. I account for 93 to 100% of the live code in that layer. It is the difference between delivering pipelines and delivering the ability to create them.
93–100% of live codeversioned and reviewed in Git
Orchestration rebuilt in steps that prove themselves
Airflow 3 · Docker
I inherited an Airflow 2.5.3 on Python 3.8, out of support, with 2,171 DAGs and about 47,000 tasks per business day — hand-patched libraries and logs that vanished within four days. I designed the replacement as an Airflow 3 platform on Python 3.14, with CeleryExecutor and queues per load profile, and split it into 17 incremental steps, from a clean base to cutover: each one carries a recipe, files, a gate and a rollback, and goes live one at a time.
Show details
I wrote the gates with a negative control: each one is shown failing against the defect it claims to catch. On the 3.3.2 revalidation, that exposed 13 gates that were broken as written and would have passed silently. A gate that has never been seen failing is an assumption.
The DAG migration is automated: I inventoried the legacy against the two production servers, reproduced from the repository the exact count the UI shows, and wrote a DAG converter and a custom container operator (stable name per run, resuming the previous attempt). The load gate rejects import errors, invalid cron expressions and DAGs duplicated across repositories. Result: from 43 to 96 of 99 files loading on the new version and from 324 to 0 deprecations. Migrating is not converting code: it is proving that what runs afterwards is what ran before.
During the project pause, I built a 12-DAG suite to stress the production environment: CPU, RAM, disk, long tasks, parallelism overflow, overlapping cadences and the health of every component every 5 minutes. Over four days: 157,525 tasks, 2 failures (both diagnosed and fixed), zero restarts and zero skipped schedule slots across 17,284 runs, with a p99 delay of 1.6 s.
Show details
The suite found two undocumented Airflow 3 behaviors — among them, max_active_runs=0 freezing the DAG forever — and quantified the fixed cost per task (~5 s) and the machine's real concurrency ceiling. Measured beats read: a number without a source is a guess with formatting.
157,525 tasks in 4 daysp99 1.6 szero restarts0 skipped slots in 17,284 runs
Observability without privilege, and log diagnosis at scale
Python · Linux
In an environment with no sudo and no Docker access, I built an X-ray of the machine using read-only access: /proc, per-container cgroups, PSI and a read-only role on the metastore. I showed, for instance, that the worker's memory “leak” was just file cache.
Show details
I wrote single-pass aggregators for 40,000 log files a day and computed the real cost of history: about 3.7 KB per task in the database, against about 6 GB a day of file logs in the legacy. None of these numbers needed admin access. They only needed knowing where the system already writes them.
40,000 logs a day3.7 KB per taskread-only access
Security, identity and retention in orchestration
Entra ID · Key Vault
I moved authentication from LDAP to Entra ID SSO, with group-based roles and MFA, keeping a break-glass access measured at 27 s; secrets live in Azure Key Vault, per environment. The TLS edge sits on an nginx proxy, and diagnosing the certificate chain resolved three overlapping causes that looked like a single error.
Show details
I designed the lifecycle of operational data in layers: task logs in Azure Blob with automatic tiers, the audit trail exported to an immutable container, and a backup off the machine. I validated the behavior by reading the source of Airflow and the Azure provider and proving it against the official emulator. That corrected three wrong assumptions in the internal documentation and caught a cleanup routine that deleted the audit trail without exporting it, before it ever ran in production. Documentation is evidence, not instruction: check it against the code before acting.
A dimensional model exists so the business can trust the number without knowing where it came from. I work with dimensions that preserve history and facts standardized under a single domain contract — which makes it possible to reorganize an entire domain without breaking those already relying on the data.
Show details
Validation comes before publication, not after. Nothing goes live without reconciling against the production base — and that is what makes it possible to rework data already in use without asking anyone to take it on faith.
reconcile before publishingSCD Type 2single domain contract
Introduction to Machine LearningDuke University · PyTorch
jul 2021
Cybersecurity FundamentalsIBM
jun 2021
English — advancedtechnical reading and conversation
—
06 — Radar
What I'm following
Where my attention is right now, in priority order. Highlighted, what I'm following closely.
Lakehouse
(highlighted) Cube — semantic layer for agents
(highlighted) RisingWave — streaming SQL
DuckDB — Iceberg writes
Geospatial
(highlighted) GeoBrix — successor to Mosaic, which I used at MyCarbon
(highlighted) CARTO — geo inside the warehouse
Traceability
(highlighted) TRM Labs — blockchain forensics
(highlighted) ar.io — data permanence
Currently reading
Financial Data EngineeringTamer Khraisha · O'Reilly, 2024
Domain study: what financial data demands of whoever engineers the platform. The book is tool-agnostic; what I take from it is judgment — and the recognition of what I already practice, under the name the industry uses.
Data contract
An agreed interface between whoever produces the data and whoever consumes it: structure, semantics, frequency, privacy. The book treats it as a design pattern, not a tool.
In my work: the single domain contract
Reconciliation
The same record lives in systems owned by different parties. Reconciling means aligning them into a single view before the divergence becomes risk.
In my work: reconcile against production before publishing
Lineage and audit trail
Lineage is visibility into the data lifecycle. The audit trail is its financial form: the regulator wants to see where a number came from.
In my work: lineage in Unity Catalog; an immutable trail in orchestration
Point-in-time history
In finance, data is not just today's value: it is what was known on each date. The warehouse appends records instead of overwriting.
In my work: dimensions with SCD Type 2
Quality with a tolerance
Quality has dimensions — error, duplicates, completeness, timeliness — and a tolerance set by the business, such as an error rate below 0.01%.
In my work: automated checks as a publication criterion
A living catalog
An outdated catalog costs more than having none, and the more sophisticated it is, the higher the curation cost.
In my work: Unity Catalog as the platform's catalog
What I still lack, and why I read it: financial domain knowledge — instruments, identifiers such as ISIN and LEI, and the regulation that shapes the data, such as BCBS 239. The reading is still in progress.
People I read
Simon Spätidata engineering design patternsssp.shLinkedIn