Data Governance: A Staff Engineer's Guide to Trust at Scale!

The Friday afternoon problem

Imagine we're running a crypto exchange called OrbitX.

The platform has Kafka for event streaming, Spark for processing, Delta Lake on S3 for storage, and dashboards serving Product, Finance, Risk, and external users. The Data Engineering team has done a solid job. Data is available, pipelines are monitored, and the business is shipping features.

Then Friday afternoon arrives.

Finance reports $1.8 million in trading fees. Product reports $2.1 million. Meanwhile, an external analytics dashboard exposes customer-level trading information that was intended to remain internal.

The investigation uncovers three problems.

First, the Finance and Product teams use different definitions of trading fees. One reports fees after certain adjustments; the other reports a broader measure. Both dashboards have valid SQL, but the metric name doesn't reveal the difference.

Second, the public dashboard's service principal can read a much broader catalog than it needs. The team relied on dashboard queries to select safe fields rather than enforcing a narrow access boundary.

Third, nobody can immediately identify who owns the business definition, who approves changes to the public data product, or which downstream reports might be affected.

Notice what's missing from this incident: a spectacular Spark failure, a Kafka outage, or a broken cluster.

The technology is doing what it was configured to do. The failure is in the rules, responsibilities, and controls around it.

A policy document may say that customer data must not be exposed. A spreadsheet may list a dataset owner. A dashboard may display a quality score. But if those things don't influence how the platform behaves, they're not enough.

A governance rule that exists only in documentation is a request—not a control.

And that distinction is where good data governance begins.

1. What data governance actually means

Data governance is the framework of ownership, decision rights, policies, standards, and controls that determines how data is defined, accessed, protected, maintained, shared, and used.

In practical terms, it answers four important questions:

• Who decides what the data means?

• Who can access it, and for what purpose?

• What guarantees make it trustworthy?

• Who is accountable when those guarantees fail?

Governance connects business decisions to engineering implementation. It establishes the rules; the platform should enforce the rules wherever enforcement is technically possible.

That doesn't mean every table needs a committee meeting before someone can query it. A temporary dataset containing synthetic test data should not face the same approval process as a financial ledger or a customer identity table.

The right level of governance depends on the consequences of getting it wrong.

These capabilities reinforce one another, but none can substitute for all the others.

A quality framework can detect a duplicate trade. It cannot decide whether the business should count that trade in a particular revenue measure. A catalog can show a dataset's owner. It cannot guarantee that the owner has the authority or information needed to resolve an incident.

And buying a governance product doesn't magically make the organization governed. That's a very expensive way to discover that software cannot attend a meeting on your behalf.

2. The six pillars that actually matter

Rather than treating governance as a mountain of policies, I would organize it around six connected responsibilities: ownership, quality, security, metadata and lineage, data contracts, and lifecycle management.

Ownership: somebody must own the meaning

Consider a dataset named fact_trade_execution.

Trading understands what an execution means. Finance understands the accounting implications of fees. Data Engineering understands how the records are ingested, transformed, and served.

Who owns the dataset?

The answer shouldn't be "the Data Engineering team" simply because the table lives in its lakehouse.

The Trading domain may own the business meaning of an execution. Finance should define which fee measures are appropriate for accounting and revenue reporting. A designated data steward can maintain definitions and quality expectations, while Engineering implements the technical controls.

One person can hold multiple responsibilities in a smaller organization. What matters is that decision-making authority is explicit.

When a metric changes, someone must be able to approve the new definition. When a quality check fails, someone must be accountable for investigating the business impact. When a dataset is exposed to a new audience, someone must approve the intended use.

Otherwise, the engineer debugging the incident becomes an accidental business decision-maker—and usually without the authority, context, or compensation for the additional job.

Staff Engineer perspective: Ownership should be connected to access approvals, change management, incident routing, and escalation. A name in a catalog is useful only if it leads to an actual decision.

Quality: green pipelines don't guarantee correct data

Suppose OrbitX processes ten million trade events without a single pipeline failure.

That sounds excellent—until we discover that the same trade was counted twice.

Pipeline reliability tells us whether the process ran. Data quality tells us whether the output meets defined expectations. Those are different things.

For a financial platform, quality rules need to reflect business invariants. Canonical trade identifiers should be unique. Asset quantities must respect defined precision and validity rules. Financial postings must satisfy the applicable ledger controls. Reconciliation checks should identify unexplained differences between the relevant records and control totals.

The severity of a failure matters just as much as its detection.

A missing optional description may be tolerable. A duplicate financial posting may require immediate investigation and containment. The response should be defined before the incident occurs, not improvised while everyone is staring at a red dashboard.

The thresholds must come from the business context. There is no universal reconciliation tolerance that should be copied blindly from one platform to another.

The important principle is that every critical rule has an owner, a measurable threshold, and a defined response.

Security: the safest data is data you never expose unnecessarily

Let's return to the public dashboard.

The dashboard needs hourly trading volume by market. The underlying source table also contains account identifiers, individual executions, fee information, and customer-related attributes.

The easy implementation is to grant the dashboard service principal access to the entire table and rely on each query to select only approved columns.

It works—until a developer adds a new view, reuses the connection, or accidentally exposes a field.

A safer design publishes a deliberately curated dataset, such as public_market_volume_hourly, containing only approved aggregates. The dashboard identity receives read access to that dataset rather than broad access to the internal catalog.

This is the principle of least privilege: each user or workload should receive only the access needed for its intended purpose.

In practice, that means using appropriately scoped identities, table or column restrictions where required, row-level controls where relevant, auditable permission changes, and regular reviews of access that is no longer needed.

Classification matters too. Data should be categorized according to the organization's policies, so the platform can distinguish public, internal, confidential, and restricted information.

One caution: masking a name doesn't automatically anonymize a record. A unique account identifier combined with precise timestamps and trading activity may still reveal someone's identity or behavior. Aggregated public data can also leak information through small groups or repeated queries.

The right question isn't merely "Did we remove the sensitive column?" It's "Could this data still expose something we have no business revealing?"

Metadata and lineage: make the data explain itself

Imagine a Finance analyst opens a table called daily_revenue.

Does it mean fees charged, fees after rebates, or accounting-recognized revenue? Who owns the definition? What source tables feed it? When was it refreshed? Which reports depend on it?

If the analyst has to ask three people and inspect twelve SQL files to answer those questions, the platform has a discoverability problem.

A data catalog and business glossary can make ownership, definitions, classification, freshness, and intended use visible. Lineage can connect source events and tables to transformations, curated datasets, and downstream dashboards.

That becomes especially valuable when an upstream field changes. If the meaning of fee_amount changes, lineage helps identify the consumers that might be affected.

But lineage is not automatically complete. Dynamic SQL, external processing, and undocumented dependencies can leave gaps. Those gaps should be acknowledged rather than hidden behind a convincing diagram.

The objective isn't to draw the prettiest lineage graph. It's to make impact analysis and investigation faster and more reliable.

Data contracts: technical compatibility isn't business compatibility

Suppose a producer changes a Kafka event. The schema remains technically compatible, so the deployment succeeds.

But the field fee_amount used to mean "fee charged" and now means "fee after rebate."

No column was removed. No data type changed. The consumer still runs.

And the business metric is now wrong.

A useful data contract defines the meaning of fields, their units, required values, identifiers, timestamp semantics, expected quality, and the rules for handling changes. It should also clarify how duplicates, corrections, and late-arriving events are represented.

A schema registry can help enforce structural compatibility. Contract tests can validate expected behavior. Neither can decide what the business means by revenue.

For a critical contract change, the team should understand downstream impact, test consumers, agree on a migration strategy, and assign responsibility for the change.

Not every optional field addition needs a committee. But changing the meaning of a financial measure deserves more than a casual merge approval.

Lifecycle: governance doesn't end when the data lands in S3

Different datasets have different retention and deletion requirements.

Financial records may need to be retained under applicable accounting or legal obligations. Operational logs may have shorter retention periods. Temporary staging data may be eligible for earlier cleanup. Personal information may be subject to privacy and deletion requirements.

The appropriate rules depend on the dataset, its purpose, the applicable jurisdiction, and any legal holds. There is no single retention duration that applies everywhere.

Engineering's job is to make those rules enforceable and verifiable.

That means understanding where copies exist, how archival works, which downstream datasets contain derived information, and how deletion or preservation requests are handled.

Deleting a row from a serving table doesn't necessarily delete it from raw archives, backups, extracts, or downstream copies.

A lifecycle policy is credible only when the organization can identify the relevant data and demonstrate what happened to it.

3. The architecture: governance must travel with the data

Here's how I'd approach governance on a platform built with Kafka, S3, Spark, Delta Lake, and an analytics layer.

The architecture separates data processing from the cross-cutting controls that determine how data is defined, validated, accessed, and published.

End-to-end governed data flow

This is a logical architecture; the controls may be implemented through integrated platform capabilities or separate tools.

The Bronze layer retains source data with appropriate metadata and access restrictions. It isn't exempt from governance simply because the data is raw.

The Silver layer applies validation, canonical definitions, deduplication, and domain rules. The Gold layer exposes curated models with documented meaning and intended use.

Internal Finance and Risk consumers receive appropriately restricted datasets. External consumers receive a deliberately approved public data product.

Across all layers, identity, classification, contracts, quality, lineage, and auditability help establish whether the data is being handled as intended.

One distinction is essential: Kafka transports events, and the lakehouse stores and transforms analytical data. Neither automatically becomes the authoritative financial ledger merely because it contains financial records.

A successful ingestion job proves that data moved through a pipeline. It doesn't prove that the data was appropriate to publish, that the business definition was correct, or that the access permissions were safe.

Those need their own controls.

4. One metric, two answers: governance in action

Let's resolve the conflicting-revenue incident.

Finance reports $1.8 million in trading fees. Product reports $2.1 million.

The first step is not to decide which dashboard looks more trustworthy. It's to investigate the definitions.

Perhaps one measure includes gross fees charged before rebates, while the other excludes certain adjustments. Perhaps the reporting periods differ, or corrections are handled differently. These are possibilities to test, not assumptions to make.

The solution is to define the metrics explicitly.

Now the organization can distinguish gross fees charged, net fees after adjustments, and accounting-recognized revenue where those measures are needed.

Different teams may legitimately need different measures. Governance does not require forcing them into one number. It requires making each number's meaning clear and ensuring that comparisons are meaningful.

The same principle applies to public data. The platform should expose an approved aggregate, enforce access at the appropriate boundary, and validate the dataset before publication—not hope that every dashboard developer remembers every restriction.

That's the difference between a governance policy and an operational governance capability.

5. How much governance is enough?

This is where I'd challenge a common mistake: applying the same controls to every dataset.

If every access request requires multiple approvals, every schema change triggers a committee, and every experiment needs a lengthy compliance review, developers will find workarounds.

The answer is not to remove controls. It's to make them proportionate to risk.

The exact requirements should follow the organization's risk framework and obligations. The table is a design starting point, not a universal compliance standard.

For implementation, I would automate the checks that are repetitive and deterministic: required ownership metadata, schema compatibility, access scopes, quality rules, and expired permissions.

Human judgement should remain where context matters: approving new uses of sensitive data, resolving conflicting business definitions, interpreting legal requirements, and accepting documented risks.

Automate the rules. Keep accountability with people.

6. How I'd roll this out as a Staff Engineer

I wouldn't start by trying to govern every table in the company. I'd start with the data products where failure has the highest cost: financial records, customer data, regulatory reporting inputs, critical metrics, and public datasets.

First, establish ownership and classification for those datasets. Next, agree on the minimum contracts, quality rules, access boundaries, and lineage requirements. Then embed the controls into the existing ingestion, transformation, and publication workflows.

Once that pattern works, expand it to other domains.

The aim is to turn governance from a recurring manual exercise into a capability built into the platform.

I'd measure whether it is working through outcomes such as:

Set baselines before choosing targets, and measure effectiveness rather than checkbox completion. A catalog can list an owner who never responds. A lineage graph can exist while missing important dependencies. An access review can be completed without meaningful scrutiny.

The point isn't to prove that governance paperwork exists. It's to prove that the controls reduce risk without making everyday engineering unnecessarily difficult.

7. What I'd challenge in a design review

Three mistakes deserve particular attention.

First, buying a catalog and calling the problem solved. A catalog improves discovery, but it doesn't automatically establish accountability, enforce access, or guarantee quality. The metadata must connect to real operational processes.

Second, treating the lakehouse as the system of truth for everything. Analytical tables are useful for reporting, but they don't automatically replace transactional controls, financial posting rules, or reconciliation. The authoritative source must be defined for each business fact.

Third, designing governance only for audits. If controls exist only to satisfy an audit, engineers will see them as overhead. The better design makes governance useful in daily work: safer self-service, faster investigations, clearer definitions, and fewer surprises.

These mistakes have something in common: they treat governance as a layer added around the platform instead of a set of engineering decisions embedded within it.

8. Questions a Staff Engineer should be ready to answer

Who owns data governance: Engineering, Security, or the business?

It is a shared responsibility with explicit decision rights. Business domains own meaning and intended use; Engineering implements and operates platform controls; Security, Privacy, Legal, and Compliance contribute the controls and expertise within their remit.

Does a data catalog guarantee good governance?

No. It helps discover and document data. Governance also requires ownership, enforceable access, quality rules, lifecycle controls, and accountability.

How do you prevent sensitive data from reaching a public dashboard?

Publish a deliberately curated data product, scope the service identity to that product, enforce access restrictions, and assess disclosure risk. Do not rely solely on dashboard queries to filter sensitive fields.

How do you resolve conflicting business metrics?

Document the intended meaning, formula, grain, time basis, and inclusions or exclusions. Assign a decision owner and publish clearly named metrics. Different measures can coexist when their definitions are explicit.

How do you avoid slowing developers down?

Automate repeatable checks, provide self-service within approved boundaries, and scale review requirements to risk. Reserve manual approval for decisions that genuinely require judgement.

How do you know governance is working?

Look for fewer unauthorized disclosures, fewer unresolved ownership gaps, earlier detection of contract failures, faster investigations, clearer business definitions, and appropriate access without unnecessary delays.

The final perspective: governance is an engineering capability

I don't see data governance as a separate phase that begins after a platform has been built.

When we define a data contract, we're governing meaning. When we scope a service identity, we're governing access. When we enforce a financial reconciliation check, we're protecting an invariant. When we record lineage, we're making the platform explainable.

And when we define ownership, we're ensuring that a failure has someone empowered to resolve it.

Engineering should not make every business or regulatory decision. But it should build systems in which agreed decisions can be enforced, observed, and tested.

A policy that cannot be implemented is an aspiration.

A control that cannot be verified is an assumption.

And a dataset nobody owns is an incident waiting for a calendar invite.

Your data platform doesn't earn trust because it stores billions of records, runs on a thousand executors, or publishes a dashboard in three seconds.

It earns trust when the right people can use the right data for the right purpose—and the organization can explain why that data deserves to be trusted.

Governance isn't about putting more locks on every door.

It's knowing which doors matter, who should hold the keys, and having the system notice when someone opens the wrong one.

That's not bureaucracy.

That's engineering.

#DataGovernance #DataEngineering #DataArchitecture #DataQuality #DataSecurity #DataPlatform #FinTech #StaffEngineer