Data Engineering for AI: why your data matters more than the model
AI projects rarely fail because of the model. They almost always fail because of dirty, inconsistent or inaccessible data. Here's how to get ready.
The industry spends a lot of energy debating GPT vs Claude vs Llama. The reality: if your data is dirty, no model will rescue the project. And if your data is clean, almost any decent model works.
The 80/20 rule of AI projects
In enterprise AI projects, 60-80% of the real effort goes into data engineering: ingestion, cleaning, structuring, indexing, monitoring. Only 20-40% is training or connecting models. Teams that don't budget for this get stuck halfway.
The 4 layers of the data stack for AI
1. Ingestion
Bring in data from every source: ERPs (SAP, Oracle, Microsoft Dynamics), CRMs (HubSpot, Salesforce), spreadsheets, vendor APIs, operational databases, product events. Tools: Airbyte, Fivetran, custom Python pipelines.
2. Transformation
Clean, structure, deduplicate, normalize. Build dimensional models (facts + dimensions) for analytics plus wide models for ML. Dominant tool: dbt (versioning, tests, documentation).
3. Storage
Analytical warehouse (Snowflake, BigQuery, Redshift, Databricks) for BI + ML features. Vector DB (Pinecone, pgvector, Weaviate) for embeddings of proprietary documents used in RAG.
4. Serving
Low-latency APIs so applications (agents, dashboards, products) can consume the processed data. Feature stores (Feast) for ML. Caches (Redis) for recurring queries.
The 5 data problems that kill AI projects
Silent duplicates
The same customer exists 3 times in the CRM with name variations. The model trains on the duplicate, predicts poorly, and nobody understands why. Solution: explicit deduplication rules + monitoring for new duplicates.
Undeclared missing data
40% of records lack a critical field, but nobody knows because it isn't documented. The model learns to ignore the field or to use NULL as a signal, producing biased predictions. Solution: automated data quality tests (dbt tests, Great Expectations).
Inconsistent histories
The "customer type" field changed from 3 categories to 7 two years ago. Old data is in the old schema, new data in the new one. The model doesn't know. Solution: mapping tables and schema versioning.
Time zones and date formats
Mexico has 3 time zones. Dates arrive in UTC from the CRM and in local time from the ERP, and the model predicts churn with a 6-hour gap between events that were actually simultaneous. Solution: normalize to UTC + flag the original event's time zone.
Schema drift without alerts
The product team added a field to the transactions table. The pipeline keeps running but ignores the field. Three months later, nobody remembers. Solution: schema change monitoring + alerts.
Recommended pattern for Mexican SMBs
Minimum viable stack for a 50-250 person company starting with AI:
- PostgreSQL (or the warehouse you already use) as the source of truth
- dbt + GitHub Actions for versioned transformations
- Airbyte (free as open source) for ingestion from common SaaS tools
- pgvector in Postgres for embeddings (you skip Pinecone for the first 6-12 months)
- Metabase or Lightdash for dashboards
- Great Expectations for data quality tests
This stack costs $200-$800 USD/month in infrastructure for SMB volumes, versus $2k-$5k+ USD/month for enterprise solutions. You scale when volume justifies it.
The right order to build
- 1Identify 1-2 specific AI use cases you want to enable
- 2Map which data you need for those use cases (no more, no less)
- 3Build pipelines ONLY for that data — don't try to build a complete data warehouse all at once
- 4Add data quality tests before using the data in production
- 5Deploy the AI use case on top
- 6Iterate: add sources and use cases one at a time
The most common mistake is trying to build "the complete data warehouse" before touching AI. It takes 18 months, eats the budget, and by the time it's done the business has already changed.
Signs your data foundation is ready
- You can answer your basic operating KPIs in <5 minutes with data, not intuition
- Your data team can answer "what's the source of truth for X?" without hesitating
- Automated tests catch data anomalies before they reach dashboards
- Table and field documentation exists and is kept up to date
- There's a clear policy on who can access which data
If you don't check 3+ of the 5 above, spend the next 2-3 months on data engineering before any ambitious AI. It will cost you 10x less than trying to do both in parallel.
What you may also be wondering
Do I need a data warehouse to do AI?
Not necessarily. For narrow cases you can run AI directly on operational databases with targeted pipelines. For complex or multiple cases, yes — the warehouse avoids duplicated effort and gives you a single source of truth.
Which warehouse should I choose: Snowflake, BigQuery or Databricks?
Snowflake if you already pay for enterprise and want pure analytics. BigQuery if you're already on Google Cloud. Databricks if you combine analytics + heavy ML. Postgres + dbt if you're starting on a small budget. There's no universal answer — it depends on context.
Who does the data engineering: my internal team or a vendor?
Ideally, a hybrid. An external vendor during the build phase (3-6 months), with hand-off and training for the internal team that maintains it afterward. Consultancies that build and leave behind them leave invisible technical debt.