Data engineering
Building data systems: schemas, SQL performance, pipelines, migrations of data.
Download all 28
- Data engineer
Acts as a data engineer who designs for idempotency, backfills and observability, treats schemas as contracts with their consumers, and asks who depends on each table before changing it.
- Design a data pipeline
Designs a batch or streaming data pipeline sized to stated volumes, covering sources, schedule, idempotency, late data, backfills and monitoring. Use before building or replacing a pipeline.
- Design a relational database schema
Designs a relational schema from requirements and access patterns, with keys, constraints, types, indexes and DDL. Use when starting a new service or feature that stores data.
- Design a star schema
Designs a dimensional model from the questions analysts need answered: business processes, grain, facts, dimensions, slowly changing dimension types and DDL. Use when building a warehouse layer.
- Generate realistic seed data
Generates realistic, referentially consistent fixture data for a database schema, with labelled edge cases and no real personal data. Use for local development, demos and integration tests.
- Plan a zero-downtime schema change
Turns current table DDL and a desired change into expand and contract steps with lock-safe SQL, app changes, backfill, verification and rollback. Use before altering a live table.
- Review database indexes against the workload
Reviews a database's indexes against its real query workload to find missing, unused, duplicate and bloated indexes, with DDL and the write cost of each change. Use for periodic index hygiene.
- Review a database migration
Reviews a schema migration for locking risk, table rewrites, unsafe defaults, missing indexes, irreversible steps and deploy-order problems, and returns a safer version. Use before merging.
- Write a data dictionary
Writes a data dictionary for database tables with each column's meaning, units, nullability, allowed values, owner and lineage, and flags every column it cannot infer. Use when documenting a schema.
- Write data-quality checks for a table
Writes data-quality checks for a table (freshness, volume, schema, validity, uniqueness, referential integrity, distribution) with severities, thresholds and owners. Use when a table feeds decisions.
- Write a dbt model
Writes a dbt model from business logic, with declared sources, a stated grain, unique, not_null and relationships tests, column docs and a safe incremental strategy. Use when adding a dbt model.
- Analytics engineer
Acts as an analytics engineer who turns raw tables into tested, documented models analysts trust, with grain first, dimensional modelling, metrics defined once, tests, contracts and clear ownership.
- Choose a database for a workload
Recommends a database type and product from access patterns, consistency needs, volume, team skills and operations budget, explaining why the boring default usually wins and what would change it.
- Turn an exploratory notebook into a tested pipeline
Converts an exploratory notebook into a parameterised script or pipeline task with functions, config, logging and a test reproducing its key outputs. Use when a notebook moves to production.
- Data backfill track
Runs a production data backfill in gated steps, from scope and a correctness check to an idempotent batched script, a sample dry run, a throttled tracked run and reconciliation.
- Database administrator
Acts as a production DBA focused on data integrity, backups that restore, safe schema changes, query plans, capacity and least-privilege access. Use for Postgres, MySQL or similar in production.
- Database migration rules
Standing rules for schema migrations an assistant writes, keeping them backward compatible, reversible, lock-aware, batched for data changes and tested on realistic data sizes.
- Design change data capture
Designs log-based change data capture from an operational database to a warehouse, search index or cache, covering snapshot and stream, ordering, deletes, schema changes, outbox and lag.
- Design an on-device database
Designs a local database for a mobile or desktop app on SQLite, Room, Core Data or similar, with schema, list-screen indexes, migrations that never lose user data, encryption and sync scope.
- Design a save game format
Designs a game save format covering what state to persist, versioning and migration of old saves, atomic writes and checksums against corruption, cloud-save conflicts and per-platform size limits.
- Design a search index
Designs a search index in Elasticsearch, OpenSearch or Postgres full-text, with mappings, analysers, relevance tuning and a reindexing plan. Use when adding search or fixing poor results.
- Design a time-series schema
Designs storage for sensor, IoT or metrics data, covering wide versus narrow tables, time partitions, retention, downsampling, late points, cardinality and the queries it must serve.
- Implement user data deletion
Implements account and personal-data deletion across a system with a data map, delete versus anonymise choices, backups, logs, audited jobs and processors. Flags legal questions.
- Move a spreadsheet to a database
Turns a business-critical spreadsheet into a small relational database, finding hidden entities, keys and cleaning rules, with an import script and a simple data entry path for non-developers.
- Plan data archival and purging
Plans archiving or purging old data with per-table retention, partitioning, throttled deletes, verified copies and a restore path. Use when tables grow without bound or retention rules apply.
- Plan table partitioning
Decides whether and how to partition a large table, from the key in real queries and range, list or hash choice to partition size, index and constraint effects, upkeep and online migration.
- Resolve database deadlocks
Diagnoses deadlocks and lock waits from database logs or lock graphs, names the transactions and lock order involved, and fixes them with consistent ordering, shorter transactions, indexes or retries.
- Write a MongoDB aggregation pipeline
Writes a MongoDB aggregation pipeline that answers a question, explains each stage, handles missing and array fields, and recommends indexes. Use when a query needs grouping, joins or reshaping.
Not: answering business questions with data (data-analysis domain).