Prompt·Writes and ingestion·PG07
Is our ingestion process more complicated than necessary?
The import pipeline is long, slow to correct, or does the same work at several stages.
Prompt
Trace one complete ingestion cycle from the original source to the final database records. Identify repeated parsing, unnecessary intermediate states, duplicate validation, repeated reads, redundant transformations and writes that could be consolidated. Recommend the simplest workflow that preserves accuracy, traceability, failure recovery and the ability to process corrections.
- What it returns
- One real trace from source to final rows, the work done twice, the checkpoints, a shorter path, and checks that the shorter path gives the same results and still replays.
- Skip it when
- The pipeline is already one short step from source to table.
- Avoid
- Removing evidence, provenance or checkpoints that recovery depends on.
- Changes
- Changes nothing
- Sessions
- Look first, then fix in a new session
- Tools
- Repository and read-only database access.
- Works on
- Any PostgreSQL host: Supabase, Neon, RDS, Cloud SQL or self-hosted.
Related prompts
- Imports that fail quietlyPromptFinds rows that go missing or land on the wrong record during import, with no error.
- Map every place that writes to the databasePromptSeveral jobs, functions, APIs or agents change the same data, or nobody is sure who owns it.
- Stop rewriting data that hasn't changedPromptUpserts and updates run at high volume, imports are costly, or vacuum and write-ahead log churn look high.
- Can we rerun everything without damaging the data?PromptImports get retried, schedulers overlap, files are replayed, or runs fail halfway.
- Find where write logic should livePromptBusiness rules are split across SQL, triggers, edge functions, server routes and workers.
More in Writes and ingestion.