{
 "title": "Today's bulk market-data file may not be final \u2014 StockQL",
 "description": "Design daily ingestion for mutable bulk snapshots, late rows, placeholder data, and explicit-date completeness checks.",
 "url": "https://stockql.ai/blog/todays-bulk-file-may-not-be-final.html",
 "slug": "blog/todays-bulk-file-may-not-be-final",
 "published": "2026-10-11",
 "site": "StockQL",
 "contact": "hello@stockql.ai",
 "body_text": "Market-data tradeoffs \u00b7 Bulk ingestion\n\nToday\u2019s bulk market-data file may not be final\n\nA successful download proves that a snapshot existed. It does not prove that every symbol for that trading date had reached the file.\n\nSeptember 13, 2026 \u00b7 5 minute read\n\nIn this article\n\nTreat bulk output as a snapshot\n\nDefine observable completeness\n\nRecheck explicit dates\n\nCompare publication systems fairly\n\nBulk endpoints make daily ingestion efficient: one response can replace thousands of per-symbol requests. Their operational risk is equally concentrated. If a once-daily job reads a file while it is still filling, every late row becomes a persistent hole.\n\nHTTP 200, valid CSV, and a plausible row count are transport checks. Completeness is a separate property that has to be observed over the provider\u2019s publication cycle.\n\nDo not infer finality from success. Unless the provider supplies and honors a completeness marker, design the file as mutable for a bounded period.\n\n1. Treat bulk output as a snapshot\n\nStockQL measured EODHD\u2019s US bulk-last-day endpoint on August 21, 2026. About two hours after the close, at 22:02 UTC, the response contained 19,559 rows. The finalized file addressed by the same trading date later contained 50,247 rows.\n\nThe difference was not merely empty catalog padding. Some late rows carried real volume as high as 32.8 million shares. A single early fetch would have left active, liquid securities missing for that date.\n\nThe final file also contained roughly 33,000 flat zero-volume placeholder rows. Counting every line as a usable market observation would make the later file look complete while importing data that did not represent a trade. Completeness must be defined over eligible, usable rows\u2014not bytes or CSV lines.\n\n2. Define observable completeness\n\nA production ingest can measure what the provider does not declare:\n\nRecord the requested trading date, fetch time, response digest, total rows, usable rows, and per-venue counts.\n\nCompare the eligible universe with symbols that received a non-placeholder observation.\n\nDistinguish a market holiday from a thin or partial file using an audited exchange calendar.\n\nAlert on cohort-sized absences rather than immediately changing thousands of security lifecycles.\n\nKeep missingness visible until a later snapshot resolves it or the lifecycle system supplies a different explanation.\n\nNo single threshold proves completion across every exchange. The contract is a bounded re-observation policy plus explicit residuals.\n\n3. Recheck explicit dates\n\nStockQL\u2019s daily pipeline performs a bulk-completeness re-pass. It fetches the prior explicit trading date again and can revisit the preceding dates when the stored universe still has a material missing cohort. It inserts only absent usable rows and skips flat placeholders.\n\nThis shape is idempotent: a repeated final file does not duplicate bars, while a later-expanded file fills the holes. Because the request names the date explicitly, the re-pass does not accidentally slide to a newer \u201clast day\u201d while the old date remains incomplete.\n\nLate data and delisting logic also stay separate. A symbol missing from one partial snapshot is not a lifecycle event. Only sustained availability evidence, guarded against correlated provider incidents, can move that state.\n\n4. Compare publication systems fairly\n\nFMP\u2019s bulk endpoint has a different shape: one measured response covered roughly 66,000 global rows in 3.8 MB and required client-side filtering for an exchange. Historical dates can also carry adjusted-price semantics that differ from the latest-day file. It is efficient, but it does not make publication timing or field mapping disappear.\n\nStockQL can compare symbol presence across feeds to surface a suspicious cohort, then return to each provider\u2019s own dated endpoint for the serving fact. The second feed is an alarm and an arbiter, not permission to merge unlike rows blindly.\n\nIngestion invariant. Every trading date remains reopenable until the completeness window closes. A file may add facts during that window, but a transient omission must not become a delisting or a permanent gap.\nNext: why a ticker is not an earnings transcript\u2019s identity \u2192"
}