Parsing the SAM.gov daily bulk extract: format, quirks, gotchas

GSA publishes free daily bulk extracts of Contract Opportunities data — the sanctioned high-volume alternative to the rate-limited API. This is the guide we wished existed before writing our own ingester; it doubles as the source for our data dictionary.

Gotcha 1: it's big — stream it

The extract is a full daily snapshot that can run to hundreds of megabytes. Don't buffer it; parse incrementally. Your CSV parser must survive a chunk boundary landing inside a quoted field:

N001,"Software maintenance, enterprise case
management",541511,SBA,...

That's a comma and an embedded newline inside quotes. A split-on-newline parser silently corrupts rows. Ours is a character-level state machine with an escaped-quote ("") state, tested by feeding fixtures in 1-byte chunks.

Gotcha 2: the columns

The header row (47 columns as of 2026-08-18) includes NoticeId, Title, NaicsCode, SetASideCode, Department/Ind.Agency, PostedDate, ResponseDeadLine, Active, Link — but treat the header row as authoritative on every run and map by name, never by position (the official data-dictionary PDF even spells a few names differently from the live file). Note the description situation: the extract's final Description column carries the notice's plain-text description inline — flattened, with a nominal 5,000-character cap that some records exceed — while the public API serves descriptions as links costing an extra rate-limited call.

Gotcha 3: diffing a snapshot

The extract tells you what is, not what changed. To get events you must keep yesterday's state and diff: hash each row's significant fields, compare by NoticeId, and treat disappearance as archival. Deadline moves are the highest-value events and only exist if you diff field-by-field.

Or skip all of it

This pipeline — streaming parse, hash index, field diffs, retries, dead-letter queues — is exactly what SamHooks runs so you don't have to (DIY vs $19/mo math, quickstart).