Skip to content

Watching a data feed for changes

Code written against a data feed breaks when the feed changes shape. Save its schema once, then compare against it whenever a new export arrives:

datanyx schema feed.csv -o schema.snapshot.json     # once; commit the snapshot
datanyx diff schema.snapshot.json feed.csv          # on every new export, in CI
$ datanyx diff schema.snapshot.json feed.csv
2 breaking, 1 compatible

breaking:
  ~ $[].email: string<email> -> string<email> | null  (can now be null)
  - $[].notes: removed

compatible:
  + $[].score: added  (number, required)

Either side can be a saved schema or any input DataNyx reads: another file, a folder, a database.

Change Counted as
A field or table removed breaking
A type widened: a number may now be text, a field may now be null, an integer may now have decimals breaking
A format lost, or changed (email to none, date to date-time) breaking
A field that was always there is now optional breaking
A field or table added compatible
A type narrowed, a format gained, an optional field now always present compatible
A database type changed, a primary or foreign key changed, key names newly hidden or shown, a type that became or stopped being unknown info

--fail-on breaking (the default) exits with 1 when anything breaking changed, --fail-on any for any difference and --fail-on never to just report. --format json is for scripts, and the options for reading inputs (--source, --schema, -r, --map-path, --keep-name) work as in datanyx schema. Nested changes have paths such as $.user.address.zip or users[].tags[]. The MCP server has a schema_diff tool, and the GitHub Action runs diff on pull requests.

A schema built from a file is built from what the file contains, so a difference can come from the data in a new export, not only from its design: a column that was never empty can show up as nullable.