Skip to content

Contributing

Architecture

Source                     --->  DataSchema (schema.py)  --->  Renderer
files, folders, databases,       no values by design           JSON Schema, text, TypeScript,
URLs (sources/)                        |                       Pydantic, SQL, summary
                                       +---------------------> Synthetic data (synth/)
                                       +---------------------> Code checks (check/)
                                       +---------------------> Schema diff (diff.py)

Text  --->  Detectors (patterns, secrets, NER, packs, backends)  --->  Placeholders (text/)
Module Purpose
schema.py Source-neutral model: DataSchema > Entity > Node
sources/ Source interface, registry and input readers (sql_source.py covers every SQL database)
sources/directory.py Describes a folder: one entry per file, with file names held to the same rules
sources/files.py What file sources share: opening a path, a compressed file or a file at a URL, typing cell text, and telling a header from data
inference.py Builds nodes from sampled values, for sources without a catalog
privacy.py KeyPolicy, and which names are structure and which are data
formats.py Recognises string formats, returning only a label
keys.py The opt-in hints drawn from values: --infer-keys, --column-hints and --public-column
scale.py --scale-hints: a row count as an order of magnitude
selection.py --include, --exclude and --max-tables, on names that may be shown only
renderers/ Output formats
config.py Reads and merges datanyx.toml and datanyx.local.toml, and rejects mistakes in them
diff.py Compares two schemas and sorts the changes by whether they could break old code
audit.py The MCP audit log and its summary: tools, paths and outcomes, never values
debuglog.py The MCP debug log: where a failure happened, never its message
fileperms.py Makes a file readable by its owner only, on every system
scan.py Finds personal data and secrets in a folder, by file, kind and line, without values
protect/ datanyx protect and the guard hook: Claude Code permissions, a hook that refuses direct access, MCP config; guard.py decides, shell.py reads a command the way a shell would, paths.py compares paths and programs.py knows what git and DataNyx do with their arguments
check/ Checks SQL and pandas code against a schema: unknown names and type mistakes
scrub.py Removes data from error messages and program output, keeping the diagnosis
synth/ Generates fake data that fits a schema, clean or messy, and writes it as JSON, JSON Lines, CSV or SQLite
mcp_server.py MCP server that only exposes schemas, synthetic samples and anonymized text
text/detectors.py Finds sensitive spans: patterns, and spaCy NER through Detector
text/documents.py Anonymizes HTML, email, Word, PDF, WhatsApp and Slack files, keeping their structure
text/batch.py Folder and stream drivers: one set of placeholders across many inputs, and restoring a folder
text/mapping.py The mapping file: created private, never overwritten, rewritten as it grows
pseudonymize/ Consistent fake values for tables: key-derived fakes, id permutation, column classification, CSV, JSON and SQLite
evaluation.py Measures recall by kind over labelled documents, for datanyx eval
text/packs.py Healthcare and finance detector packs, with check-digit validation
text/backends.py Presidio, GLiNER and local-LLM (Ollama) detectors, loaded only when asked for
text/custom.py Detectors described in a file: regular expressions and dictionaries of terms
text/secrets.py Finds credentials by their shape: API keys, tokens, private keys, passwords
text/anonymizer.py Merges spans, finds repeat mentions and assigns placeholders. Anonymizer keeps them across many texts
cli/ Command line, which reports errors without quoting data: one module per command (schema.py, synth.py, scan.py, ...), parser.py for the arguments and common.py for what they share

Documentation

The pages are in docs/ and are published with MkDocs (uv sync --group docs, then uv run mkdocs serve). tests/test_docs.py keeps them honest: every page is in mkdocs.yml, every link and heading it points at exists, and every command is explained on some page.

docs/cli-reference.md is generated from the argument parser, so change a command's help in cli/ and run uv run python tools/gen_cli_reference.py; the test fails while the page is stale. The accuracy tables in docs/anonymization.md come from uv run python tools/ner_benchmark.py --write.

Adding a source

Implement Source and add it to the registry in sources/__init__.py. Documents, API responses and CSV rows can reuse Observation from inference.py, and sources that know their types up front (like Parquet) can use build_object from sources/catalog.py.

A SQL database with a SQLAlchemy dialect needs no new source. Add its short URL, driver and connection timeout argument to sql_source.py, add the extra to pyproject.toml, and add a server to the live tests. Anything else with a catalog should read it directly and build Object nodes, with one Entity per table, so it never has to query any rows.

Every new source needs a builder in tests/test_canary_everywhere.py: a small function that writes a file with the canary planted in values, names, comments, defaults and metadata. The test then runs it through the library, the command line and the MCP tools. A source that can't be written to a file (a database server) gets a test in the same file, or a server test. A source with no such test is documented as experimental until it has one.

Development

The project uses uv. uv sync creates .venv from uv.lock, so everyone gets the same versions:

uv sync --all-extras                 # every reader and database driver, plus the dev tools
uv sync --all-extras --group models  # also the large NER models (about 500 MB)
uv run datanyx doctor                # shows what's installed

Tests

uv run pytest                        # needs no services; model tests skip without the models
uv run pytest -n auto                # in parallel, in about half the time (each worker loads the models,
                                     # so use -n 4 if a MemoryError shows up)
uv run ruff check src tests
uv run mypy
uv run pytest --cov                  # coverage report

Where a test goes. Tests are named for the behaviour they cover (test_scan_reports.py, test_mcp_access_and_tools.py, test_document_anonymization.py), not for the review or the fix that produced them. Put a new test with the behaviour it checks. The test_security_*.py files are different on purpose: each test there is a way past one of DataNyx's promises that once worked, kept so it can't come back, grouped by what was attacked (the MCP server, the guard, schema values, documents, resources and logs). A new bypass fix goes there, with a comment on what the bypass was. Helpers used by more than one file live in a small module next to the tests (text_corpus.py, docx_helpers.py).

The coverage floor in pyproject.toml is set from a measurement, with the number and the reasoning beside it. Raise it when coverage really goes up; don't lower it to get a change through.

tests/test_fuzz_leaks.py generates documents with a canary planted in every value, in JSON, JSON Lines, CSV, YAML and XML, and fails if the canary reaches any output format or note. Run it with other seeds using uv run pytest tests/test_fuzz_leaks.py --hypothesis-seed=N.

To run the server tests against real databases, using Docker:

docker compose -f tests/docker-compose.yml up -d --wait
DATANYX_TEST_SERVERS=1 uv run pytest -m server
docker compose -f tests/docker-compose.yml down

See the CHANGELOG.md in the repository root for what's new.