When your data isn’t fully open

You’ve built something good, but you can’t share all the data. Now what?

A lot of research software uses data that isn’t fully open. Some common cases:

  • Commercial airborne LIDAR used as ground-truth labels for satellite-derived bathymetry, forestry, or urban modelling. Costs $100k+; vendor licenses forbid redistribution.
  • Medical imaging with patient data protected by IRB / HIPAA / GDPR.
  • Restricted satellite imagery — Planet SkySat, WorldView, and other sub-meter commercial products.
  • Species-occurrence data under conservation embargo (moving nesting sites, endangered-species locations).
  • Human-subjects survey data with re-identification risk.
  • Industry partnership data covered by NDA.

You cannot post the raw training data. But you can still publish a citable, installable, reproducible open-source package — you just need a slightly different playbook. This chapter describes it.

The bad path is to make your code private-repo-only (“it’s not useful without the data”) or to publish it without any reproducible example (“bring your own data”). Both waste the effort you already put in. There’s a middle path.

Three levers you can pull

Lever 1: Ship trained model weights, so inference is reproducible

Even when your training data is closed, your model weights usually aren’t. You trained them; you own them (check your data agreement — sometimes vendor contracts contain clauses about derived products, but most academic licenses do not).

Publish the trained weights alongside your code as an inference-only demo:

  • Hugging Face Hub (huggingface.co) is the standard modern venue. Free hosting for public models, model cards for licensing/provenance, integrates with transformers and PyTorch.
  • Zenodo works too — DOI-backed, but no ML-specific tooling.
  • GitHub releases for small weights (< 100 MB per file).

The value proposition: a reader can install your package, download the weights, and run inference on their own data — reproducing the science even without seeing your training set. For many ML papers this is enough.

Add a section to your README:

## Trained model weights

The pretrained models from our paper are hosted at:

- Hugging Face:  https://huggingface.co/yourorg/yourthing-weights
- Or download directly:  yourthing.load_pretrained("bathymetry-v1")

Weights are released under [licence]. See MODEL_CARD.md for details on
training data, intended use, and known biases.
TipPublish a MODEL_CARD.md

The model card format — a short structured description of a trained model — has become standard in ML publishing. What data it was trained on, what tasks it was validated on, what biases it may have, what its intended use is. JOSS reviewers appreciate it because it substitutes for the raw data they can’t see.

Lever 2: Build a smaller open-data end-to-end example

Even a small open-data example that runs end-to-end has enormous value. It lets a reader:

  • Verify your install works,
  • See a training loop actually complete,
  • Sanity-check your pipeline on a case they can compare to a public baseline,
  • Adapt the code to their own data by editing one config file.

This is the workflow reviewers will actually run before they check the “installation works” box on the JOSS review. Give it to them.

Concrete: for a bathymetry package trained on commercial LIDAR, build a demo notebook that trains on public bathymetry (NOAA’s Continuously Updated Digital Elevation Model, the GEBCO bathymetry, OpenBathymetry). The scientific accuracy will be worse than the paper’s result — say so explicitly — but the pipeline itself is fully reproducible.

For medical imaging on private records: switch to a public dataset (BraTS, OASIS, PhysioNet). The pipeline demonstrates itself; the private-data result stays in the paper.

For restricted satellite imagery: switch to Sentinel-2 or Landsat 8/9 (both open-access) for the demo, even if the paper reports on commercial WorldView data.

This isn’t a compromise — it’s the version of your software that most people will actually use, because most people don’t have your private data either. The demo is your marketing.

Lever 3: Document data access clearly, for the readers who do have access

The subset of readers who do have access to the private data (fellow employees at the vendor, other IRB-approved researchers, licensees) need clear instructions to reproduce the full paper. Write a docs/DATA.md file that spells out:

  1. Where the data comes from (vendor, dataset name, version).
  2. How much it costs, if applicable, and how to purchase it.
  3. What academic-access programs exist (e.g., Planet’s Education and Research Program, NASA’s DAAC user agreements).
  4. What IRB / DUA / MTA / NDA process is needed, if applicable.
  5. Once obtained, where the code expects to find it (data/lidar_raw/, environment variables, etc.).

Someone who has jumped through those hoops before will follow your instructions in 20 minutes. Someone who hasn’t will realise reproducing your paper isn’t feasible for them, and will pivot to the open-data demo instead. Both outcomes are fine.

What this looks like in practice

A well-set-up mixed-open-closed repo has:

yourthing/
├── README.md               # summarises the three-lever situation up top
├── docs/
│   └── DATA.md             # detailed data access instructions
├── examples/
│   ├── open_data_demo.ipynb    # trains on public data, works for anyone
│   └── full_paper_run.ipynb    # documents the private-data run, requires DATA.md setup
├── weights/                # published to HF Hub or Zenodo; NOT in git
│   ├── MODEL_CARD.md
│   └── (download instructions)
├── yourthing/
│   └── ...                 # the code itself is fully open-source

The code is fully open. The weights are downloadable and licensed for use. The open-data demo works out of the box. The private-data path is documented but requires the reader to have access.

This is a completely legitimate open-source research package, and JOSS reviews these all the time. The Statement of need in paper.md should be honest about the situation — one sentence like “The models in the paper were trained on commercial LIDAR bathymetry; the shipped weights allow inference on user-provided imagery, and the demo notebook trains an equivalent model on public GEBCO bathymetry.” — is enough.

What not to do

  • Do not “sanitise” the private data by publishing a downsampled or aggregated version unless your data-use agreement explicitly permits it. Reidentification is a live risk in medical / geospatial / behavioural data even from summary statistics.
  • Do not fake a demo by fitting your model on the private data and then presenting the weights as if they’d been trained on public data. Reviewers will find out (they always do), and you’ll have burned your credibility.
  • Do not hide the fact that key data is closed. Reviewers will notice, and unstated restrictions look worse than stated ones.

What JOSS reviewers actually check

The JOSS review checklist has an item about “Example usage” and one about “Automated tests”. Both can be satisfied without the private data:

  • Example usage: your open-data demo notebook.
  • Automated tests: cover the code paths that don’t depend on the private data (data loading, model architecture, feature engineering, evaluation metrics).

The checklist does not have an item that says “the reviewer can reproduce the exact numbers in the associated science paper.” That’s the science paper’s problem, not JOSS’s.

Checklist