Skip to content

MDS Delivery and DNT

Delivery protocol

HCMR delivers MED-WAV products to the Copernicus Marine Data Store (MDS) as a Production Unit.

The NRT delivery sequence is:

sequenceDiagram
    participant PU as HCMR / JUNO PU
    participant FTP as MDS FTP Buffer
    participant DV as Delivery Validator
    participant MDL as Marine Data Lake

    PU->>FTP: Upload NetCDF files
    PU->>FTP: Upload DNT XML
    FTP->>DV: Trigger delivery processing
    DV->>DV: Validate MD5 / metadata / DNT
    DV->>MDL: Publish valid native files
    DV->>FTP: Write DNT response XML
    PU->>FTP: Read response

Directory model

For daily/sub-daily native files, MDS uses a product/dataset/TAG/year/month structure.

Conceptually:

/<product>/<dataset>_<TAG>/yyyy/mm/

DNT files are uploaded separately under:

/<product>/DNT/

Responses are retrieved from the product DNT response area.

DNT rules

The DNT is sent after the intended data uploads complete.

Operational requirements include:

  • include MD5 for every delivered file;
  • keep the DNT aligned with the actual filenames and dataset/TAG;
  • limit the number of files per DNT; fewer than 100 is required and around 30 is considered a practical best practice;
  • use explicit Delete/Move keywords for lifecycle actions when required;
  • do not declare a file that failed local validation or transfer.

Overwrite and deletion semantics

Re-sending the same filename overwrites/replaces that native object; a separate Delete is not required first.

Forecast products have a shorter lifecycle and obsolete forecast files are removed as newer same-cycle bulletins arrive.

Analysis files follow the native rolling-retention policy, approximately two years.

Success response

The response XML is the authoritative application-level result for the delivery transaction.

Monitoring should capture at least:

  • response received/not received;
  • validation status;
  • ingestion status;
  • error message;
  • affected dataset/TAG;
  • DNT identifier/timestamp.

Security

Credentials are specific to the pushing entity and environment and must be kept out of Git.

Legacy scripts currently contain embedded secrets. Refactoring should move those secrets to protected configuration files or a proper secret-management mechanism.