Skip to content

Current Production Workflow

Daily cycles

Two NRT upload cycles run every day.

Cycle HCMR cron Expected files TDT
12 03:30 local server time 21 06:00 UTC
00 19:30 local server time 21 20:00 UTC

The expected 21 files are one analysis file plus twenty forecast files.

Input locations

  • cycle 00: /netapp/ocean2/MEDFILESV2023_Q4/<today>
  • cycle 12: /netapp/ocean2/MEDFILESV2023_Q4_FC12/<yesterday>

Processing sequence

flowchart TD
    A([Cycle starts]) --> B[Check production directory]
    B --> C{21 / 21 files available?}
    C -->|No| W[WAITING_FILES]
    W --> B
    C -->|Yes| R[FILES_READY]

    R --> U[Upload NetCDF files to MDS FTP]
    U --> X{Transfers successful?}
    X -->|Retry| U
    X -->|Yes| UD[FILES_UPLOADED]

    UD --> MD5[Calculate / include MD5]
    MD5 --> DC[Create DNT XML]
    DC --> DU[Upload DNT]
    DU --> WR[WAITING_RESPONSE]

    WR --> RESP{DNT response}
    RESP -->|Validated and Ingested| OK([INGESTED])
    RESP -->|False / timeout / error| ERR([FAILED / Recovery])

    OK --> S3[Independent S3 / CMT verification]

MDS processing semantics

The data files are uploaded to the product/dataset/TAG hierarchy first. The DNT is uploaded only after the file transfer phase has completed. The DNT tells MDS which files are part of the delivery and may also contain Delete/Move operations.

The Delivery Validator then processes the DNT, validates checksums and publishes valid native files into the Marine Data Lake.

A delivery can take up to roughly 1.5 hours from first upload to appearance in the data lake, so transfer completion and public availability are intentionally represented as separate monitoring states.

Existing script state

The current script exposes internal flags including:

  • filesFound
  • upStarted
  • upDone
  • reSuccess
  • upError
  • nepSleep

Completion semantics

upDone is not final success. It means that the upload phase completed.

Application-level success requires the DNT response to confirm successful validation/ingestion, in practice represented by reSuccess and an Ingested="True" result.

Forecast replacement and retention

The operational DNT also manages data lifecycle:

  • forecast files from the previous same-cycle bulletin are deleted when superseded;
  • analysis files are retained on a rolling basis of approximately two years;
  • re-sending the same filename is an overwrite operation rather than requiring a prior Delete.

These lifecycle operations are operationally significant and should be visible in future monitoring/event reporting.

Progress-log limitation

The existing progress log is mainly a current-state snapshot and is overwritten during execution. It is not a historical audit log.

Monitoring must also reject stale state from a previous run. For example, an old reSuccess=1 must not cause a new cycle to appear immediately completed.