Home › reference › formats and qc

Data formats and QC flags

Gauge data arrives in whatever shape the agency that collected it chose thirty years ago. tidalcore reads six layouts and refuses to guess about four things. The refusals matter more than the readers, so they come first.

What the readers will not guess

Time zone. If timestamps carry no offset and tz was not passed, the reader raises TimeZoneUnknownError. Assuming UTC on a local-time file shifts every phase by a constant and nothing downstream will notice.
Units. A column of numbers around 200 could be centimetres or a record in feet from a very deep harbour. The unit must be in the header or in the call.
Duplicate timestamps. Two rows with the same instant raise FormatError with the line number. Deciding which of the two is correct is not the library's business.
Order. A non-monotonic time column is an error rather than something to sort quietly, because the usual cause is two concatenated files with an overlap.

Readers

Pass an id as `format=`, or leave the default `"auto"` and let the sniffer decide from the first non-comment lines.
Format id	Layout	Detected by	Notes
csv2	timestamp, level	two comma-separated fields	flags default to 0, unchecked
csv-flagged	timestamp, level, flag	three fields, third is 0 to 9	the format used in the tutorial
tsv2	tab-separated, same as csv2	tab in the first data line	accepts a leading comment block
fixed-24	one line per day, 24 fields of five characters	line length and a numeric date prefix	hourly only; missing values are a sentinel, declared in the header
jsonl	one object per line with keys t, v, q	first byte is an opening brace	streams without loading the file
netcdf-cf	CF conventions, a time coordinate and one level variable	file magic	needs the `netcdf` extra; the time units attribute must state a zone

Sniffing reads at most the first 64 kilobytes and never the whole file, which means a file that changes shape halfway through will be read with the wrong reader and then fail on a line number. That is the intended outcome: a loud failure at line 40 311 is better than a quiet reinterpretation.

rec = tc.read_gauge("harbour-1958.txt", format="fixed-24", tz="+01:00")
print(rec.units, rec.reference, rec.step)
# m  admiralty chart datum  60min

Note what happened there: the reference string was copied verbatim from the file header and is carried through to the solution. tidalcore does not know what it means and does not convert anything to it or from it.

Quality-control flags

One uint8 per sample. The vocabulary is small on purpose, and the only values the library writes by itself are 3, 5 and 9.

Default weights. Change them with `analyse(..., weights={2: 0.5})`, and record the fact next to any published amplitude.
Flag	Meaning	In the fit by default	Who sets it
0	unchecked	yes, weight 1	readers, when a file has no flag column
1	good	yes, weight 1	the data provider
2	probably good	yes, weight 1	the data provider
3	probably bad	no	`qc.flag_spikes` and `qc.flag_flat`
4	bad	no	the data provider, or you by hand
5	filled from a solution	no	`fill_gaps`
9	missing	no, the value is NaN	`qc.flag_gaps` and the readers

Flag 0 is admitted to the fit and flag 2 is too. That is a deliberate choice for a tool whose main input is decades-old records with no quality column at all, and it is also the setting most likely to surprise you. Solution.weights_used always prints the mapping that was actually applied.

Time, units and gaps

Timestamps: ISO 8601 with an offset, parsed to numpy.datetime64 in UTC at microsecond resolution. Leap seconds are not represented, consistent with the underlying type.
Levels: Floating point in the file's own unit, positive upwards. A negative level is perfectly normal and means below the stated reference.
Gaps: NaN in level and flag 9 in flags. The two always agree; a NaN with flag 1 is rejected at read time as an internal inconsistency.
Sampling: Regular or irregular. step is the median interval and is reported, not enforced; the fit uses the timestamps as given.

Output

A solution written with to_json is a flat object, deliberately boring, and stable enough to parse with a shell one-liner:

{
  "tidalcore": "0.9.3",
  "station": "station-7",
  "units": "m",
  "reference": "station zero",
  "ref_time": "2026-04-18T12:00:00Z",
  "nodal": true,
  "coverage": 0.989,
  "n_used": 88050,
  "constituents": {
    "M2": {"amplitude": 1.412, "phase": 211.6, "error": 0.0044},
    "S2": {"amplitude": 0.463, "phase": 238.2, "error": 0.0044}
  }
}

The key names may change once before version 1.0, as noted on the overview; a converter will ship in the release that changes them.

See also: the tutorial shows the flaggers in sequence on a real-shaped record, and the API overview lists every exception a reader can raise.