CHESSProcessing API Reference

abstract type CorrectionMethod end

Common supertype for correct's pluggable correction methods (see CHESSProcessingGPExt's GPCorrection), mirroring CHESSExperiments.QCMethod's extensibility pattern. Every concrete CorrectionMethod implements apply_correction.

struct GPCorrection <: CorrectionMethod

Two-stage spatial correction (see CHESSProcessingGPExt for the actual fit, ported from CHESSQC.gp_correction): first fits a Gaussian process to the absolute deviation of negative-relation control wells from their median, corrects every present well by it, then fits a second GP to the relative deviation of positive-relation control wells (on the already negative-corrected data) and corrects by that. negative/positive name which keys of correct's role_masks each stage uses – default :negative/:positive, matching normalize's default relations.

Defined here in the base package (its fields need nothing from GaussianProcesses) so it's nameable without GaussianProcesses loaded; only apply_correction's actual numerical fit, in CHESSProcessingGPExt, needs the weak dependency.

lengthscale/lb_lengthscale/ub_lengthscale are the isotropic Matern(5/2) kernel's lengthscale and its optimization bounds (same defaults as CHESSQC).

struct ProcessingRecord

The Experiment-level analog of CHESSCore.Read: an append-only entry describing one call to a CHESSProcessing operation (aggregate, normalize, correct, flag, merge). Never overwritten or removed – an Experiment's full processing history is legible by reading its accumulated records, the same way a Well's history is legible by reading its accumulated Reads. Recorded purely for provenance/audit: nothing in CHESSProcessing reads a record back out to decide what to compute next (the sole, deliberate exception is merge, which reads flag records to add trust columns – see flag's docstring).

Fields:

• operation: which operation produced this record (:aggregate, :normalize, :correct, :flag).
• params: the keyword arguments the call ran with (e.g. relation, fn, on_missing), so the record is self-describing without needing the original call site.
• timestamp: when the operation ran.
• values: the operation's output values Dict, stored on the record itself so an Experiment is replayable/inspectable without the caller having kept the value set around.
• upstream: indices into processing_log(experiment) (as it stood immediately before this record was appended) identifying which prior records fed this call's input, if any – provenance only, e.g. to reconstruct "this normalize call used aggregate-record #2" after the fact.

merge(experiment::Experiment, run_map::RunMap, plate_maps::AbstractVector{<:Pair},
      labeled_values::Pair{Symbol,<:AbstractDict}...) -> DataFrame

The terminal consolidator: joins experiment.design against whichever values result(s) a caller names, one output column per Pair (column_name => values), plus a trust column per :flag record found on experiment's append-only log. run_map/plate_maps are accepted but unused – part of the uniform six-operation signature; the join itself needs neither, since a values/judgment Dict is already keyed by the same node identity schedule_layout fixed as 1:nrow(experiment.design).

Every other operation only ever writes to experiment's append-only log; merge is the one deliberate, narrow exception that reads it back – specifically for :flag records, since flag is a null operation on values and never returns its judgment as something a caller could hand to merge as a values Pair the way every other operation's output can be. Flag columns are named Symbol("flag_$(i)_$(relation)") (1-based index over matching log records, in log order, plus the relation that call judged) so multiple flag calls – even against the same relation with different rules – get distinct columns.

A design row's node (1:nrow(design)) missing from a given values/flag Dict becomes missing in that column, not an error – merge doesn't require every input to cover every row.

_plate_role_data(pm::PlateMap, run_map::RunMap, values::AbstractDict{N,V},
                  relation_types::AbstractVector{Symbol}, allowed::Union{Nothing,Set{N}})
    -> (data::Matrix{Float64}, present::BitMatrix, role_masks::Dict{Symbol,BitMatrix},
        node_at::Dict{Tuple{Int,Int},N}) where {N,V}

Shared per-plate extraction: resolves one plate's occupants against values and run_map's relation-membership into the (data, present, role_masks) shape apply_correction consumes, plus a node_at lookup for scattering results back to node keys. Factored out of correct so plotting functions (see CHESSProcessingPlottingExt) can build the exact same per-plate view without duplicating this extraction.

aggregate(experiment::Experiment, run_map::RunMap, plate_maps::AbstractVector{<:Pair},
          values::AbstractDict{N,V}; relation=:duplicate, fn=Statistics.mean, on_missing=:error,
          nodes=nothing) -> (Experiment, Dict{N,Union{Missing,V}}) where {N,V}

Collapse a RunMap relation: for each anchor node (default every node in run_map; see nodes), reduces its relation-linked group's values down to one number with fn. RunMap has no intrinsic "run" node type – an anchor is simply any node with outgoing relation edges, so a control with its own :duplicate-linked replicate controls collapses exactly the same way a design row's duplicates do.

An anchor with no outgoing relation edges passes its own values entry through unchanged (a lone node's "aggregate" is just itself); a missing anchor entry in that case stays missing. Group reduction follows combine_group's missing-data policy (on_missing).

nodes scopes which anchors are processed, but note that the output Dict only contains entries for those anchors – a downstream call reading this output (e.g. normalize looking up a control node's value) won't find any node aggregate didn't process, even one that would have just passed through unchanged. The default (nodes=nothing, every node in run_map) avoids this by construction; scope nodes only when a caller genuinely doesn't need the rest of the graph's values to survive into the next call.

plate_maps is accepted but unused – part of CHESSProcessing's uniform (experiment, run_map, plate_maps, values; ...) signature shared by all six operations, so any operation can sit in any slot of a chain without the caller checking which context arguments a particular operation actually needs.

Appends a ProcessingRecord (:aggregate, params relation/fn/on_missing, this call's output values) to experiment and returns (experiment, values) – the returned values is the live thing a caller passes as the next operation's values argument; the record is provenance only.

append_record(experiment::Experiment, operation::Symbol, params::AbstractDict, values::AbstractDict;
              upstream::AbstractVector{Int}=Int[]) -> Experiment

Append a new ProcessingRecord to experiment's :processing_log and return the updated Experiment (copy-on-write, like every other CHESSExperiments parameter write). Every CHESSProcessing operation calls this internally as its side effect; callers of the six operations never need to call it directly.

apply_correction(method::CorrectionMethod, data::Matrix{Float64}, present::BitMatrix,
                  role_masks::Dict{Symbol,BitMatrix}) -> Matrix{Float64}

Extension point: given one plate's data (only meaningful where present[i,j] is true – other entries are placeholders, not real values), and one BitMatrix per relation in role_masks marking that relation's control positions (e.g. role_masks[:negative]), return a same-shape matrix of corrected values. correct only reads the returned matrix at positions where present was true. Every concrete CorrectionMethod implements this – no fallback.

combine_group(group_values::AbstractVector, fn::Function; on_missing::Symbol) -> Union{Missing,Any}

Shared missing-data policy for reducing a group of linked nodes' values (aggregate's :duplicate group, normalize's per-relation control group) down to one number with fn. Every CHESSProcessing operation that reduces a group uses this, so the policy vocabulary is identical everywhere:

• A fully missing group (nothing present) always returns missing, regardless of on_missing – this isn't unexpected data loss, it's "there was never anything here" (e.g. a run with no duplicates scheduled), and forcing an error there would punish the normal case.
• A partially missing group (some but not all present) is governed by on_missing:
  • :error (the default) throws – partial data loss is exactly the surprise a caller should be forced to notice, not silently average over.
  • :skip reduces over whatever's present.
  • :propagate returns missing the moment any input is missing, regardless of how much of the group survives.
• A fully present group is always reduced with fn, independent of on_missing.

correct(experiment::Experiment, run_map::RunMap, plate_maps::AbstractVector{<:Pair},
        values::AbstractDict{N,V}; method::CorrectionMethod, relations=[:positive, :negative],
        nodes=nothing) -> (Experiment, Dict{N,Union{Missing,V}}) where {N,V}

Positional/spatial correction: unlike normalize (which rescales a run's value against its specific linked controls via run_map edges), correct corrects for plate-position effects using plate_maps for well position and run_map only to know which wells, among all wells on that plate, count as controls for the correction fit – a node counts as a relation control if relation in RunMaps.roles(run_map, node) (the same "is this node the target of at least one relation edge from anything" check normalize's control groups use, but here applied per-plate rather than per-run).

Processed per plate (unlike CHESSQC's single-plate-only gp_correction) – spatial effects are plate-specific, so each plate_maps entry is corrected independently; a duplicate/control cluster spanning multiple plates (already guaranteed not to happen by PlateMaps/RunMaps scheduling) isn't a concern here regardless. A well with a missing (or unresolved) values entry is excluded from both the correction fit and the output for that plate – there's no missing-data policy to arbitrate here (unlike aggregate/normalize's on_missing); a spatial fit simply can't train on or correct a point it has no value for.

nodes, if given, restricts which occupant nodes are eligible to participate at all (fit or correction) – an occupant not in nodes is treated as if its well were empty.

Appends a ProcessingRecord (:correct, params method/relations, this call's output values) to experiment and returns (experiment, values).

cv_threshold(max_cv::Real) -> Function

A ready-made flag rule: returns true (trustworthy) when the coefficient of variation (std/mean) of a group's present values is <= max_cv, false otherwise. Returns missing when fewer than two values are present (a CV needs at least two points) or when the group mean is zero (CV undefined). Missing entries in the group are dropped before computing CV, mirroring combine_group's :skip behavior, but without erroring on partial missingness – flagging a low-replicate group as untrustworthy is rule's job, not a missing-data policy violation.

flag(experiment::Experiment, run_map::RunMap, plate_maps::AbstractVector{<:Pair},
     values::AbstractDict{N,V}; relation, rule, nodes=nothing) -> (Experiment, Dict{N,Union{Missing,V}}) where {N,V}

A null operation on values: judges a relation-linked group without transforming any data, unlike every other value-producing operation. flag's own return value is values, completely unchanged, so it can be inserted anywhere in a chain (before, after, or interleaved with aggregate/normalize/correct) without perturbing the data at all.

For each anchor node (default every node in run_map; see nodes), gathers its relation-linked group's raw values (no missing-data policy applied here – rule sees exactly what's present, including any missing entries, since a rule like a CV threshold needs to know when data is literally absent to judge it) and calls rule(group_values) -> judgment. An anchor with no linked group gets missing (nothing to judge). judgment can be any type rule returns (a Bool, a category, a (passed, reason) tuple) – unlike every other operation's values, a flag judgment is never constrained to the V value type, because it's never meant to flow into values at all.

The judgment is written only into the ProcessingRecord appended to experiment (as that record's values field, despite the name – flag is the one operation where the record holds something other than what the function itself returns). This is the one deliberate exception to "the log is write-only": merge is the sanctioned reader of :flag records, since a flag's judgment otherwise has no way to reach a caller who wants it, given flag never returns it as a values Pair.

plate_maps is accepted but unused – part of the uniform six-operation signature.

See cv_threshold for a ready-made rule.

normalize(experiment::Experiment, run_map::RunMap, plate_maps::AbstractVector{<:Pair},
          values::AbstractDict{N,V}; relations=[:positive, :negative], combine=Statistics.mean,
          fn, on_missing=:error, nodes=nothing) -> (Experiment, Dict{N,Union{Missing,V}}) where {N,V}

Rescale each anchor node's own value against its linked control groups. Unlike aggregate, which collapses one relation's linked group down to a replacement for the anchor's own value, normalize keeps the anchor's own value and combines it with one or more relation-linked control groups via fn.

For each anchor node (default every node in run_map; see nodes): if its own values entry is missing, the output is missing (nothing to normalize) and fn is never called. Otherwise, for each relation in relations, that relation's linked control nodes' values are reduced to one number via combine (using combine_group's missing-data policy, on_missing; a relation with no linked nodes contributes missing). The per-relation results become groups::Dict{Symbol,V}, passed to fn(own_value, groups) -> V alongside the anchor's own value. If a control node's own replicate structure needs collapsing first (e.g. a control with :duplicate-linked replicate controls), that's aggregate's job, run on the control nodes beforehand – combine here only reduces across distinct linked control nodes for one relation, a different reduction level from aggregate's, and (unlike aggregate's output) never recorded on its own.

plate_maps is accepted but unused – part of the uniform six-operation signature.

Appends a ProcessingRecord (:normalize, params relations/combine/fn/on_missing, this call's output values) to experiment and returns (experiment, values).

See subtract_and_normalize for the repo's one existing normalization method, expressed against this signature.

plot_control_data(run_map::RunMap, plate_maps::AbstractVector{<:Pair}, values::AbstractDict;
                   relations=[:positive, :negative], nodes=nothing) -> Dict{String,Plots.Plot}

Per-plate categorical scatter of values by role (relations... plus :run for wells present but not in any relation's mask) – the restored, generalized successor to CHESSQC.plot_control_data (hardcoded to exactly run/positive control/negative control), now supporting an arbitrary relation set and colored via LabwarePlotting.role_palette for consistency with every other role-colored plot in the CHESS ecosystem.

Extension point: see plot_plate_data.

plot_plate_data(run_map::RunMap, plate_maps::AbstractVector{<:Pair}, values::AbstractDict;
                 relations=[:positive, :negative], nodes=nothing) -> Dict{String,Plots.Plot}

Per-plate heatmap of values over each plate's wells – the restored, generalized successor to CHESSQC.plot_plate_data (a bare heatmap), now drawn inside the shared grid/gridline/lettered-row skeleton. One plot per plate with at least one resolvable well, keyed by the plate_maps label (stringified).

Extension point: the actual implementation lives in CHESSProcessingPlottingExt, loaded automatically once LabwarePlotting is also using'd. CHESSProcessing's base package has no Plots/LabwarePlotting dependency.

processing_log(experiment::Experiment) -> Vector{ProcessingRecord}

Convenience accessor for get_parameter(experiment, :processing_log).

resolve(experiment::Experiment, run_map::RunMap, plate_maps::AbstractVector{<:Pair},
        reads::AbstractVector{<:Pair}) -> (Experiment, Dict{N,Union{Missing,Float64}}) where {N}

The one operation with no values input – it's where a value first enters the model. Reads are inherently well-scoped, so resolve resolves against plate_maps (position), not run_map (run_map is accepted but unused, part of the uniform six-operation signature).

reads is a labeled Pair vector, [labware_name => LabwareRead, ...], mirroring plate_maps' own labware_name => PlateMap shape – the caller pairs each CHESSParsers.LabwareRead with the plate it belongs to, exactly the way CHESSParsers.record_reads! already requires the caller to pair one LabwareRead with one CHESSCore.Labware per call, rather than resolve guessing a pairing from LabwareRead.metadata["plate"]. Pass multiple pairs sharing the same labware_name for multiple channels/reads on one plate (mirroring record_reads!'s Vector{LabwareRead} overload, which just loops).

For each (labware_name, lr) pair: matches plate_maps by labware_name, builds that plate's well-to-node mapping from PlateMaps.DataFrame(pm) (well-label generation lives in PlateMaps itself – CHESSCore-free), and assigns lr.data.value to the matching node for each well. lr.data must have exactly one row per well (an endpoint read); a kinetic LabwareRead with multiple timepoints per well needs a separate future reduction step first, per CHESSProcessing's scalars-only model. A well with no occupant node on its plate (never scheduled) is silently skipped – nothing to attach that reading to; a well claimed by more than one reads pair throws (a data integrity conflict, not a missing-data policy question).

Output is keyed over every node each matched PlateMap knows about, seeded missing before reads are applied – run nodes and duplicate/control nodes alike, since resolve doesn't know or care about role, only position. A CHESSProcessingCoreExt extension (loaded when CHESSCore is present) adds an overload for callers who already attached reads to CHESSCore.Labware/Well objects via record_reads! instead of working from raw LabwareReads.

Appends a ProcessingRecord (:resolve, no params, this call's output values) to experiment and returns (experiment, values).

subtract_and_normalize(own, groups::AbstractDict{Symbol}) -> Any

(own - groups[:negative]) / (groups[:positive] - groups[:negative]) – the repo's one existing normalization method (previously CHESSQC.subtract_and_normalize), expressed as a normalize fn. Use with relations=[:positive, :negative] (the default).