Reproducible Preprocessing and Numerical Methods

by Giux GX-TXT, python-en, scripts-en 2 min read

Scientific Python in GX-TXT · Module 10 · Lesson 57

Before the fit we prepare a view of the data necessary for analysis and check that the values and uncertainties are usable. We keep the synthetic source separated from the derived output.

Objectives

  • Project variables into a new Dataset.
  • Check finiteness and positivity of uncertainties.
  • Interpolate some points without treating them as measurements.
  • Document the choices without deleting lines to improve the fit.

Execution

After lesson 56, import preprocessing.py to the corso_python Session, save it and execute it. The transformation creates corso_finale_a_selezionato: if the ID exists, choose a free ID and update it even in subsequent readings.

Preparation script in the GX-TXT editor.
Preparation script in the GX-TXT editor.
import numpy as np
from gxtxt import gxtxt_api_call, gxtxt_api_require

for api in ("DATASET_TRANSFORM_API_v1@1", "CANONICAL_DATASET_API_v2@2", "NUMERIC_API_v1@1"):
    gxtxt_api_require(api)
source = "corso_finale_a"
output = "corso_finale_a_selezionato"
manifest = {"api_version": 1, "operation": "project_variables",
    "sources": [{"dataset_id": source, "variable_ids": ["tempo", "segnale", "sigma"]}],
    "output": {"dataset_id": output, "contract": "auto", "storage": "auto"},
    "options": {}, "schema_policy": {}, "metadata_policy": "inherit_single_source"}
result = gxtxt_api_call("DATASET_TRANSFORM_API_v1", "execute", [manifest])
r = gxtxt_api_call("CANONICAL_DATASET_API_v2", "read_range", [output, ["tempo", "segnale", "sigma"], 0, 21])
t = np.asarray(r["tempo"]["values"]).reshape(-1)
y = np.asarray(r["segnale"]["values"]).reshape(-1)
sigma = np.asarray(r["sigma"]["values"]).reshape(-1)
valid_values = np.isfinite(t) & np.isfinite(y) & np.isfinite(sigma) & (sigma > 0)
query = np.array([0.125, 1.125, 2.125])
estimates = gxtxt_api_call("NUMERIC_API_v1", "interpolate", [t, y, query, {"method": "linear", "extrapolation": "error"}])
print("Source:", source, "output:", result["dataset_id"])
print("Issues valid:", int(valid_values.sum()), "su", len(t))
print("Time interpolated (s):", query)
print("estimates, non-measures (mV):", np.asarray(estimates["yq"]).reshape(-1))
print("No line excluded from the fit")

project_variables publishes a derivative copy with time_values, signal and sigma, keeping the source separate with the ideal curve. The mask controls that time, signal and uncertainty are finite and that sigma is positive. In the result shown all 21 rows pass the checks; the mask does not delete rows.

Output showing projection, validation checks and interpolated estimates.
Output showing projection, validation checks and interpolated estimates.

The call interpolate calculates three linear estimates between the original times. They are derived values, not new observations; they are not added to the points of the fit of the next lesson. The extrapolation: error option rejects requests outside the domain. If a check fails, it would be necessary to document and motivate the management of the row, not to remove it only because it worsens the result.

Exercise

  1. What Dataset remains the original source?
  2. Are interpolated estimates new measurements?
  3. Why check that sigma is positive?

Worked solution

  1. corso_finale_a; output has a distinct identity.
  2. No, they’re estimates according to the linear method.
  3. A standard uncertainty used for inverse weights must be positive and finite.

Download the Module 10 scripts (ZIP)

#GX-TXT #Python #Reproducibility

Other steps of this project

Leave a Reply

Your email address will not be published. Required fields are marked *