Iterators, Generators and Scientific Streaming

by Giux GX-TXT, python-en, scripts-en 2 min read

Scientific Python in GX-TXT · Module 9 · Lesson 51

A stream reads data progressively, without requiring to load a whole table at once. A Python generator can deliver a block at a time and keep the reading code separated from processing.

Objectives

  • Open a canonical stream and read limited blocks.
  • Use yield to produce values progressively.
  • Close the resource with try/finally.
  • Distinguish exhaustion of the generator and early interruption.

Execution

At the corso_python Session, import stream_generators.py, save it and run it in the Runner. The example uses four synthetic rows and blocks of two, so the operation is also visible on a small Dataset.

Generator and stream in the Script Workspace.
Generator and stream in the Script Workspace.
import numpy as np
from gxtxt import gxtxt_api_call, gxtxt_api_require

gxtxt_api_require("TABLE_STREAM_API_v3@3")
api = "TABLE_STREAM_API_v3"
def chunks(dataset_id):
    stream = gxtxt_api_call(api, "open_dataset", [dataset_id, ["tempo", "segnale"], 2])
    try:
        while True:
            chunk = gxtxt_api_call(api, "next", [stream])
            values = np.asarray(chunk["values"]["segnale"]).reshape(-1)
            if values.size:
                yield values
            if int(chunk["eof"]):
                break
    finally:
        gxtxt_api_call(api, "close", [stream])
        print("Stream closed")
item_count = 0
total = 0.0
for chunk in chunks("corso_python_serie_sintetica"):
    print("Synthetic block (mV):", chunk)
    item_count += len(chunk)
    total += float(chunk.sum())
print("Righe:", item_count, "media (mV):", total / item_count)

open_dataset returns a temporary reference to the stream. Each call to next reads the next block; yield delivers the carrier to the external cycle and suspends the function until the next request. The eof signal indicates the end. The finally lock closes the stream when the generator is exhausted or closed.

Two chunks, confirmation of stream closure and the synthetic mean.
Two chunks, confirmation of stream closure and the synthetic mean.

The output shows two blocks, four rows and an average of 1,4 mV. The processing retains a block at a time; the advantage becomes relevant on larger Dataset. This example does not measure memory consumption and does not claim to demonstrate a specific threshold.

If you stop the cycle before the end, please explicitly close the generator, for example with contextlib.closing(chunks(dataset_id)), to run the finally without depending on Python waste collection. Do not reuse a stream after closing and do not interpret its handle as an arbitrary path of the filesystem.

Exercise

  1. How many blocks are obtained here and how many values do each contain?
  2. What is the role of yield?
  3. Why is it necessary to explicitly close an interrupted generator before the end?

Worked solution

  1. Two blocks of two values.
  2. Deliver a value to the caller and suspend the function, keeping the status until the next request.
  3. To ensure promptly the execution of the finally and the release of the stream.

In the next lesson we will use exceptions and logging to make the failures more clear.

Download the Module 9 scripts (ZIP)

#Debugging #GX-TXT #Python

Other steps of this project

Leave a Reply

Your email address will not be published. Required fields are marked *