2026-07-01

Capillaries: data acquisition

What is data acquisition?

The classic definition describes data acquisition as the "process of collecting real-world physical information - such as temperature, pressure, sound, or voltage - and converting it into digital data so a computer can analyze, display, and store it."

In the context of Capillaries-style batch data processing, we can define data acquisition as the process of obtaining data from outside the Capillaries inputs.

Imagine that, halfway through processing a Capillaries script, we realize that we need to add a couple of values to a specific data row - but we have no idea how to calculate them ourselves. We need to obtain those values from an external source.

Some may argue that "external data retrieval" is a more appropriate term, since we are retrieving data from external services or datasets, not PLCs or sensors.

A photorealistic photo of an average-aged man in a dark business suit standing on his tiptoes on the wooden chair, stretching his arm up high to reach a book sitting on a very tall wooden shelf among other books. The pair the man stands on is behind a desk with a lot of open books and papers on it. High-key lighting, shallow depth of field, natural bokeh background. Shot like a candid lifestyle photograph. The picture width-hight ratio is 1:1.

Why is data acquisition needed?

Real-time data

This is probably not a very common Capillaries use case, but a customer may need a near-real-time snapshot of some external measurement - for example, the current price of a stock.

The data set is too large to prepare in advance

Capillaries simply cannot preload the entire data domain. To use the analogy from the picture above, the man cannot put every book from the shelf onto his desk - there are simply too many of them.

External calculations

The required values may:

  • be too complex to implement within the batch-processing infrastructure;
  • depend on proprietary third-party data or algorithms.

What does Capillaries have to offer?

There are two possible approaches.

1. Call an external API from Python

Pros:

  • Access to real-time data.
  • No need to wait until all rows have been processed.

Cons:

  • Chatty communication.
  • Security considerations - exposing an API may be off the table, for example

2. Use an intermediate file

Prepare all data required for acquisition in a separate data file at the end of a Capillaries run or script. Perform the acquisition using external (non-Capillaries) infrastructure in a scalable manner - for example, by processing batches in parallel - to produce a result file. Then start the next Capillaries run or script using that result file as input.

Are there any tests to look at?

There are no tests that implement either of these two approaches directly. However, the the end-of-day price provider in the Portfolio test is a good candidate for externalization.

Notice the hundreds of stock quotes hardcoded in the test. This is acceptable in a test environment, but in a real-world system those values would probably come from an external service that provides the price of a specific stock at a specific date and time.

For example, under Approach 1, every call to

eod_price_provider.get_price(d, ticker)

could invoke an external API.

Alternatively, under Approach 2, we could:

  • Create a temporary request file containing ticker symbols and timestamps.
  • Send the file to an external processor that enriches it with stock quotes.
  • Start a Capillaries run that uses the enriched file.

The file might look like this:

Ticker Ts Quote (populated by external infrastracture)
AMZN 2020-10-16-18:02:03.000+0000 3272.71
AMZN 2020-10-16-18:02:13.000+0000 3271.21

© capillaries.io