Capillaries: notes
2023-02-20
First look at Capillaries UI
UI snapshots
Below are two scraped snapshots of Capillaries UI that give the idea about the state of the test environment. Click any of the screenshots below and navigate through status screens - all blue elements are clickable, buttons are disabled (button tooltips are working though).
Second snapshot also contains status information about other integration test runs: lookup quicktest (smaller number of orders and order items), py_calc_quicktest (performs Python formula calculations) and tag_and_denormalize_quicktest (tags data records using regular expressions and calculates tag totals).
For both snapshots, the navigation starts at the very top "Keyspaces" level.
lookup_bigtest run in progress:
lookup_bigtest (and all other test runs) complete:
What does lookup_bigtest do, anyways?
lookup_bigtest is an integration test for Capillaries Deploy tool, it creates test data (~50K orders with ~100K order items) deploys it to an Openstack public cloud and runs Capillaries script that:
- reads orders from 10 files and order items from 100 files in parallel
- performs four types of joins in parallel
- saves sorted results as CSV files (one file - one worker)
UPDATE 2024:
- lookup/bigtest_cloud is now maintained in the Capillaries repository
- Capillaries Deploy tool is now maintained at https://github.com/capillariesio/capillaries-deploy
- Capillaries Deploy tool now does not support Openstack, and is oriented towards AWS/Azure/GCP; reason - it's nearly impossible to find a public Openstack provider that could run Capillaries at scale
This diagram shows the workflow executed by lookup_bigtest:
The diagram was generated by this Capillaries command that takes Capillaries script/paramaters and generates a graph in DOT graph description language.
cd pkg/exe/toolbelt
go run capitoolbelt.go validate_script -script_file=../../../test/data/cfg/lookup_bigtest/script.json -params_file=../../../test/data/cfg/lookup_bigtest/script_params_one_run.json -idx_dag=true
DOT graphs can be visualized by any online Graphviz DOT viewer and saved as vector or raster image file. Open this diagram in a separate tab/window to see it in detail (tooltips are working).