Skip to content

Render the documentation at v1.4.3 #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mikub97 merged 1 commit into from
Sep 10, 2026
Merged

Render the documentation at v1.4.3 #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
239 changes: 162 additions & 77 deletions docs/analyses-plots.js
View file
Open in desktop

Large diffs are not rendered by default.

28 changes: 14 additions & 14 deletions docs/analyses.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
<main class="docs-main">

<div class="crumbs">Analyses</div>
<h1>The analyses DIMS ships</h1>
<h1 id="the-analyses-dims-ships">The analyses DIMS ships</h1>
<p>Three analyses come with the core. Each one reads prepared time series from
<code>assets/timeseries/</code>, and each one writes a single JSON payload that a dashboard
tab draws and a notebook can read back.</p>
Expand All @@ -44,7 +44,7 @@ <h1>The analyses DIMS ships</h1>
</thead>
<tbody>
<tr>
<td><a href="crosswavelet.html">Cross-wavelet &amp; coherence</a></td>
<td><a href="crosswavelet.html">Cross-wavelet analysis</a></td>
<td>a <strong>pair</strong></td>
<td>at which timescales, and when, do two signals share energy — and do they keep a consistent timing relationship?</td>
<td><code>assets/crosswavelet/{video}_crosswavelet_data.json</code></td>
Expand Down Expand Up @@ -74,11 +74,11 @@ <h1>The analyses DIMS ships</h1>
<a href="assets.html">asset layout</a> says where every file goes and what a
<code>visualization</code> block holds; <a href="analysis-output.html">analysis output</a>
says what any analysis must record and why.</p>
<h2>The running example</h2>
<h2 id="the-running-example">The running example</h2>
<p>Every figure in this section is real output. A generator builds six synthetic
signals whose answers are known before anything runs, feeds them to
<code>dims-analysis run</code>, and the pages plot the payloads that come out —
<a href="https://github.com/dims-network/dims/blob/v1.4.2/docs/analyses/demo/make_demo_data.py"><code>docs/analyses/demo/make_demo_data.py</code></a>, and the
<a href="https://github.com/dims-network/dims/blob/v1.4.3/docs/analyses/demo/make_demo_data.py"><code>docs/analyses/demo/make_demo_data.py</code></a>, and the
committed results beside it. Nothing here is an illustration of what the code
is supposed to do.</p>
<p>Two of those six are the running example, carried through all three pages:</p>
Expand All @@ -93,7 +93,7 @@ <h2>The running example</h2>
the kind of thing the chance level is estimated against.</p>
<p>That pair is the reason both cross-wavelet <em>power</em> and <em>coherence</em> exist: power
finds the burst, coherence does not.</p>
<h2>What goes in</h2>
<h2 id="what-goes-in">What goes in</h2>
<p>One CSV per recording per signal, at <code>assets/timeseries/{video}_{type}.csv</code>,
with a <code>Time</code> column in seconds and one value column:</p>
<pre><code class="language-csv">Time,sig_a
Expand All @@ -107,7 +107,7 @@ <h2>What goes in</h2>
series is sorted by time. What differs between them is the guard afterwards —
cross-wavelet refuses fewer than 50 samples, the recurrence analyses refuse
fewer than 10 and refuse a signal that does not vary at all.</p>
<h2>Turning them on</h2>
<h2 id="turning-them-on">Turning them on</h2>
<p>Each analysis is gated by one key in the study's <code>config.json</code>, and the shape of
that key is not the same for all three:</p>
<pre><code class="language-jsonc">{
Expand Down Expand Up @@ -140,14 +140,14 @@ <h2>Turning them on</h2>
<p>Tuning goes under <code>analysis.&lt;step id&gt;</code>, where the step id is <code>rqa</code>, <code>crqa</code> or
<code>crosswavelet</code> — the names <code>dims-analysis list</code> prints, not the gate keys. Each
page below documents its own parameters; the schema that validates them is
<a href="https://github.com/dims-network/dims/blob/v1.4.2/docs/contracts/config.schema.json"><code>contracts/config.schema.json</code></a>.</p>
<h2>What comes out</h2>
<a href="https://github.com/dims-network/dims/blob/v1.4.3/docs/contracts/config.schema.json"><code>contracts/config.schema.json</code></a>.</p>
<h2 id="what-comes-out">What comes out</h2>
<p>One file per recording per analysis, always this shape:</p>
<pre><code class="language-jsonc">{
&quot;video_id&quot;: &quot;demo&quot;,
&quot;payload_version&quot;: 2,
&quot;&lt;container key&gt;&quot;: { /* keyed by data type, or by &quot;{a}_vs_{b}&quot; */ },
&quot;provenance&quot;: { &quot;core_version&quot;: &quot;1.4.2&quot;, &quot;...&quot;: &quot;the settings actually used&quot; },
&quot;provenance&quot;: { &quot;core_version&quot;: &quot;1.4.3&quot;, &quot;...&quot;: &quot;the settings actually used&quot; },
&quot;precision&quot;: { &quot;significant_figures&quot;: 6, &quot;note&quot;: &quot;...&quot; }
}
</code></pre>
Expand Down Expand Up @@ -179,7 +179,7 @@ <h2>What comes out</h2>
</table>
<p>Writing merges one level deep rather than replacing the file, so a study-owned
analysis can add its own entries to <code>rqa_data</code> without erasing the core's.</p>
<h3>Numbers are rounded, arrays are packed</h3>
<h3 id="numbers-are-rounded-arrays-are-packed">Numbers are rounded, arrays are packed</h3>
<p>Every small field is rounded to <strong>six significant figures</strong> — significant
figures, not decimal places — and non-finite values become <code>null</code>, because
<code>json.dump</code> writes a bare <code>NaN</code> token that <code>JSON.parse</code> rejects outright.</p>
Expand Down Expand Up @@ -230,7 +230,7 @@ <h3>Numbers are rounded, arrays are packed</h3>
<p>In Python, <code>dims_analysis.common.arrays.unpack()</code> walks a payload and returns
whichever it finds, so you do not have to know in advance which fields grew
large enough to be packed.</p>
<h3>A picture and the analysis are not the same file</h3>
<h3 id="a-picture-and-the-analysis-are-not-the-same-file">A picture and the analysis are not the same file</h3>
<p>Every payload holds a <code>visualization</code> block that has been <strong>reduced</strong> to
something a browser can draw, and records how:</p>
<pre><code class="language-jsonc">&quot;reduction&quot;: {
Expand All @@ -255,7 +255,7 @@ <h3>A picture and the analysis are not the same file</h3>
threshold, which each page shows.</p>
<p>The rule and its rationale are contract A3 in
<a href="analysis-output.html">analysis output</a>.</p>
<h2>Where these figures came from</h2>
<h2 id="where-these-figures-came-from">Where these figures came from</h2>
<p>The demo payloads were produced by running the real steps over the synthetic
signals, against the pinned dependencies. Re-generate them with:</p>
<pre><code class="language-sh">python docs/analyses/demo/make_demo_data.py
Expand All @@ -264,8 +264,8 @@ <h2>Where these figures came from</h2>
coherence chance level — so two runs produce identical files, and a change in a
figure means a change in the analysis.</p>
<p class="source-note">Generated from
<a href="https://github.com/dims-network/dims/blob/v1.4.2/docs/analyses/index.md"><code>docs/analyses/index.md</code></a>
at <code>v1.4.2</code>. Edit it there, not here.</p>
<a href="https://github.com/dims-network/dims/blob/v1.4.3/docs/analyses/index.md"><code>docs/analyses/index.md</code></a>
at <code>v1.4.3</code>. Edit it there, not here.</p>

<footer>
DIMS-network · <a href="https://github.com/dims-network">github.com/dims-network</a>
Expand Down
46 changes: 23 additions & 23 deletions docs/analysis-common.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
<main class="docs-main">

<div class="crumbs">Extending DIMS</div>
<h1><code>dims_analysis.common</code> — the shared machinery</h1>
<h1 id="dims-analysiscommon-the-shared-machinery"><code>dims_analysis.common</code> — the shared machinery</h1>
<p>Fourteen modules that the analysis steps share. Nothing here is an analysis; it is
the plumbing every step needs, factored out so that a rule exists <strong>once</strong>.</p>
<p>This is contributor reference. If you are writing a step, read
Expand All @@ -40,16 +40,16 @@ <h1><code>dims_analysis.common</code> — the shared machinery</h1>
name from outside the package.</p>
</blockquote>
<hr />
<h2>Payload encoding and precision</h2>
<h3><code>arrays</code> — the two wire encodings</h3>
<h2 id="payload-encoding-and-precision">Payload encoding and precision</h2>
<h3 id="arrays-the-two-wire-encodings"><code>arrays</code> — the two wire encodings</h3>
<p><code>PAYLOAD_VERSION = 2</code>, <code>BITMAP = &quot;bitmap-b64&quot;</code>, <code>FLOAT32 = &quot;f32-b64&quot;</code>.</p>
<p><code>pack_bitmap(matrix)</code> / <code>unpack_bitmap(obj)</code> — one bit per cell, rows byte-aligned.
<code>pack_f32</code> / <code>unpack_f32</code> — little-endian float32, NaN preserved.
<code>is_packed(obj)</code>, <code>unpack(obj)</code> — dispatch without knowing which encoding you have.
<code>nan_to_none(values)</code> — for the plain-list fields, since JSON has no NaN.</p>
<p>The browser's half of this contract is
<a href="dims-api.html"><code>DIMS.decodeArray</code></a>.</p>
<h3><code>payload</code> — significant figures, and saying what produced a file</h3>
<h3 id="payload-significant-figures-and-saying-what-produced-a-file"><code>payload</code> — significant figures, and saying what produced a file</h3>
<p><code>PAYLOAD_SIGNIFICANT_FIGURES = 6</code>.</p>
<p><code>round_significant(x)</code> and <code>round_payload(obj)</code> round to <strong>significant figures, not
decimal places</strong> — a coherence of <code>0.0000123</code> and a power of <code>12345.6</code> cannot share
Expand All @@ -60,43 +60,43 @@ <h3><code>payload</code> — significant figures, and saying what produced a fil
real cases drove this: a study computed partly at 100 surrogates and partly at 300
was silently inconsistent, and a recurrence analysis that <em>reached</em> 33.7 % against
a 7 % target looked identical to one that was <em>asked for</em> 33.7 %.</p>
<h3><code>results</code> — merging rather than clobbering</h3>
<h3 id="results-merging-rather-than-clobbering"><code>results</code> — merging rather than clobbering</h3>
<p><code>VERSION_KEY</code>, <code>UnversionedPayload</code>, <code>check_version</code>, <code>same_version</code>,
<code>merge_payload</code>, <code>read_existing</code>, <code>compare_entries</code>, <code>write_payload(path, payload, compact=True)</code>.</p>
<p>Re-running one pair must not delete the other pairs already in the file, and a
payload from a different format version must not be merged into one that is not.</p>
<hr />
<h2>Reading input</h2>
<h3><code>series</code> — loading a time series, with the failures named</h3>
<h2 id="reading-input">Reading input</h2>
<h3 id="series-loading-a-time-series-with-the-failures-named"><code>series</code> — loading a time series, with the failures named</h3>
<p><code>TIME = &quot;Time&quot;</code>, <code>SeriesError</code>.</p>
<p><code>time_column(df)</code> finds the time column case-insensitively.
<code>load(path, min_points=0, value_column=None, min_variance=None)</code> raises
<code>SeriesError</code> with a sentence saying what is wrong.
<code>load_or_none(...)</code> is the same for the steps that want to warn and continue.
<code>normalise(values)</code> is the z-normalisation the recurrence analyses run on.</p>
<h3><code>assets</code> — where a private study's data actually is</h3>
<h3 id="assets-where-a-private-studys-data-actually-is"><code>assets</code> — where a private study's data actually is</h3>
<p><code>MARKER = 'data.local.json'</code>.</p>
<p><code>assets_root(project_dir)</code>, <code>resolve(path, project_dir, root)</code>, <code>describe(...)</code>.
A private study keeps recordings outside the repository; this is the one place
that knows how to follow the pointer. See
<a href="data-visibility.html"><code>contracts/data-visibility.md</code></a>.</p>
<h3><code>config</code> — reading the <code>include_*</code> keys</h3>
<h3 id="config-reading-the-include-keys"><code>config</code> — reading the <code>include_*</code> keys</h3>
<p><code>ConfigError</code>, <code>gate_value</code>, <code>enabled(config, key)</code>,
<code>as_list(config, key, what)</code>, <code>tuning(config, step_id)</code>,
<code>tuned_number(config, step_id, key, default)</code>.</p>
<p>Three keys use three different conventions — a list of measures, a list of pairs,
a boolean — so the reading of them lives here rather than being re-guessed per
step. <code>as_list</code> raises with an example of the right shape rather than a type error.</p>
<h3><code>limits</code> — refusing what will not fit</h3>
<h3 id="limits-refusing-what-will-not-fit"><code>limits</code> — refusing what will not fit</h3>
<p><code>MIN_POINTS = 10</code>, <code>MIN_VARIANCE = 0.0</code>, <code>MAX_MATRIX_BYTES = 2 GiB</code>,
<code>InputTooLarge</code>.</p>
<p><code>matrix_bytes(n)</code>, <code>max_points(budget)</code>, <code>check_length(n, what, budget)</code>.</p>
<p>A recurrence matrix is quadratic and <code>cdist</code> returns float64, so 8 bytes a cell —
which puts the ceiling near <strong>16,000 points</strong>. Being told that before the
allocation is better than an <code>OOM</code> twenty minutes in.</p>
<hr />
<h2>Recurrence</h2>
<h3><code>recurrence</code> — one definition of every recurrence quantity</h3>
<h2 id="recurrence">Recurrence</h2>
<h3 id="recurrence-one-definition-of-every-recurrence-quantity"><code>recurrence</code> — one definition of every recurrence quantity</h3>
<p><code>RATE_TOLERANCE = 0.01</code>.</p>
<table>
<thead>
Expand Down Expand Up @@ -134,7 +134,7 @@ <h3><code>recurrence</code> — one definition of every recurrence quantity</h3>
series, so there is no line of identity and nothing to exclude. Getting that wrong
shifts every number. One flag, one implementation, both steps.</p>
<p><code>L_MAX</code> is returned in <strong>seconds</strong> — the longest run multiplied by <code>dt</code>.</p>
<h3><code>reduce</code> — making a picture small without lying about it</h3>
<h3 id="reduce-making-a-picture-small-without-lying-about-it"><code>reduce</code> — making a picture small without lying about it</h3>
<p><code>factor_for(n_points, max_points)</code> — <strong>rounds up</strong>, so the result never exceeds the
cap.
<code>block_mean</code>, <code>block_mode</code>, <code>block_binary</code>, <code>rate_of(m)</code>.</p>
Expand All @@ -145,7 +145,7 @@ <h3><code>reduce</code> — making a picture small without lying about it</h3>
<p><code>rate_of</code> is a plain mean and <strong>includes</strong> the line of identity, unlike
<code>recurrence.recurrence_rate</code> — the two numbers in a payload are close but not
identical, and they are not measuring quite the same thing.</p>
<h3><code>window</code> — sliding windows that fit the recording</h3>
<h3 id="window-sliding-windows-that-fit-the-recording"><code>window</code> — sliding windows that fit the recording</h3>
<p><code>MIN_WINDOWS = 20</code>, <code>MIN_WINDOW_POINTS = 2</code>, <code>WindowPlan</code>, <code>plan(n, dt, window_sec, step_sec, min_windows)</code>.</p>
<p>A long window over a short recording gives a handful of windows and no time
course, so <code>plan</code> shortens what it must — and the two limits are separate:</p>
Expand All @@ -164,15 +164,15 @@ <h3><code>window</code> — sliding windows that fit the recording</h3>
the substitution — and DET and LAM depend on the window length, so an adjusted
analysis is not comparable with an unadjusted one.</p>
<hr />
<h2>Wavelet</h2>
<h3><code>coherence</code> — smoothing, delegated deliberately</h3>
<h2 id="wavelet">Wavelet</h2>
<h3 id="coherence-smoothing-delegated-deliberately"><code>coherence</code> — smoothing, delegated deliberately</h3>
<p><code>smoothed_spectra(W1, W2, scales, dt, dj, mother_wavelet)</code>,
<code>coherence_from_spectra(S1, S2, S12)</code>, <code>coherence(...)</code>.</p>
<p>The smoothing operator is pycwt's, validated, rather than a local
reimplementation. That is not incidental: a hand-rolled smoother that failed to
cancel phase is exactly the defect that shipped once and was found by a notebook
rather than a crash.</p>
<h3><code>tc98</code> — Torrence &amp; Compo (1998), transcribed</h3>
<h3 id="tc98-torrence-compo-1998-transcribed"><code>tc98</code> — Torrence &amp; Compo (1998), transcribed</h3>
<p><code>DEFAULT_LEVEL = 0.95</code>, <code>Z1_95_PUBLISHED = 2.182</code>, <code>Z2_95_PUBLISHED = 3.999</code>,
<code>MAX_DIRECT_NU = 60.0</code>.</p>
<p><code>cross_wavelet_pdf</code>, <code>survival</code>, <code>survival_by_quadrature</code>, <code>cross_wavelet_z(nu, level)</code>, <code>ar1_background(alpha, period, dt)</code>, <code>local_significance</code>,
Expand All @@ -186,8 +186,8 @@ <h3><code>tc98</code> — Torrence &amp; Compo (1998), transcribed</h3>
distinction between the four significance levels — and which question each answers
— is on <a href="crosswavelet.html">the cross-wavelet page</a>.</p>
<hr />
<h2>Step plumbing</h2>
<h3><code>step_io</code> — the parts of a step that are always the same</h3>
<h2 id="step-plumbing">Step plumbing</h2>
<h3 id="step-io-the-parts-of-a-step-that-are-always-the-same"><code>step_io</code> — the parts of a step that are always the same</h3>
<p><code>parse_args(prog, description, default_output_dir, argv)</code>,
<code>tuning(config, step_id, window_sec, step_sec, target_recurrence_default)</code>,
<code>resolve_io(input_dir, output_dir, project_dir)</code>,
Expand All @@ -196,19 +196,19 @@ <h3><code>step_io</code> — the parts of a step that are always the same</h3>
<p><code>parse_args</code> defaults <code>--window</code> and <code>--step</code> to <code>None</code> rather than to a number,
<strong>deliberately</strong>: <code>None</code> means &quot;the config decides&quot;, and a default here would
silently win over the study's own <code>analysis</code> block.</p>
<h3><code>manifest</code> — what a complete <code>assets/</code> looks like</h3>
<h3 id="manifest-what-a-complete-assets-looks-like"><code>manifest</code> — what a complete <code>assets/</code> looks like</h3>
<p><code>NAME = 'MANIFEST.json'</code>, <code>scan</code>, <code>path_for</code>, <code>load</code>, <code>write(project_dir, deep=True)</code>, <code>compare(project_dir, deep=False)</code> → <code>(missing, changed, extra)</code>.</p>
<p>Driven by <a href="cli-dims-analysis.html#manifest"><code>dims-analysis manifest</code></a>. Skips
<code>MANIFEST.json</code> itself, <code>.gitkeep</code> and <code>.DS_Store</code>; hashes in 1 MiB chunks.</p>
<h2>See also</h2>
<h2 id="see-also">See also</h2>
<ul>
<li><a href="step.html"><code>contracts/step.md</code></a> — writing a step.</li>
<li><a href="analysis-output.html"><code>contracts/analysis-output.md</code></a> — the rules
these modules exist to enforce, each with the measurement behind it.</li>
</ul>
<p class="source-note">Generated from
<a href="https://github.com/dims-network/dims/blob/v1.4.2/docs/reference/analysis-common.md"><code>docs/reference/analysis-common.md</code></a>
at <code>v1.4.2</code>. Edit it there, not here.</p>
<a href="https://github.com/dims-network/dims/blob/v1.4.3/docs/reference/analysis-common.md"><code>docs/reference/analysis-common.md</code></a>
at <code>v1.4.3</code>. Edit it there, not here.</p>

<footer>
DIMS-network · <a href="https://github.com/dims-network">github.com/dims-network</a>
Expand Down
Loading
Loading