Repository navigation
Ship as an installable package, fix first-launch data, add an MCP server - #12
Conversation
…al CLI The package was a top-level `src` (plus `config`), and the `hyperdata` entry point imported run_dashboard.py from the repo root. Any install that was not an editable clone (pipx, uvx, a wheel, PyPI) crashed with ModuleNotFoundError: run_dashboard, and dropped generic `src`/`config` packages into site-packages. - move src/ and config/ into hyperdata_terminal/; drop every sys.path hack - state and logs resolve via hyperdata_terminal.paths: HYPERDATA_DATA_DIR, else an existing <checkout>/data, else the per-user data dir; never site-packages - `hyperdata` gains subcommands: one dashboard directly (heatmap, whales, cvd --symbol ETH, ...), api, paper, verify, mcp; --no-boot, --version - `hyperdata paper -s cvd_momentum -s ./my_strategy.py` runs strategies on live data; previously nothing in the CLI or API could start the PaperTrader - .env is loaded from the working directory, as the README always claimed - pyproject: PyPI metadata, build-system, dynamic version - run_dashboard.py / run_api.py stay as thin shims for existing clones - tests import the package directly and run against a temp data dir
- HLP: the tracker watched only the HLP parent vault, which holds idle USDC and no positions, so the panel showed "0 pos, net delta 0" forever. Child vaults now come from vaultDetails (hourly, with a static fallback) and each snapshot aggregates the parent plus every child (~$79M AUM, ~177 positions live). Absorptions are read from the `liquidation` object Hyperliquid puts on the fill instead of guessing from `crossed` / zero PnL. Per-vault seen sets replace a global set whose size pruning re-admitted still-returned fills as new. - Liquidations: Hyperliquid has no public feed, and the trade-size heuristic flagged every trade >= $10K (44 of 50 "liquidations" in a test run). Those are now "large prints": still in the feed and API with confirmed=false, excluded from every terminal total, alert digest, the cascade strategy and the LLM prompt. Real HL liquidations come from HLP fills Hyperliquid flagged, emitted as confirmed events. /v1/liquidations/stats gains ?include_estimated=false. - Basis and L/S fell back to nothing where Binance answers 451 (US, Singapore, China). Spot now falls back Binance -> Coinbase -> OKX and L/S Binance -> Bybit -> OKX, each snapshot carrying its source; the health monitor no longer cross-checks a fallback venue against Binance. - Combined view: distance-to-liquidation was formatted with the ratio formatter (x100), so a 970% whale showed +97058%. Distances above 999% are capped. - CVD: right after start every timeframe holds the same minute of trades and was labelled STRONG_BULL on 1h/4h. Snapshots carry window coverage; unfilled windows show "warm NN%" and the aggregate shows WARMING_UP until the 1h window is half covered. - Smart money re-analyzes every 30s until the first wallet ranks (the first pass ran before discovery, leaving the panel empty for 5+ min). - Smaller: SIDE header no longer truncates, the system panel says POSITION SCANS, market prices fit their column, the stream title no longer says BTC for an all-symbol feed.
`hyperdata mcp` serves the hub over the Model Context Protocol (stdio), so Claude Code, Claude Desktop, Cursor and other MCP clients can ask for live data instead of scraping dashboards. Ten read-only tools: market overview, per-asset view (funding on every venue, L/S, basis, order flow, positions near liquidation), liquidations with coverage notes, liquidation heatmap, whale positions, positions near liquidation, order flow, HLP vault, funding extremes, data health. Every result carries a meta block with uptime and warmup warnings so an agent can tell "not loaded yet" from "nothing happened". Ships as an optional extra (pip install "hyperdata-terminal[mcp]", mcp>=2.2,<3); the base install is unchanged. Also: the BTC price cross-check in the health monitor falls back to the OKX perp mark where Binance futures answers 451, instead of reporting "price unavailable".
…backs and MCP tools
…release - CI lints and imports hyperdata_terminal (it still pointed at src/) - the pyproject leg installs the [mcp] extra so the MCP tests run - new package job: build sdist + wheel, install the wheel into a clean venv, run hyperdata from outside the checkout, and fail if a top-level src/ or config/ package is installed - release.yml: on a v* tag, check the tag matches __version__, build, smoke test, publish with PyPI Trusted Publishing (no stored token)
…o GIF - README leads with a live demo GIF and a one-line install (pipx / uvx / pip), then a command table, the MCP server setup, the API, paper trading with a strategy file outside the package, data sources with the coverage and regional-fallback caveats, and configuration - assets/demo.gif is recorded from assets/demo.tape with vhs, so it can be regenerated every release; the AI avatar hero image is removed - GROWTH.md removed: a public file of pre-written promo posts reads as astroturfing to anyone who opens the repo - .env.example and the README no longer advertise BINANCE_API_KEY, which nothing reads - docs/DATA_INTEGRITY.md: Hyperliquid confirmed vs large prints, CVD warmup, regional fallbacks, HLP child vaults, new CLI commands - CLAUDE.md: new package layout, commands and patterns - site: one-command install, MCP section, hyperdata api/paper commands, corrected coverage note, llms.txt for AI answer engines - `hyperdata cvd --symbol` rejects symbols that are not streamed instead of crashing; symbols are upper-cased for every dashboard - cvd_momentum skips a 5m window that is still warming up
…ry start userFills returns each vault's last 2000 fills. With seven child vaults that is ~12k days-old fills per start, each pushed through the on_hlp_trade callbacks into the bounded SQLite writer queue (and re-inserted on every restart, since hlp_trades has no dedupe). Fills older than session start minus two minutes now seed the display history only. README offers a git+ install line that works before the first PyPI release.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eec7a92869
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| stats = feed.get_stats(window_minutes=minutes, include_estimated=include_estimated) | ||
| events = feed.get_recent(minutes=minutes, symbol=self._sym(symbol) if symbol else None) |
There was a problem hiding this comment.
When get_liquidations is called with a symbol, only recent is filtered; the advertised totals, estimated counts, and per-exchange breakdown still come from every symbol in the feed. An agent asking for BTC liquidations can therefore receive market-wide totals labeled as BTC, producing materially incorrect analysis. Aggregate the symbol-filtered events or add a symbol filter to get_stats().
Useful? React with 👍 / 👎.
| "llm_agent), a path to your own .py file, or module:Class. Repeatable. Default: cvd_momentum", | ||
| ) | ||
| paper.add_argument("--interval", type=int, default=30, help="seconds between evaluations (default 30)") | ||
| paper.add_argument("--balance", type=float, default=10_000.0, help="starting paper balance in USD") |
There was a problem hiding this comment.
With hyperdata paper --balance 0, shutdown calls trader.get_portfolio(), which divides by starting_balance when calculating total_pnl_pct. The resulting ZeroDivisionError occurs before trader.stop() and hub.stop() in the finally block, so the command crashes instead of printing the portfolio and skips graceful cleanup. Require a strictly positive balance during argument validation.
Useful? React with 👍 / 👎.
|
Soak test:
Run 1 is why eec7a92 exists: every start replayed each vault's last 2000 fills into SQLite (and would have duplicated them on every restart). The writer coped (no drops), but the rows were garbage. Run 2 shows history now only seeds the panel. Three transient errors in 10 minutes: two Hyperliquid WS reconnects (recovered) and one failed HLP |
MCP - tools are async def: the SDK runs plain def tools on a worker thread, which raced the hub loop (reproduced: 'deque mutated during iteration'; CVD bucket sums could lose updates) - get_liquidations(symbol=...) now filters totals and breakdowns too, not only the event list (it reported all-coin totals under the asked symbol) HLP - AUM comes from the parent's vaultDetails portfolio ($179.9M live) instead of summed clearinghouseState ($79M), which cannot see Strategy X - exposure is gross notional summed per vault before netting (A and B hold opposite sides of most coins; the netted figure understated it ~27x) - fills: userFillsByTime from a per-vault watermark instead of userFills (the last 2000 fills every poll): ~80 of the 1200/min weight budget instead of ~370; empty or short responses can no longer reset dedupe and replay fills; absorptions kept in their own buffer so 24h counts survive ~220 ordinary fills a minute - persistence stores absorptions only, not ~390k market-making fills a day Liquidations - estimated events (HL large prints) get their own buffer so they can no longer evict confirmed liquidations from the 4h/24h windows - get_stats(symbol=...), get_recent(include_estimated=...) - HL large-print socket subscribed to PEPE, which Hyperliquid lists as kPEPE, and was closed and reconnected every ~2s (pre-existing): coins are now filtered against the meta universe API - WS heartbeat totals, per-exchange counts and cascade labels use confirmed liquidations only; /v1/health adds total_estimated_liquidations and reports the real package version - /v1/long-short-ratio and /v1/basis carry the source venue; schema v5 stores the L/S source so history never mixes Binance and OKX accounts
- HYPERDATA_DATA_DIR in .env now works: .env is loaded before the data
dir is resolved (cli no longer imports paths at module import)
- hyperdata api exits 1 with a clear message when the port is taken
instead of claiming the API is up; --api-port before `api` is honoured
- Ctrl+C during the boot animation still stops the hub
- strategy loader: files can import sibling helper modules; constructor
arguments and a missing name are clean errors; module:Class sees the
working directory; a user strategy shadowing a built-in name is an
error instead of being silently dropped; a file given twice loads once
- hyperdata cvd --symbol SOL no longer labels everything Bitcoin; the
third-party 'Moon Dev' brand is gone from titles and docstrings
- Market Intelligence: basis and L/S share one row per asset so they fit
a 50-row terminal; spot older than 60s reads 'stale'
- health: a fallback L/S ratio is reported as warn ('not cross-checked'),
not pass
- docs: confirmed HL liquidations are rare backstop takeovers; HLP AUM,
gross exposure and Strategy X visibility; pyproject summary no longer
claims order flow across 5 exchanges; MANIFEST.in makes the sdist's
tests runnable
- 22 regression tests, one per finding
|
Independent review round: 3 high, 8 medium, ~10 low findings, all fixed in ba2a271, 39e9abf and 9e3f53a
22 regression tests in |
If vaultDetails failed at start, AUM began as the summed clearinghouse value (~$100M short, no Strategy X) and the first successful refresh (5 minutes later, since failures also reset the refresh timer) showed session PnL of +$100M in the panel, alerts and MCP. Session PnL is now measured against the first snapshot from the same AUM source, snapshots record their aum_source, and a failed vaultDetails call is retried on the next 30s pass.
|
Soak on the final code (cf80d48): real hub, 250s, stopped, 250s again on the same data dir. This supersedes the earlier soak comment, which ran before the HLP rework.
|
…ing) High - HLP session PnL was the change in AUM, so deposits and withdrawals read as profit (last day: AUM -$103K while PnL +$20K). It now comes from Hyperliquid's cumulative allTime.pnlHistory; '--' / null until known. Medium - one liquidation filled by Strategy A and B produced 2-3 confirmed events: absorptions are grouped by transaction hash (sizes summed), emitted once, and upserted by hash in hlp_trades (schema v5 adds fill_hash unique + vault), so restarts and late shares never duplicate - busy vaults read one 2000-fill page per poll and could fall behind in a spike and silently lose absorptions: pages forward up to 6 per poll and warns if still behind - docs said confirmed HL liquidations were rare backstops; live data shows mostly market fills by Strategy A/B, several an hour Low - vaultDetails AUM older than 15 min falls back to the summed value, and the source is shown: '(partial)' in the panel, aum_source/aum_is_partial in MCP, aum_source in hlp_snapshots - no 120s backfill emission any more (it re-emitted on quick restarts); pre-start fills are history, stored but never emitted - source fallback cooldown: 10 min only for 401/403/451, 30s for blips (one shared outage froze basis and L/S for 10 min); L/S rows go 'stale' - strategy helper modules from different folders no longer collide - HLP panel footer fits a 50-row terminal (trailing blank rows removed) - sdist includes ci.yml so its tests pass
DBs created by an earlier build of this branch already recorded schema v5 without hlp_trades.fill_hash/vault and hlp_snapshots.aum_source, so the amended v5 step would never run for them and the absorption upsert would fail. Moving those columns to v6 upgrades them; a test builds such a DB.
Medium - strategy loader evicted every module first imported from under the strategy's folder, including a project .venv's libraries: pandas split into two copies and raised on the first DataFrame. Only the folder's own top level modules/packages are evicted now. - the absorption upsert overwrote size, so a restart that re-read only one vault's share shrank a stored row; it now only ever grows - demo HLP crashed every cycle on the removed _session_start_value and bypassed absorption grouping; fixed and routed through _add_absorption Low - session PnL goes unknown (not frozen) when vaultDetails is stale - absorptions read mid pass are flushed on stop() - busy vaults resume one poll interval before start (no restart gap) - warning when a page starts past Hyperliquid's ~10k fill retention (the old 'still full after N pages' warning could never fire); pages per poll 3 to bound the rate burst - all zero transaction hashes are not used for grouping - HLP panel shows 3 absorption rows so the footer fits in a busy hour - unknown PnL is 'unknown' in alerts and pnl_known=0 in hlp_snapshots (schema v7) - docs: cooldowns, paging, restart behaviour, top level helper imports
|
Review rounds 2 and 3: every finding fixed (55d1dec, e2bb9bd, fd93ae9) Round 2 (independent reviewer, verifying round 1's fixes): 1 high, 2 medium, 7 low.
Round 3 (narrow, on round 2's fixes): 3 medium, 7 low.
Verification on fd93ae9: 380 tests (84 new), ruff clean, CI green including the clean wheel job. Live, from a wheel install outside the repo: combined view at 160x50 ($179.82M AUM, $7.88M gross, absorptions and footer visible), all MCP tools from a real stdio client with no errors. A 2 x 420s soak (legs longer than the 5 minute vaultDetails refresh) on 55d1dec: session PnL +$12 and +$184, zero dropped writes, zero 429s; the repeat on fd93ae9 found 3 smart money 429s, fixed with two more bugs in the comment below. |
A 2 x 400s soak with a restart drew 3 HTTP 429s: smart money analyzes 20 wallets per batch at 8 requests a second, and an active wallet's userFills costs ~120 weight, so a batch spent ~2400 of the IP's 1200 per minute in seconds (on main every 5 minutes; the warmup cadence in this PR moved the first batch to startup, next to the scanner and HLP first poll). Smart money now tracks the weight it spends and stays under 360 a minute, ranking every 5 wallets so the panel still fills early. site/og-image.png re-rendered from og.html: pipx install command and the MCP tool count instead of the old pip install -e line.
…allet OKX reports liquidation size in contracts, and the feed multiplied contracts by price: BTC-USDT-SWAP (0.01 BTC a contract) read 100x too large, DOGE (1000 a contract) 1000x too small, and inverse swaps were priced as coins. Contract values now come from OKX's instrument list; a swap missing from it is dropped and counted, never guessed. Smart money: rank after every analyzed wallet and count only ranked wallets, so the panel can no longer show 'Ranked:10' with no tiers. The panel explains the warmup until tiers exist.
The previous recording showed OKX liquidations at $10.5M an hour (the contract sizing bug). Off camera warmup is now 4 minutes so smart money tiers are populated, and the site's combined view screenshot is the GIF's last frame (1500x960) instead of a stale capture.
Codex review P2: 'hyperdata paper --balance 0' divided by zero in the exit summary, before the trader and hub were stopped.
- smart money: analyze never analyzed wallets first, then the longest since analyzed; under the weight budget, sorting by recent activity let busy analyzed wallets crowd out new ones so tiers could stall - liquidation feed: drop events with a non finite or non positive size or price (float() accepts nan and inf, which poisoned window totals) - OKX: count unsized drops in get_stats, await the reload task on stop - cli: --balance/--interval/--minutes must also be finite - test: the DOGE symbol assertion always passed (operator precedence)
|
Final verification pass: three more bugs found and fixed (c5da12d, bb9c89e, d110672, d2b40f4, c3fcbbb) 1. Smart money overran Hyperliquid's rate limit (c5da12d). A soak on fd93ae9 logged 3 HTTP 429s in 13 minutes: a batch of 20 active wallets'
The margin is thin: 1134 of 1200 on average, and the worst rolling minute in that run was ~1557 by this estimate, with no 429s. A soak on c5da12d (wheel install, 2 x 400s with a restart) logged 0 HTTP 429s from any component, 0 paging warnings, PnL known on all 157 snapshots, 0 dropped writes; a local network outage during it cost one HLP request and everything reconnected on its own. If 429s show up for users, the next lever is the market data refresh (5s) or the scanner. Trade off: tiers need 10 ranked wallets, so the smart and dumb money tables now fill in over about 4 minutes (measured: 3 ranked at 45s, 11 at 225s). bb9c89e ranks after every wallet (it was every 5, so the panel could read 2. OKX liquidation sizes were wrong, in both directions (bb9c89e, pre-existing on main). OKX reports liquidation size in contracts and the feed multiplied contracts by price. Live sample: BTC-USDT-SWAP 3. Codex review P2 was still open (d2b40f4). Review round 4 (independent, on the diff fd93ae9..d2b40f4): 0 high, 1 medium, 6 low; fixed in c3fcbbb.
Live on c3fcbbb (full hub from the wheel, 5 minutes): OKX 34 liquidations / $57K, Bybit 3 / $2.6K, 0 parse errors, 0 unsized drops, 0 HTTP 429s, smart money 14 ranked with 1 smart and 1 dumb. GIF and site screenshot re-recorded from the fixed build (d110672). 406 tests (26 new since fd93ae9), ruff clean, CI green. |
Why
A review of the repo found four things blocking growth, each verified by installing and running it:
pip install ., pipx, uvx or a wheel all failed withModuleNotFoundError: run_dashboard, and dropped genericsrcandconfigpackages into site-packages. Only an editable clone worked, so PyPI, Terminal Trove and every directory that wants a single install command were out of reach.What changed
Installable package and CLI
src/andconfig/move intohyperdata_terminal/; everysys.pathhack is gone. PyPI namehyperdata-terminalis free (hyperdatais taken).HYPERDATA_DATA_DIR, else an existing./datain a checkout, else the per user data dir. Never site-packages.hyperdata heatmap --symbol ETH,hyperdata whales,hyperdata all, ... open one dashboard directly;--no-boot,--version.hyperdata api,hyperdata verify, andhyperdata paper -s cvd_momentum -s ./my_strategy.py(strategy files can live anywhere).hyperdata-terminalis a second script name souvx hyperdata-terminalworks.run_dashboard.pyandrun_api.pystay as thin shims for existing clones.Data that tells the truth on first launch
vaultDetails($179.9M live, matching Hyperliquid's UI), discovers the child vaults, nets positions per coin (177 live) and reports gross exposure summed per vault ($7.8M). Strategy X's positions are not visible through the API; that is documented. Session PnL comes from Hyperliquid's own cumulative PnL series, not the AUM change (deposits and withdrawals move AUM far more). Absorptions come from theliquidationobject Hyperliquid puts on fills, read withuserFillsByTimefrom per vault watermarks with forward paging (under 10% of Hyperliquid's rate budget), grouped by transaction hash (one liquidation filled by Strategy A and B is one absorption), and stored as one row per hash.confirmed=false), kept in their own buffer and excluded from every terminal total, the alert digest, the cascade strategy, the LLM prompt and the WebSocket heartbeat. Confirmed Hyperliquid liquidations are the ones an HLP vault took the other side of (mostlymarketfills by Strategy A/B, several an hour), emitted once per transaction hash./v1/liquidations/stats?include_estimated=falseadded; the default is unchanged. Also fixed (pre-existing): OKX reports liquidation size in contracts and the feed multiplied contracts by price, so BTC liquidations read 100x too large and DOGE 1000x too small; sizes now use each swap's contract value from OKX's instrument list (live: 58 OKX liquidations in 4 minutes, $60.8K total, largest $9.2K). And the large print socket subscribed to PEPE, which Hyperliquid lists as kPEPE, so it was closed and reopened every ~2 seconds.coverage; unfilled windows showwarm NN%and the aggregate shows WARMING_UP.MCP server for AI agents
hyperdata mcp(optional extrahyperdata-terminal[mcp], mcp 2.x): ten read only tools covering market overview, per asset view, liquidations, heatmap, whales, positions near liquidation, order flow, HLP vault, funding extremes and data health. Every result has ametablock with warmup warnings. Tools areasyncso they run on the hub's event loop (a plaindeftool runs on a worker thread and races the hub).Docs, site, CI
vhs assets/demo.tape), one line install, command table and MCP setup. GROWTH.md and the avatar image removed.site/llms.txtadded for AI answer engines (ChatGPT and Perplexity already send traffic).release.ymlpublishes to PyPI on av*tag via Trusted Publishing (no stored token).Merge order
hyperdata-terminal, ownerCo-Messi, repositoryHyperData-Terminal, workflowrelease.yml, environmentpypi.ignoreCommandonly diffs the last commit (HEAD^..HEAD), and after a rebase merge that would be a commit that does not touchsite/, so the site would not redeploy. Vercel then serves the site, which already sayspipx install hyperdata-terminal.git tag v1.0.0 && git push --tags. The release workflow checks the tag against__version__, builds, smoke tests and publishes.Until steps 2 and 3, the README GIF (raw.githubusercontent.com on
main) and the PyPI badge render as broken in this PR's preview. The README also offerspipx install git+https://github.com/Co-Messi/HyperData-Terminal, which works the moment this merges.Breaking changes
src.tohyperdata_terminal.. Existing clones must rerunpip install -e .(the oldsrc.clientry point is gone);run_dashboard.pyandrun_api.pykeep working.HubStatus.total_liquidationscounts confirmed events only; estimated ones are intotal_estimated_liquidations.methodispartial(washeuristic)./v1/liquidations/statsstill includes estimated events by default.quantityis in coins (was contracts) andsize_usdis correct. OKX rows stored by earlier versions are wrong and are not migrated.hlp_tradesstores absorptions only.long_short_ratios.source(old rows default tobinance),hlp_snapshots.aum_sourceandpnl_known,hlp_trades.fill_hash(unique) andvault.hlp_tradesnow holds one row per absorbed liquidation instead of every HLP fill.Verification