MSX API

A data pipeline for the Muscat Stock Exchange (MSX): scrapes the symbol universe, backfills full daily + intraday history per symbol, stores everything in a local SQLite database, and runs data quality checks over the result.

Important: unofficial, reverse-engineered API

MSX has no official public API. This project talks to two endpoints on www.msx.om that were discovered through live network inspection of the public website (browser devtools, watching XHR/fetch traffic), not from any published documentation. They are unauthenticated and appear to back the site's own market-watch and charting widgets.

Because these endpoints are undocumented:

• They can change shape, move, or disappear without notice.
• There is no server-side date-range filtering on the chart-data endpoint. Every pull fetches a symbol's entire series; the update command filters new-vs-seen records client-side after the fact rather than asking the server for only what's new.
• Field semantics (e.g. exactly what Value vs LTP represents) are inferred from observed payloads, not from a spec. Treat them as best-effort.
• Be a considerate client: keep the inter-request delay in place, don't parallelize requests against the server, and don't run this more often than you need to.

Use accordingly. This is a personal/research tool, not a vendor-guaranteed data feed.

Confirmed endpoints

1. Market summary, POST /api.aspx/MarketToday with body {} and Content-Type: application/json. Returns {"d": [{...}]}, a single-item list of index-level fields (MSX30, Change, Turnover, Volume, Trades, ...). Numeric fields arrive as comma-thousands strings ("167,387,554").

2. Per-symbol time series, GET /company-chart-data.aspx?s={SYMBOL}. Returns a flat JSON array mixing daily closes (older records, open/ high/low null) and minute-level intraday ticks (recent records, which additionally carry Year/Month/Day/Hour/Minute).

3. Symbol discovery, POST /companies.aspx/List with body {} and Content-Type: application/json. Returns {"d": [{...}, ...]}, one record per listed instrument with a Symbol field plus classification codes (Market, Sector, SubSector, Listed).

   This replaces the originally targeted market-watch-custom.aspx + regex approach: that page turns out to render its symbol list client-side from a jQuery template (snapshot.aspx?s=${Symbol}), so the static HTML never contains real symbol values. The regex only ever matches page markup, not data. companies.aspx/List is the actual JSON endpoint the page's own JavaScript (js/securities.js) calls to populate that template, discovered by reading the site's JS during development and confirmed working live.

   Caveat on scope: this endpoint returns every instrument MSX tracks, equities, bonds, mutual funds, preferred shares, rights issues, around 290 records as of this writing. symbols.py narrows this to an equities-only universe (currently 105 symbols) by combining several signals found by cross-referencing sibling endpoints:

   • Listed == "1" (currently-listed/active status; confirmed against known reference symbols BKSB and BKMB).
   • Subtracting symbols returned by POST /mutual-funds.aspx/List (closed funds) and POST /bonds.aspx/List (bonds; currently zero overlap with companies.aspx/List but fetched for defense-in-depth).
   • Excluding names containing "RIGHT" or "PREF" (rights issues, preferred shares, identifiable by consistent naming, e.g. "GISP50 - GULF INV. SER. 50%- PREF. SHARES").

   105 is still above a commonly-cited ~77 headline-equity count; no further field was found that isolates a smaller set cleanly (there may be a REIT-like category MSX doesn't expose a distinguishing field for through this endpoint). Treat 105 as the best available approximation, not a guaranteed exact match. See the rationale documented next to SYMBOL_LISTED_STATUS_FILTER in config.py.

Project layout

msx_pipeline/
  __init__.py
  client.py     HTTP client: retries, timeouts, delay, logging
  symbols.py    symbol discovery + JSON cache (equities-only filter)
  storage.py    SQLite schema + upsert logic
  ingest.py     orchestration: symbols -> fetch -> store
  quality.py    data quality checks + report rendering
  strategy.py   momentum signal generation (ranking)
  portfolio.py  position sizing, rebalancing, transaction cost + slippage model
  simulator.py  day-by-day backtest engine + incremental live/paper mode
  metrics.py    performance metrics (CAGR, Sharpe, drawdown, fill-time, etc.)
  export.py     self-contained JSON/CSV/xlsx export of a simulation run
  cli.py        argparse CLI: discover-symbols / backfill / update / validate /
                 backtest / run-daily / list-runs / export
  config.py     all constants (URLs, delay, DB path, thresholds, strategy params)
requirements.txt
README.md

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

No credentials or API keys are required (and none should ever be added. These endpoints are unauthenticated).

Getting started: from a clean clone to a running simulation

This walks through every step from nothing to a locked, exportable backtest run. Commands are copy-pasteable in order; nothing here assumes you've read the rest of this README first.

1. Install

git clone <this-repo-url>
cd msx_pipeline   # or whatever you cloned it as
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

No credentials or API keys. The endpoints this project talks to are unauthenticated (and none should ever be added, see below).

2. Initial data setup

# Scrape and cache the current equities universe (currently 105 symbols)
python -m msx_pipeline.cli discover-symbols

# Full historical pull for every cached symbol -- first-time only, takes a
# few minutes at the default 1.5s inter-request delay (thousands of daily
# records per symbol for long-listed names, going back to 2007)
python -m msx_pipeline.cli backfill

This creates data/msx.sqlite3 (gitignored, local to your machine). Check it landed cleanly:

python -m msx_pipeline.cli validate

3. Start a new simulation run

python -m msx_pipeline.cli backtest --start 2024-01-01 --end 2026-07-23

This replays the momentum strategy over the given date range and prints a performance report (CAGR, Sharpe, max drawdown, alpha vs. benchmark, entry fill-time distribution) at the end. Every strategy/simulation parameter is a CLI flag with a config.py default, so you don't need to edit source to experiment:

Flag	config.py constant	Meaning	Default
--lookback	MOMENTUM_LOOKBACK_DAYS	N-trading-day momentum window	20
--top-k	MOMENTUM_TOP_K	how many symbols to hold, equal-weighted	10
--capital	STARTING_CAPITAL	starting capital, OMR	5,000,000
--cost-rate	TRANSACTION_COST_RATE	transaction cost, fraction of trade notional	0.0015 (0.15%)
--position-limit	POSITION_LIMIT_PCT	max portfolio fraction in one symbol	0.15 (15%)
--adv-cap	ADV_PARTICIPATION_CAP	max trade size as a fraction of trailing ADV	0.10 (10%)
--rebalance-frequency	REBALANCE_FREQUENCY	weekly or monthly re-ranking cadence	weekly

Two more knobs exist only in config.py (not exposed as CLI flags, since they shape the benchmark construction rather than the strategy itself, but are still recorded per-run for reproducibility, see below): BENCHMARK_CONSTITUENT_COUNT (how many of the highest-turnover symbols make up the reconstructed benchmark, default 30) and BENCHMARK_TURNOVER_LOOKBACK_DAYS (the trailing window used to rank them, default 60).

Pass --run-id my-experiment-name to tag the run with a name you'll recognize later; omit it and one gets auto-generated (backtest-YYYYMMDDTHHMMSS).

# Example: a named run with a couple of parameters overridden
python -m msx_pipeline.cli backtest --start 2024-01-01 --end 2026-07-23 \
    --run-id my-experiment --top-k 8 --rebalance-frequency monthly

4. Find a run's run_id afterward

python -m msx_pipeline.cli list-runs

Prints every simulation_runs row (most recent first): run_id, mode, status, date range, and the full parameter set (rebalance frequency, lookback, top-K, capital, cost rate, ADV cap, position limit). No need to hand-write a sqlite query to find or distinguish between runs.

5. Continue a locked run going forward

Once you've picked a run to treat as "the" ongoing strategy (i.e. its run_id), advance it one trading day at a time as fresh market data comes in:

python -m msx_pipeline.cli update              # pull the latest data first
python -m msx_pipeline.cli run-daily --run-id my-experiment

update fetches whatever's new since the last pull; run-daily then advances the named run by exactly one day, the latest date now in daily_history, loading its prior cash/positions from that run's most recent snapshot. Run this pair once per MSX trading day (Sunday through Thursday, MSX is closed Friday/Saturday), e.g. via a daily cron job after market close. Calling run-daily again before update has pulled a newer day is a harmless no-op (logged, not an error), so it's safe to over-run rather than under-run the schedule.

6. Export results

python -m msx_pipeline.cli export --run-id my-experiment --format json --out my-experiment.json
python -m msx_pipeline.cli export --run-id my-experiment --format csv  --out my-experiment-csv/
python -m msx_pipeline.cli export --run-id my-experiment --format xlsx --out my-experiment.xlsx

Pulls everything tied to that run_id into one self-contained file (or folder of CSVs). No DB access needed downstream. See the export section below for the exact JSON/CSV/xlsx shape.

---

Global flags (any subcommand): -v / --verbose for debug-level logging. Ingestion-only flags (discover-symbols, backfill, update):

• --delay SECONDS, override the inter-request delay (default 1.5s, also settable via MSX_REQUEST_DELAY).
• --rediscover, re-scrape the symbol list live instead of using the cache.

Configuration

All tunables live in msx_pipeline/config.py and can be overridden via environment variables:

Variable	Purpose	Default
MSX_REQUEST_DELAY	seconds between requests	1.5
MSX_DATA_DIR	directory for DB + symbol cache	<repo root>/data
MSX_DB_PATH	SQLite file path	<data dir>/msx.sqlite3
MSX_SYMBOLS_CACHE	symbol cache JSON path	<data dir>/symbols_cache.json

All three resolve relative to the repo root (config.PROJECT_ROOT, derived from config.py's own file location), not the process's current working directory, so python -m msx_pipeline.cli ... lands on the same ./data/msx.sqlite3 regardless of which directory you invoke it from. If you set one of these env vars to a relative path, it's joined onto PROJECT_ROOT too, for the same reason; set an absolute path if you deliberately want the DB somewhere else entirely. data/ (the DB, symbol cache) and exports/ (saved run exports) are both gitignored. They're local/generated artifacts, not source.

Strategy/simulation parameters (--lookback, --top-k, --capital, --cost-rate, --adv-cap, --position-limit on backtest/run-daily) default to the constants documented in the next section and can be overridden per-invocation without editing config.py.

Database schema

• symbols: symbol (PK), first_seen_date, last_updated
• daily_history: symbol (FK), date, close, volume, turnover, open, high, low, unique on (symbol, date)
• intraday_ticks: symbol (FK), timestamp, close, volume, turnover, unique on (symbol, timestamp)

Records from company-chart-data.aspx are routed automatically: a record carrying non-null Hour/Minute fields goes to intraday_ticks, otherwise it goes to daily_history. All writes are upserts, so re-running backfill or update is idempotent and safe to schedule on a cron.

Data quality checks (validate)

• Gaps: missing MSX trading days (Sun-Thu; Fri/Sat excluded) between consecutive daily records beyond MAX_ALLOWED_GAP_DAYS.
• Zero-liquidity stretches: runs of near-zero volume days at or above ZERO_LIQUIDITY_RUN_THRESHOLD, likely illiquid small-caps worth excluding from anomaly detection rather than treating as data errors.
• Turnover consistency: turnover ≈ volume × close within TURNOVER_TOLERANCE (default 5%); persistent large deviations usually mean a units mismatch rather than genuine market behavior.
• Duplicates / out-of-order dates: sanity check on the stored series, since the UNIQUE(symbol, date) constraint should already prevent true duplicates at write time.

The report is printed to stdout and can optionally be written to a file with --output.

Notes on running backfill

With the default 1.5s delay, a full backfill of the ~105-symbol equities universe takes a few minutes of request time alone (before payload size / parse time), each pulling a symbol's entire history (thousands of records for long-listed names like BKSB, going back to 2007). Run it once, then switch to update for subsequent refreshes.

Strategy & fund simulation

v1 strategy: cross-sectional momentum, long-only. For each symbol, compute the N-day (N trading days, default 20) return. Rank symbols by that return descending; go long the top K (default 10), equal-weighted, rebalanced on a configurable cadence (REBALANCE_FREQUENCY in config.py, or --rebalance-frequency {weekly,monthly} per run): the first trading day of each new ISO week or calendar month, not a fixed weekday/day-of-month, so an MSX holiday shifting which day is first in a period doesn't silently skip a rebalance. No shorting, no other signals.

Fund simulation model

• Starting capital: 5,000,000 OMR.

• Transaction cost: 0.15% of trade notional per trade. This is the rate given in the original spec, not a verified MSX brokerage fee. No endpoint or published fee schedule was found during development (a web search for one was attempted and came back unavailable at the time); treat it as a placeholder and adjust TRANSACTION_COST_RATE in config.py if you find an authoritative figure.

• Slippage: modeled as a volume-participation cap, not a price-impact formula (the spec didn't ask for one). A single trade can't exceed 10% of the symbol's trailing 20-day average daily volume (ADV_PARTICIPATION_CAP / ADV_LOOKBACK_DAYS). If the desired size exceeds that, only the allowed portion fills; the trade is logged with partially_filled=1 and the gap is recorded in trades.unfilled_shares. BUY orders are additionally capped so they never push cash negative (a separate, cash-affordability constraint layered on top of the same mechanism).

• Position limit: max 15% of portfolio value in any one symbol (POSITION_LIMIT_PCT). With the defaults (K=10, limit=15%) equal weighting (10% each) never actually triggers the cap; it only matters if you lower K below ~7 via --top-k.

• Benchmark: no historical MSX30 time series exists through any endpoint found during development. MarketToday is a live current-value snapshot only, and performance.aspx/PerformanceChartData (found by reading js/indices.js) returns a single period-summary record (current value/high/low/turnover), not a daily series. The benchmark is instead a reconstructed proxy: equal-weighted daily returns of the BENCHMARK_CONSTITUENT_COUNT (default 30) symbols with the highest trailing average turnover, reconstituted daily and compounded from the same starting capital. Equal-weighting all 105 equities was tried first and discarded. Several illiquid micro-caps (e.g. SHPS, HECI) regularly move 50-160% in a single day on a few hundred OMR of turnover, which is real data (not corrupted) but makes a naive all-symbol average wildly non-representative of "the market." Restricting to the most liquid names is closer in spirit to MSX30's own curated, largely liquidity-driven constituent selection.

  The turnover ranking used to pick constituents is a rolling BENCHMARK_TURNOVER_LOOKBACK_DAYS-day (default 60) trailing average, lagged by one day, recomputed every trading day, not an average over the whole comparison period. An earlier version used the whole-period average, which is look-ahead bias (day 1 of a multi-year backtest would already "know" which symbols end up most liquid years later); this was caught in review and fixed. It is still a proxy, not the official index. Treat comparisons against it as directional.

backtest / run-daily

python -m msx_pipeline.cli backtest --start 2024-01-01 --end 2026-07-23 \
    --run-id my-run --top-k 10 --lookback 20

python -m msx_pipeline.cli run-daily --run-id live

backtest replays the strategy over a historical window in one call, tagging every trade/snapshot with a run_id (auto-generated if you don't pass --run-id) so multiple experiments can coexist in the same DB. run-daily advances a persistent run by exactly one day, the latest date in daily_history, loading prior cash/positions from that run's most recent snapshot. Run it once per day after update; calling it again before new data has landed is a no-op (logged, not an error).

New database tables

• simulation_runs: run_id (PK), mode (backtest/live), created_at, start_date, end_date, every SimulationParams field (momentum_lookback_days, top_k, starting_capital, transaction_cost_rate, adv_participation_cap, position_limit_pct, rebalance_frequency), plus the benchmark/ADV settings that aren't per-run CLI knobs but are recorded anyway for reproducibility (adv_lookback_days, benchmark_constituent_count, benchmark_turnover_lookback_days, snapshotted from config.py at run-creation time), and status.
• trades: run_id (FK), date, symbol, side, requested_shares, filled_shares, price, gross_value, transaction_cost, unfilled_shares, partially_filled, one row per simulated order.
• portfolio_snapshots: run_id (FK), date, cash, positions_value, total_value, benchmark_value, positions_json ({"SYMBOL": shares, ...}), num_positions, unique on (run_id, date), one row per trading day per run.

Databases created before rebalance_frequency and the three benchmark_*/adv_lookback_days columns existed are migrated automatically the first time storage.get_connection() runs (idempotent ALTER TABLE ... ADD COLUMN, backfilling existing rows with the then-current defaults, accurate, since those runs did in fact use weekly rebalancing and the config values in place at the time).

Entry fill-time metric

Every backtest run also reports the entry fill-time distribution: for each position entry (the day a symbol's held shares go from zero to nonzero, i.e. it was newly selected), how many calendar days until its mark-to-market weight reaches the target weight (within POSITION_FULL_TARGET_TOLERANCE, default 1%). Three outcomes per entry:

• completed, reached target weight; contributes a days-to-fill value.
• incomplete, sold back to zero (dropped by a later re-rank) before ever reaching target.
• still_open, still building at the end of the backtest window (right-censored, not a failure).

metrics.compute_entry_episodes(conn, run_id) returns the raw per-entry records; metrics.compute_entry_fill_times(conn, run_id) aggregates them into mean/median/min/max. Implemented by walking portfolio_snapshots (no need to replay trades), so it's a pure post-hoc analysis over already- persisted data.

export

python -m msx_pipeline.cli export --run-id my-run --format json --out my-run.json
python -m msx_pipeline.cli export --run-id my-run --format csv  --out my-run-csv/
python -m msx_pipeline.cli export --run-id my-run --format xlsx --out my-run.xlsx

Pulls everything tied to one run_id out of the DB into a self-contained file (or folder of files). No DB access needed to use the output. --format json (default) writes one JSON file shaped:

{
  "run": { /* full simulation_runs row: params, rebalance_frequency,
             adv_lookback_days, benchmark_constituent_count,
             benchmark_turnover_lookback_days, date range, status */ },
  "metrics": {
    "strategy": { /* PerformanceMetrics: cagr, sharpe_ratio, max_drawdown, ... */ },
    "benchmark": { /* same fields, for the benchmark series */ },
    "alpha_cagr": -0.3293  // strategy.cagr - benchmark.cagr
  },
  "portfolio_snapshots": [
    { "date": "2024-01-01", "cash": ..., "positions_value": ..., "total_value": ...,
      "benchmark_value": ..., "num_positions": 10, "positions": {"BWPC": 10683.08, ...} },
    ...
  ],
  "trades": [
    { "date": ..., "symbol": ..., "side": "BUY"|"SELL", "requested_shares": ...,
      "filled_shares": ..., "price": ..., "gross_value": ..., "transaction_cost": ...,
      "unfilled_shares": ..., "partially_filled": true|false },
    ...
  ],
  "fill_times": [
    { "symbol": ..., "signal_date": ..., "status": "completed"|"incomplete"|"still_open",
      "completion_date": "2024-07-08" | null, "days_to_fill": 14 | null },
    ...
  ],
  "benchmark": [
    { "date": "2024-01-01", "benchmark_value": 5052927.60 },
    ...
  ]
}

trades.unfilled_shares is the "slippage" quantity the task spec asked for logged per trade, see the fund simulation model section above for why it's a volume-participation gap, not a price-impact cost. portfolio_snapshots and benchmark overlap (benchmark_value appears in both) deliberately, so benchmark can be used standalone as its own series without reaching into snapshot records.

--format csv writes the same data as separate flat files in --out/: run_metadata.csv (single row), metrics.csv (one row per series, strategy/benchmark, with alpha_cagr repeated on both), snapshots.csv (with positions flattened to a positions_json string column, since CSV has no native nested-object support), trades.csv, fill_times.csv, benchmark.csv. All read cleanly with pandas.read_csv (verified during development); positions_json needs json.loads per cell to unpack.

--format xlsx writes the same six tables as sheets in one workbook (Snapshots, Trades, FillTimes, Metrics, Benchmark, RunMetadata) via openpyxl, a header row, columns auto-sized to their content, no charts or conditional formatting beyond that. Opens directly in Excel with no pandas/CSV-import step. positions is flattened to JSON text in Snapshots the same way as the CSV export. One thing worth knowing: inf/nan float values (e.g. adv_participation_cap on a diagnostic run with the ADV cap disabled) are silently written as blank cells by openpyxl by default. This export casts them to their string form ("inf") first so the value stays visible instead of disappearing.

Diagnostic finding: liquidity constraint dominates the underperformance

Two diagnostic backtests (not candidate strategies) were run over 2024-01-01 -> 2026-07-23 to isolate why the strategy underperforms its benchmark, both starting from the standard weekly/K=10/ADV-capped baseline (CAGR 6.00%, alpha -32.93% vs benchmark, only 39.4% of entries ever reached target weight):

1. ADV cap disabled (--adv-cap inf, diagnostic only, not a tradeable strategy, since it assumes unlimited instant fills at the quoted close): CAGR jumped to 44.95%, alpha flipped to +6.01%, and the entry completion rate jumped to 93.2%. This shows the momentum signal itself has positive edge over the benchmark when execution is unconstrained. The underperformance is not primarily a signal-quality problem.

2. Monthly rebalancing instead of weekly (--rebalance-frequency monthly, ADV cap fully active/realistic): completion rate improved somewhat (39.4% -> 45.5%), confirming slower re-ranking does let more positions finish building. But it did not close the gap to benchmark. CAGR fell further, to 4.09%, and alpha widened slightly to -34.84% (Sharpe improved to 1.33 and max drawdown shrank to -2.47%, but off a much smaller total return). Reported plainly as a negative result: MSX's liquidity constraint is severe enough that rebalance cadence alone does not fix it. No further parameter tuning was applied to this run.

Both diagnostics are saved as simulation_runs rows (see exports/ for JSON dumps) rather than deleted, since they're real analysis output.

Known v1 limitations

• Momentum ranking and position valuation use two different price views on purpose: only symbols with an actual print today are tradeable, but existing holdings are marked to market using the last known price (forward-filled) so a thinly-traded position isn't valued at zero on a day it simply didn't trade.
• 23 rows in daily_history have close == 0 (almost certainly halted/suspended-trading days MSX reported as 0 rather than omitting, spread across 2007-2012). These are treated as missing data, not a real price, everywhere in simulator.py. A literal 0 would otherwise produce an infinite one-day return the moment the price resumes.
• Because of ADV-based partial fills, the strategy can end up holding more than top_k symbols at once. Thin names take multiple weekly rebalances to fully enter or exit, so small residual positions accumulate beyond the current top-K target. This is a real consequence of MSX's liquidity profile, not a bug.