Skip to content

Stop dropping Postgres tick tables between runs #43

Description

@Hackatro

Problem

tick_writer.py runs DROP TABLE IF EXISTS on the tick tables at the first write_tick() of a run. The rows already carry id_scenario and id_run — the schema is shaped for keeping many runs, and the code deletes them.

The #38 branch narrowed this further: the drop list is now derived from the loaded PDL roster instead of a fixed list, so tables written by a previous run under a different PDL survive with stale rows while Result_Simulator_Environment is always dropped. A query joining them returns plausible but wrong numbers.

Nothing calls run_stepwise() today — this path is pre-built for palaestrAI and other RL agents. Fixing the drop list is not worth doing, because this ticket removes dropping entirely.

Decision — append-only

  • Never drop a table
  • run_id is a UUID generated once in TickWriter.__init__, shared with the per-run CSV directory from the CSV output ticket
  • A runs registry table mirrors runs.json for ordering and metadata
  • Index on (id_scenario, id_run, period)
  • Deleting a run becomes DELETE FROM ... WHERE run_id = ..., scoped rather than destructive

Schema drift

Columns come from the agent's role (_PROPS_BY_ROLE in data_collector.py). If an entity id appears under a different role in another PDL, the table name matches but the column set does not, and df.to_sql(..., if_exists="append") fails with column ... does not exist. write_tick() catches that and sets self.enabled = False, so the whole run would write nothing after one log line.

Before appending to an existing table, read its columns and ALTER TABLE ... ADD COLUMN for anything missing.

Open question for the team: can an entity id legitimately change role between PDLs?

Scale

palaestrAI training means many episodes. A demonstrator run worth keeping and a throwaway training episode should not be written identically — flag the difference in the registry, or skip tick writes during training.

Scope

tick_writer.py, model.py (id_run=0 is hardcoded at the call site).

Acceptance

  • Two runs against different PDLs both keep their rows
  • Every row is attributable to exactly one run
  • The runs table lists both runs with their PDL and start time
  • A role change on an existing entity id extends the table instead of disabling the writer

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    improvementFor everything that improves an already existing feature.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions