[DMP 2026]: IDinsight Experiment Analytics & Bandits Engine Enhancements

[markbotterill]

Ticket Contents

Description

Evidential is an open-source experimentation platform designed for nonprofits to run rigorous, low-cost experiments (A/B tests, Bayesian methods, and bandits) on digital and hybrid programs. This ticket focuses on building and improving the experimentation analytics and adaptive allocation engine.

Goals & Mid-Point Milestone

Goals

• Continue to build out the multi-armed bandit engine (e.g. Thompson Sampling / UCB)
• Add contextual bandits support using user-level features
• Improve experiment results unified dashboard
• Improve existing integrations with platforms like Turn.io

Setup/Installation

Quickstart guide is here

Expected Outcome

1-2 months collaboration with the team culminating in multiple feature Pull Requests into the Agency Fund / IDinsight repository. Features that are most appealing and productive for the collaboration can be clarified in subsequent early-stage meetings.

Acceptance Criteria

Knowledge of Python, basic statistics (we are willing to mentor the RL aspects of the project) and (ideally) some broader deployment/ infrastructure knowledge (integrating with APIs, deploying via Railway, GHA etc.)

Implementation Details

Backend:

• Python (FastAPI)
• SQLAlchemy / Postgres
• Frontend in React

Experimentation:

• Frequentist methods:
  • t-tests / proportion tests
• Bayesian methods:
  • Beta-Bernoulli models
• Bandits:
  • Thompson Sampling
  • UCB
  • Contextual bandits (Bayesian linear regression)

Data:

• Event data + warehouse integration
• Metric computation layer

APIs:

• Experiment creation
• Result retrieval
• Allocation updates

Mockups/Wireframes

No response

Product Name

Evidential

Organisation Name

IDinsight

Domain

Open Source Library

Tech Skills Needed

Python, SQL, Docker, CI/CD, React

Mentor(s)

@markbotterill @poornimaramesh

Category

Data Science

[Somsubhra-Nandi]

Hi @markbotterill @poornimaramesh,

I'm very interested in tackling Issue #766 for DMP 2026. I have strong experience building out complex Python backends (FastAPI) and automated data pipelines, and I'd love to help build out the experimentation analytics engine.

I noticed the quickstart guide link and located the main codebase over at agency-fund/evidential-be and have started reviewing the current FastAPI routes and SQLAlchemy models.

Before I draft a formal implementation plan for the Multi-Armed Bandit (Thompson Sampling / UCB) logic, I had a quick architectural question: Are we planning to calculate the metric computation layer purely on the fly within the FastAPI backend, or should the logic rely heavily on the Postgres warehouse integration for pre-aggregating the event data?

I will drop by the Discord soon to discuss further, but I'm excited to start working on this!

[santhosh-7777]

Hi @markbotterill @poornimaramesh,
This is a really interesting problem space, especially building a robust experimentation platform with adaptive allocation using bandits.
I’ve been working on projects involving data-driven decision systems and ML-backed pipelines, including:

• Building data processing pipelines for extracting and structuring real-time inputs for downstream decision logic
• Designing backend systems where model outputs dynamically influence system behavior (inference → decision → response)
• Working with Python-based APIs and integrating statistical/ML components into production-style workflows
  These experiences have given me a strong foundation in handling event-driven data, metric computation, and designing systems where decisions evolve based on incoming data — which aligns closely with experimentation engines and bandit-based allocation.
  I’ve started exploring the Evidential backend to understand how experiment data flows across ingestion, metric computation, and allocation. I’m particularly interested in contributing to:
• Thompson Sampling / UCB implementation
• Designing scalable allocation update mechanisms
• Extending toward contextual bandits using user-level features
  One aspect I’m thinking about is how to design the exploration–exploitation balance in a way that remains stable under sparse or noisy data, especially in early stages of experiments.
  I’ll continue exploring the repository and share a concrete implementation plan soon.
  Looking forward to contributing!

[Sarfaraz604]

@markbotterill @poornimaramesh

I'm Sarfaraz, a 4th semester CS student at IIIT Agartala. I've been going through the Evidential codebase and this ticket genuinely excites me — the intersection of adaptive experimentation and nonprofit impact is a problem space I want to invest real time in.

From exploring the backend, I can see the allocation layer currently handles Thompson Sampling at a per-experiment level. A few things I'm already thinking about:

• For contextual bandits, covariate shift will happen for user-level features through time, so I'm interested in implementing an online posterior Bayesian linear regression that would enable us to perform stable allocation decisions without batch retraining.
• For UCB algorithms, it's important to know whether bounds are calculated each time for allocation decisions or are they cached? This might become crucial in production when running multiple tests simultaneously.
• For the dashboard, it is crucial to display per-arm posteriors as well in order to provide more insightful information about the results of experiments for our nonprofit customers who don't necessarily have expertise in statistics.

I feel comfortable on the full stack: from Python/FastAPI backend, passing by SQLAlchemy database layer and React frontend to Docker containerization.

Having experience in working with data driven decision systems in combination with statistical models gives me some additional advantage regarding this task.

I will propose my approach to implementing this module next week. I can discuss implementation via call or even use Discord if that would help moving things faster.

This issue seems very interesting to me, so I'd really like to work on it.

[Diyapareta]

Hi @markbotterill @poornimaramesh

I’ve been going through Issue #766 and exploring the Evidential backend, and I’d really like to contribute to this.

My background is primarily in Python and backend development using FastAPI, where I’ve worked on building APIs, handling data flow, and structuring systems that process and respond to incoming data. While I’m relatively new to bandit algorithms, I’m comfortable with core statistics and I’m actively learning concepts like Thompson Sampling and UCB as I go through the codebase.

From my initial exploration, I’m particularly interested in contributing to the allocation layer and understanding how experiment data flows from event ingestion to metric computation and finally to decision-making. I’d like to start by working on a clear and simple implementation of the bandit logic (Thompson Sampling/UCB), and then gradually explore improvements or extensions once I have a solid understanding of the current system.

One thing I’m trying to understand better is how tightly coupled the allocation logic is with the metric computation layer — whether decisions are expected to rely on real-time computed metrics or pre-aggregated data from the warehouse.

I’ll continue digging into the repository and share a more concrete approach soon. Looking forward to contributing and learning through this project!

Thanks!

[likhhithh]

hello @markbotterill I’m interested in this project and would love to contribute. Could you please assign this issue to me?

[prisha-sh]

Hi @markbotterill @poornimaramesh sir,

I have started exploring the Evidential backend and going through the FastAPI routes, and data flow across experiment creation, metric computation, and allocation. This problem space of building an experimentation platform with adaptive decision-making is very interesting to me.

My initial approach would be to first implement a clear and well-tested version of Thompson Sampling within the current allocation layer, ensuring correct reward updates and stable behavior. Once that foundation is solid, I would like to extend it to UCB and then gradually move towards contextual bandits using user-level features.

I am also trying to better understand how the metric computation layer interacts with the allocation logic, specifically whether allocation decisions are expected to rely on real-time computed metrics or pre-aggregated data from the warehouse.

I have experience working with Python and backend development, and I am comfortable working with APIs and data-driven systems. While I am still strengthening my understanding of bandit algorithms, I am actively learning and would really value guidance from the mentors on this.

I will continue exploring the repository and will share a more concrete implementation plan shortly. Looking forward to contributing.

Thank you.

[AshutoshSingh058]

Hi mentors @markbotterill @poornimaramesh,

I’m quite interested in the IDinsight Experiment Analytics & Bandits project and have started exploring the repositories to understand the system better.

I noticed 'experiments-engine' and 'evidential-turn-app' — could you please let me know which repo would be the best starting point for contributors to focus on?

I’d like to study the right codebase and see where I can contribute meaningfully.
Thanks

[dhruveshmishra]

Hi @markbotterill @poornimaramesh,

I’m really interested in contributing to Issue #766 for DMP 2026. I have experience with backend development and I’m currently expanding my skills in Python and FastAPI, along with learning more about experimentation systems and data-driven applications.

I’ve also had a Pull Request merged in an open-source project, which helped me understand collaboration workflows and code quality expectations.

You can check my GitHub here:
https://github.com/dhruveshmishra

I’ve started exploring the repository to understand the current architecture, especially around experiment APIs and data flow.

I had a quick question regarding the implementation:
Should the metric computation and bandit allocation logic be handled dynamically within the FastAPI backend, or should it rely more on pre-aggregated data from the Postgres/data warehouse layer?

I’m excited to contribute, learn, and take on tasks step by step.

Thanks!

[markbotterill]

Hi all - I have been directed to point you in the direction of the C4GT Badal portal here if you are interested in this opportunity. Many thanks and all the best :)

[Siri-026]

Hi @markbotterill @poornimaramesh,

I’ve been exploring the Evidential backend and understanding how data flows from event ingestion to metric computation and allocation decisions.

I’m particularly interested in approaching the bandit engine as a modular system, starting with Thompson Sampling, then extending to UCB and contextual bandits, while ensuring stability under sparse or delayed feedback, which seems important for real-world experiments.

I’ll continue going through the repository and refine a concrete implementation approach. Looking forward to contributing!

[neha222222]

Hi @markbotterill @poornimaramesh, I'm Neha Dhruw — building a proposal for DMP 2026 around the Evidential bandits engine, and before commenting on scope I built a small bandit engine sharing the interface I'd propose for the Evidential extension to ground the discussion in working code.

Repo: https://github.com/neha222222/thompson-sampling-demo

What's there:

• BanditPolicy ABC mirroring the proposed Evidential extension (select_arm / update / posterior_summary) implemented as a working interface rather than just a design sketch so it can be validated in practice
• Six policies sharing that interface: Thompson Sampling for Beta and Normal distributions, UCB1, UCB-V, Contextual Thompson Sampling using Bayesian linear regression with Sherman-Morrison rank-1 updates, and LinUCB, all implemented consistently to compare behavior
• MetricStore abstraction that decouples reward computation from allocation, with Postgres-backed and in-memory implementations sketched so policies remain testable without infrastructure dependencies
• Audit log plus JSON persistence so any allocation can be deterministically replayed from logs given a seed, enabling debugging and reproducibility
• Bernoulli, Gaussian, and contextual-linear simulation environments to exercise different reward structures and validate policy behavior
• Unit tests including simulation-regret correctness checks where regret must be sub-linear and below uniform-allocation regret, along with a CI matrix on Python 3.10, 3.11, and 3.12
• Colab notebook walking through cumulative-regret plots, posterior densities, and Monte Carlo probability-of-best summaries to make results interpretable

A few things I'd like to validate with you before I draft the full proposal:

1. Repo to focus on. @AshutoshSingh058 asked the same — is IDinsight/experiments-engine or agency-fund/evidential-be the canonical target for this DMP work, so I align contributions correctly from the start?
2. Metric layer vs allocation layer. I want to decouple metric computation from allocation via a MetricStore interface with PostgresMetricStore (warehouse-backed, eventual-consistent) and InMemoryMetricStore (real-time) implementations so bandit policies stay independently testable, and this structure is already sketched in the demo repo — does it match how you'd want it upstream?
3. Per-arm vs shared posteriors for contextual. Independent per-arm posteriors are simpler and commonly used in production, while shared feature posteriors with arm dummies allow statistical strength sharing when feature space matters more than arm identity, so which direction would you prefer for nonprofit-scale experiments?
4. Allocation bound caching. @Sarfaraz604 raised this earlier, and my current take is to recompute on every allocation since it's microseconds per request at this scale and caching introduces staleness risks that are hard to detect post-hoc, but I'm curious if you see a stronger case for caching.

I'm aiming to land a full proposal in the next ~5 days and would be happy to share an early draft once these are clarified.

— Neha (https://github.com/neha222222)

[paanipoorie]

Hi @markbotterill while preparing my proposal for DMP 2026 I built a small adaptive experimentation sandbox inspired by the Evidential workflow to better understand the assignment → outcome → telemetry lifecycle in practice.

I focused mainly on:

• Thompson Sampling allocation flow
• replay/audit-oriented telemetry
• service-layer separation
• PostgreSQL-backed experiment state
• async-ready posterior recomputation concepts

The sandbox repo is pinned on my GitHub profile and includes a working API walkthrough, telemetry flow exploration, and experimentation lifecycle implementation used to ground my proposal in actual engineering decisions rather than only architectural discussion.

Looking forward to refining the proposal further based on maintainer feedback.

[anmolsinghh614]

Hi @markbotterill @poornimaramesh,

I’m Anmol Singh, a pre-final year student at IIT Kharagpur currently working as a Backend Engineering Intern at TrueFoundry, where I work on scalable backend systems, APIs, and infrastructure. I’ve also been involved in research internships at IIT Kharagpur and Louisiana State University, where I worked on ML systems, probabilistic modeling, distributed learning setups, and data-intensive experimentation workflows.

I’ve submitted my proposal for this project through the C4GT portal and have kept it among my highest priorities because the problem space genuinely excites me. The combination of experimentation infrastructure, adaptive decision-making, analytics, and real-world impact for nonprofits makes this project extremely meaningful from both a systems and research perspective.

My previous experience aligns very closely with the engineering challenges involved here. Through TrueFoundry, I’ve worked on backend architectures, scalable APIs, databases, CI/CD workflows, and production-oriented engineering. My research experience exposed me to probabilistic ML systems, Bayesian-style reasoning, distributed computation, and data pipelines, which directly connects with experimentation engines and adaptive allocation systems. I’m also comfortable with Python, FastAPI, SQL, Docker, and analytics-heavy workflows.

From exploring the repository and architecture discussions, my current implementation direction would be to structure the allocation engine around modular policy abstractions supporting Thompson Sampling, UCB, and future contextual bandit extensions. I also want to decouple metric computation from allocation logic through clean interfaces so experimentation policies remain independently testable and scalable across both real-time and warehouse-backed metric pipelines. On the analytics side, I’d like to improve interpretability through richer posterior visualizations, allocation history tracking, and experiment summaries.

Tentatively, my expected execution timeline would be:
• Week 1–2: Deep repository exploration, architecture understanding, and stabilization of existing experimentation workflows
• Week 3–5: Modular bandit engine improvements (Thompson Sampling/UCB), testing, and allocation abstractions
• Week 6–8: Contextual bandit extensions, analytics/dashboard improvements, and deployment/testing refinements

What motivates me most is the opportunity to work on a system that combines strong engineering, statistics, experimentation theory, and practical social impact. I’d genuinely love the opportunity to contribute meaningfully to Evidential and collaborate closely with the team throughout the project.

[aayushprsingh]

Hi @markbotterill @poornimaramesh,

I’m Aayush. I’m interested in this project because it combines backend engineering, statistics, experimentation systems, and public-good impact — the kind of practical systems work I enjoy.

To ground my proposal in working code, I built a small standalone bandit policy demo here:

https://github.com/aayushprsingh/idinsight-bandits-demo

What it includes:

• BanditPolicy interface with select_arm, update, and posterior_summary
• Thompson Sampling for Beta-Bernoulli rewards
• UCB1 policy
• replayable simulation harness
• unit tests for updates, exploration, reward validation, and learning behavior

Verification:

uv run python -m unittest discover -s tests
uv run python simulate.py

My proposed direction for Evidential would be to keep the allocation layer modular: policy classes own allocation/update logic, while the API/service layer handles assignment logging, reward ingestion, dashboard summaries, and later contextual features.

Architecture question: in the current Evidential backend, would you prefer bandit policies to be persisted as explicit policy state tables (posterior parameters per arm) or recomputed from event/reward history for auditability, with caching for performance?