PRB_Allocation_xApp_Dec2025

O-RAN ISAC PRB Allocation & Numerology Selection with Dueling Double DQN (D3QN)

An O-RAN near-RT xApp that jointly learns PRB allocation and numerology (subcarrier spacing) selection across Communication, Sensing, and ISAC (Integrated Sensing and Communication) traffic modes, for a 5G URLLC cell. A Dueling Double DQN (D3QN) agent is trained and benchmarked against a standard DQN agent and a fixed Static allocation baseline.

Team members

Sachin KS · Hirthik Raj A · Nitheshkumar S

---

Overview

Modern O-RAN deployments must support URLLC (Ultra-Reliable Low-Latency Communication) alongside ISAC traffic on a shared pool of Physical Resource Blocks (PRBs). Static, rule-based PRB splits cannot adapt to changing channel quality, buffer occupancy, or UE-mode mix, and they ignore the trade-off between numerology (subcarrier spacing) and latency/PRB granularity.

This project implements a near-RT RIC xApp that:

1. Observes per-slot cell state (channel quality, buffer load, UE-mode distribution, PRB utilization).
2. Selects a joint action — a PRB allocation strategy and a numerology level — every control step.
3. Optimizes a multi-objective reward that balances latency, reliability, PRB waste, and fairness.
4. Is benchmarked against a static equal-split baseline and a non-dueling, non-double DQN agent.

---

Problem Setup

A single gNB O-RAN cell serves 12 URLLC UEs, each dynamically assigned to one of three modes every step:

Mode	Assignment rule (per UE)
Comm	Buffer > 50 packets and CQI > 10
Sensing	Buffer < 10 packets
ISAC	Otherwise

Key challenge: allocate PRBs across the three modes using 25 discrete fractional strategies, while simultaneously choosing one of 3 numerology levels — jointly optimizing:

• Average per-UE latency (Shannon-rate based, target < 1 ms)
• Reliability (packet success rate)
• URLLC latency-violation rate
• PRB utilization (waste minimization)
• Jain's fairness index across UEs

---

Repository Structure

.
├── mainCode2.m          # Training/evaluation driver: Static vs DQN vs D3QN
├── MyEnvironment1.m      # Custom rl.env.MATLABEnvironment (state, dynamics, reward)
├── DRMxAppAgent1.m       # D3QN agent factory (network + rlDQNAgent options)
├── TrainedAgent_DQN.mat          # Saved trained DQN agent (generated after run)
├── TrainedAgent_D3QN.mat         # Saved trained D3QN agent (generated after run)
└── TrainingResults_Static_DQN_D3QN.mat  # All logged stats/metrics (generated after run)

---

Environment (MyEnvironment1.m)

Custom subclass of rl.env.MATLABEnvironment, NumGNB = 1, NumUE = 12, episode length MaxSteps = 200.

Observation space — 6-D, normalized to [0, 1]

[ MeanCQI_norm | MeanBuffer_norm | NumUE_Comm_frac | NumUE_Sensing_frac | NumUE_ISAC_frac | AllocPRBs_frac ]

Element	Definition
NormCQI	mean per-UE CQI (0–15) / 15
NormBuffer	mean per-UE buffer occupancy / MaxBufferSize (200)
NormComm / NormSens / NormISAC	fraction of UEs currently in each mode
NormAllocPRBs	total allocated PRBs / TotalPRBs for the current numerology

Action space — 75 discrete joint actions

Action ∈ {0, 1, ..., 74}
  strategyIdx   = mod(Action, 25) + 1      → 1 of 25 PRB allocation strategies
  numerologyIdx = floor(Action / 25)       → 1 of 3 numerology levels

25 allocation strategies — [fComm, fSensing, fISAC] fraction triplets built from levels {0, 0.33, 0.50, 0.67, 1.0} (fISAC derived as the residual, normalized if over-committed).

3 numerology levels (adaptive subcarrier spacing, changes TotalPRBs each step):

μ	SCS	TotalPRBs
0	15 kHz	106
1	30 kHz	51
2	60 kHz	24

PRB → per-UE distribution

• Comm UEs: weighted by CQI (better channel → more PRBs)
• Sensing UEs: equal share
• ISAC UEs: 60% of ISAC PRBs used for data, weighted by CQI

Latency model (Shannon capacity, per UE)

Latency_i [s] = NumBits_i / (BW_i · log2(1 + SNR_i))
BW_i  [Hz]    = PRBs_PerUE_i × SCS × 12        (12 subcarriers/PRB, SCS adaptive per numerology)
SNR_i [lin]   = 10^(SNR_dB_i / 10)             (SNR_dB ~ Uniform[-5, 25] dB per step)

A UE is counted as a latency violation if Latency_i > LatencyTarget (1 ms).

Reward function

R = -α · Latency_norm  -  β · Latency_penalty  -  γ · Waste_penalty  +  δ · Fairness_index

α = 1.5   (direct latency cost)
β = 2.0   (URLLC violation penalty)
γ = 0.5   (unused-PRB waste penalty)
δ = 0.8   (Jain fairness bonus)

Term	Definition	Range
Latency_norm	mean per-UE latency / LatencyRef (5 ms), clipped	[0, 1]
Latency_penalty	fraction of UEs violating the 1 ms target	[0, 1]
Waste_penalty	unused PRBs / TotalPRBs	[0, 1]
Fairness_index	Jain's fairness index on per-UE PRB allocation	[0, 1]

---

D3QN Agent (DRMxAppAgent1.m)

Dueling Double DQN, built for O-RAN xApp inference speed (BatchNorm-free, latency-optimized).

Shared path      : FC(256) → ReLU → FC(256) → ReLU → FC(128) → ReLU
Value stream     : FC(64)  → ReLU → FC(1)                         V(s)
Advantage stream : FC(64)  → ReLU → FC(75)                        A(s,a)
Aggregation      : Q(s,a) = V(s) + [ A(s,a) − mean_a A(s,a) ]

Design choices vs. a naïve DQN

• No BatchNormalization — removed as a statefulness/throughput bottleneck for a 6-D input in an RL training loop.
• Dueling architecture — separates state-value estimation from per-action advantage, improving learning stability when many actions share similar value.
• Double DQN (UseDoubleDQN = true) — decouples action selection from evaluation to reduce Q-value overestimation bias.

Hyperparameters

Parameter	Value
Learn rate	2e-4
Gradient threshold	1
L2 regularization	1e-5
Discount factor (γ)	0.997
Replay buffer	50,000 transitions
Mini-batch size	128
Target smoothing factor (τ)	5e-4
Target update frequency	every 4 steps
ε-greedy	1.0 → 0.02, decay = 0.995

---

Baseline & Comparison Schemes (mainCode2.m)

Scheme	Description
Static	Fixed equal-split (33/33/34% Comm/Sensing/ISAC), always μ = 0 (106 PRBs @ 15 kHz). No learning. Encoded action = 12.
DQN	Flat network 256-256-128, LR = 2e-3, buffer = 30k, batch = 64, γ = 0.97, TargetUpdateFrequency = 8, Double DQN off.
D3QN	Dueling Double DQN via DRMxAppAgent1 — see hyperparameters above.

Training configuration

• MaxEpisodes = 500, MaxStepsPerEpisode = 200
• Score-averaging window = 20 episodes
• Checkpoint saved when EpisodeReward > -1

Evaluation

Each scheme is evaluated over 50 episodes (greedy / fixed policy, no exploration), logging: average latency, reliability (packet success rate), violation rate, episode reward, PRB utilization, Jain fairness, and numerology usage distribution.

---

Outputs

Running mainCode2.m produces:

1. Training reward curves (DQN & D3QN, moving-avg-20) vs. Static evaluation mean, plus an evaluation reward box-plot
2. URLLC key-metrics bar chart (mean ± std): latency, reliability, violation rate
3. Per-episode latency & violation-rate line plots
4. Reliability CDF
5. Latency vs. number of RBs — ISAC vs. non-ISAC sweep
6. Numerology selection distribution — DQN vs. D3QN (Static is fixed at μ = 0)
7. Comprehensive 2×3 URLLC summary dashboard (histograms, per-episode trends, numeric summary table)

Saved artifacts:

• TrainedAgent_DQN.mat
• TrainedAgent_D3QN.mat
• TrainingResults_Static_DQN_D3QN.mat (all stats, metrics, and RB-sweep data)

---

Requirements

• MATLAB R2023a or later
• Reinforcement Learning Toolbox
• Deep Learning Toolbox

Usage

% From the repository root
mainCode2

This will initialize the environment, build and train the DQN and D3QN agents, evaluate all three schemes (Static/DQN/D3QN), generate all plots, and save the trained agents and results to disk.