LensLessFace

A Hybrid ResNet–Transformer Framework with Sub-Center ArcFace for Privacy-Preserving Face Recognition from Lensless Camera Measurements

Recognition and verification performed directly on coded sensor measurements —
no viewable face image is ever reconstructed.

---

Abstract

Lensless cameras replace the conventional optical lens with a thin coded amplitude mask placed directly above the image sensor. The resulting measurements are visually unintelligible yet informationally rich: identity is preserved, appearance is not. This optical encoding provides an intrinsic privacy guarantee — but it also makes downstream perception substantially harder than in the standard imaging pipeline, because the sensor no longer records a face, it records a globally-mixed projection of one.

This repository implements and documents a complete learning framework that performs identity recognition and face verification directly from FlatCam-style lensless measurements, without ever reconstructing a viewable image. Raw sensor frames are converted offline into a compact multi-resolution Discrete Cosine Transform (DCT) representation packed into a fifteen-channel tensor spanning five frequency subbands. A hybrid backbone then processes this tensor: an ImageNet-pretrained residual convolutional encoder extracts local feature maps, which are flattened into patch tokens and refined by a compact Transformer encoder that models long-range dependencies across the whole coded aperture. The resulting 768-dimensional embedding is supervised with a sub-center ArcFace objective (K = 3) and regularized with Mixup, label smoothing, and Stochastic Weight Averaging.

On a benchmark of 87 identities captured through a real lensless hardware prototype, the proposed model reaches 97.94 % Rank-1 accuracy, 99.88 % Top-5 accuracy, and an AUC of 0.9763 over a 10,092-pair verification protocol — an absolute improvement of nearly four points in Rank-1 over the five-branch convolutional baseline. Twenty-two logged experiments isolate the contribution of every design choice and expose a clear, reproducible recognition–verification trade-off induced by CutMix augmentation.

---

Table of Contents

Section	Contents
Highlights	What is new in this work
Results at a Glance	Headline numbers and evaluation figures
The Problem	Why lensless recognition is hard
Method	DCT front-end, hybrid backbone, ArcFace head
Architecture Evolution	The eight-phase research trajectory
Repository Layout	Where everything lives
Quickstart	Install, train, evaluate, plot
Documentation	The full docs set
Citation	How to cite this work
References	Prior art

---

Highlights

1 — A multi-resolution DCT front-end. Raw FlatCam measurements are decomposed into five frequency subbands (LL, LH, HL, HH, plus a half-resolution DC band) and stacked into a 15 × 64 × 64 tensor. This concentrates identity-relevant energy into a small number of coefficients and partially disentangles the spatial mixing imposed by the coded aperture — enabling deep feature learning without image reconstruction.

2 — A hybrid CNN–Transformer backbone. A modified ResNet-18 supplies the inductive bias that a small (~22 k sample) dataset demands; a 4-layer, 8-head pre-norm Transformer supplies the global receptive field that a globally mixed lensless measurement demands. Neither component reaches 97 % alone — a pure DCT-ViT collapses to ~77 %.

3 — Sub-center ArcFace tuned for coded-aperture noise. Three sub-centroids per identity (K = 3), a softened angular margin (m = 0.35) and scale (s = 40) absorb the intra-class variation that lensless capture injects. Tuning the margin down from the textbook m = 0.5, s = 64 was worth a consistent gain at every budget tested.

4 — Privacy by construction, not by policy. The pipeline never inverts the optical encoding. There is no point in the system — training, inference, storage, or matching — at which a viewable face exists. Privacy is a property of the optics and the representation, not of an access-control layer.

5 — A fully documented ablation ledger. Twenty-two experiments, each with its exact command line, are recorded in docs/EXPERIMENTS.md — including the runs that failed, the NaN divergence that motivated the eta_min floor, and the CutMix trade-off that exchanges 3.85 points of Rank-1 for 0.0206 AUC.

---

Results at a Glance

Headline model — t1_ls_swa

Metric	Value	Protocol
Rank-1 recognition accuracy	97.94 %	2,523 held-out measurements, 87 classes, 5-view TTA
Top-5 recognition accuracy	99.88 %	same
Verification AUC	0.9763	cosine similarity of ℓ₂-normalized embeddings
Pair-protocol AUC	0.9730	10,092 pairs (50 % genuine / 50 % impostor)
Equal error rate	4.22 %	derived from the committed pair scores
Operating threshold	0.8982	cosine similarity — 94.55 % TPR at 0.77 % FPR
Trainable parameters	≈ 24.5 M	ResNet-18 + Transformer + projection + ArcFace
Embedding dimension	768	ℓ₂-normalized

_{Configuration: ResNet-18 backbone · 4-layer Transformer · sub-center ArcFace (K = 3, m = 0.35, s = 40) · label smoothing ε = 0.1 · SWA over the final 25 % of epochs · 250 epochs · batch 64 · NVIDIA RTX A6000. Both AUC figures are reported; see docs/RESULTS.md for why they differ.}

Against the field

Model	Rank-1	Verification AUC	Top-5	Params
Five-branch VGG CNN + PALayer (baseline)	93.97 %	0.9784	—	~2.5 M
Pure DCT-ViT Transformer	~77 %	low	—	~4.5 M
Hybrid ResNet-18 + Transformer + ArcFace (v1)	94.40 %	0.9948	—	~23.8 M
+ sub-center ArcFace, 768-d, warm-up, freezing (v2)	94.33 %	0.9204	99.25 %	~24.5 M
+ tuned margin, 150 ep, RTX A6000	96.67 %	0.9699	99.68 %	~24.5 M
+ 200 ep, m = 0.35	97.62 %	0.9689	99.80 %	~24.5 M
+ label smoothing, 250 ep	97.50 %	0.9736	99.64 %	~24.5 M
+ SWA — t1_ls_swa 🏆	97.94 %	0.9763	99.88 %	~24.5 M
+ CutMix — t1_ls_swa_cutmix	94.09 %	0.9969 🥇	99.45 %	~24.5 M
ResNet-34 backbone — t2_resnet34	96.63 %	—	—	~34.5 M

Read the table this way. Rank-1 climbs monotonically once the training recipe is fixed (93.97 → 97.94), but AUC does not follow it. The single best verification model (t1_ls_swa_cutmix, AUC 0.9969) is nearly four points worse at Rank-1. Identification wants tight, well-separated class prototypes; verification wants a smooth, uniformly-populated embedding manifold. CutMix buys the second at the cost of the first. Pick your deployment target before you pick your checkpoint.

Evaluation figures

ROC curve	Score distribution
Genuine/impostor separation over the verification protocol	Cosine-similarity histograms around the 0.8982 threshold

Per-class accuracy	Verification confusion matrix
All 87 identities; worst class 86.21 %, 54 classes at 100 %	Operating-point decomposition

_{Summary dashboard — regenerate any of these from the committed metrics with python plot_results.py; no GPU, model weights, or dataset required.}

---

The Problem: Seeing Without a Lens

A conventional camera uses a lens to enforce a one-to-one map between scene points and sensor pixels. That map is what makes an image an image — and what makes every downstream vision model possible.

A lensless camera removes the lens and places a coded amplitude mask microns above the sensor. Every scene point now illuminates a large region of the sensor, and every sensor pixel integrates light from a large region of the scene. The measurement is a globally mixed linear projection of the scene, which for a separable FlatCam mask takes the form

M  =  Φ_L · S · Φ_Rᵀ  +  N

where S is the scene, Φ_L and Φ_R are the left/right coded-mask operators recovered by calibration, and N is sensor noise. To a human eye — and to an off-the-shelf face recognizer — M is texture. It looks like nothing.

This yields three consequences that shape every design decision in this repository:

Consequence	Why it follows	How we respond
Privacy is intrinsic	Recovering S requires the calibration matrices; a stolen measurement is not a stolen face	We never reconstruct. The model consumes M directly
Local features are insufficient	Identity information is smeared globally by Φ	A Transformer over the full token grid, not a CNN alone
Data is scarce and noisy	Hardware capture is expensive: 87 identities, ~22 k frames	Pretrained backbone, heavy regularization, sub-center margins

The task. Given raw lensless measurements: (i) identify which of 87 enrolled subjects produced a measurement, and (ii) verify whether two measurements originate from the same subject — both without inverting the optics.

---

Method

_{Figure 1. End-to-end pipeline, from photons on a coded mask to an identity decision.}

Stage 1 — Multi-resolution DCT decomposition

_{Figure 2. Offline preprocessing: Bayer demultiplexing → calibrated rotation and crop → 2-D DCT-II → five-subband partition → 15-channel tensor.}

The raw mosaic frame is demultiplexed into its four Bayer components (R, G_r, G_b, B), rotated by the calibrated sensor angle, center-cropped to the usable aperture, and resized to 128 × 128 × 3 (the two green planes are averaged). An orthonormal separable 2-D DCT-II is then applied per colour plane, and the coefficient matrix is partitioned into four 64 × 64 quadrants — LL, LH, HL, HH. A fifth DC band is computed independently by DCT-transforming a 64 × 64 down-sampled copy of the measurement, capturing coarse global structure that quadrant-splitting alone would fragment.

Stacking the three colour planes within each of the five subbands yields the final network input:

X  ∈  ℝ^(15 × 64 × 64)        # 5 subbands × 3 colour planes

Why the frequency domain? Because the coded-aperture convolution that destroys spatial locality becomes far more structured in frequency. The DCT does not undo the encoding, but it re-sorts it: energy smeared across the entire sensor plane collapses into a compact, learnable band structure.

→ Full derivation and the exact preprocessing commands: docs/DATASET.md

Stage 2 — Hybrid ResNet–Transformer backbone

_{Figure 3. Modified ResNet-18 spatial encoder, pre-norm Transformer refiner, projection head, and sub-center ArcFace classifier.}

X  (B, 15, 64, 64)
│
├─ bilinear resize ──────────────────────► (B, 15, 224, 224)
│
├─ ResNet-18, ImageNet-pretrained
│    conv1 replaced: 3×7×7 → 15×7×7
│    init W₁₅ = repeat(W₃, 5) / 5        ← preserves activation scale
│    layers 1–4 unchanged                ─► (B, 512, 7, 7)
│
├─ tokenize: flatten(2).transpose(1,2)   ─► (B, 49, 512)
├─ + learned positional embedding
│
├─ TransformerEncoder
│    depth 4 · heads 8 · FFN 2048
│    GELU · dropout 0.1 · norm_first     ─► (B, 49, 512)
│
├─ pooling: mean over tokens (or GeM)    ─► (B, 512)
├─ projection: 512 → BN → ReLU → 768     ─► (B, 768)
└─ ℓ₂-normalize                          ─► e on the unit hypersphere

Three details carry disproportionate weight:

• The conv1 surgery. ResNet-18 expects three channels; the DCT tensor has fifteen. Repeating the pretrained kernel five times and dividing by five keeps the expected activation magnitude at initialization identical to the pretrained network — so the ImageNet prior survives contact with data that looks nothing like ImageNet.
• Pre-normalization (norm_first=True). Post-norm Transformers are notoriously unstable at this depth-to-data ratio. Pre-norm made the difference between a training run and a NaN.
• Progressive freezing. conv1, bn1, layer1, layer2 are frozen for the first 10 epochs so the randomly-initialized Transformer can adapt to the convolutional features before large gradients reach — and destroy — the pretrained stem.

Token count is computed, never assumed. The model runs a dummy forward pass at construction to measure the true feature-map size. This matters: at --input_size 112 the correct grid is 4 × 4 (16 tokens), not the 3 × 3 that naive input_size // 32 arithmetic predicts.

→ Layer-by-layer specification: docs/ARCHITECTURE.md

Stage 3 — Sub-center ArcFace objective

Standard softmax cross-entropy optimizes separability; face recognition needs discriminability — embeddings that generalize to pairs never seen during training. ArcFace enforces this by imposing an additive angular margin on the unit hypersphere.

With K sub-centroids per identity, the class score is a max over sub-centroids,

s(i, c)  =  max     cos( e_i , w_{c,k} )
            k=1..K

and the margin m is applied only to the ground-truth column before rescaling by s:

                    exp( s · (cos θ_y + m) )
L  =  − log  ─────────────────────────────────────────────
             exp( s · (cos θ_y + m) ) + Σ  exp( s · cos θ_j )
                                        j≠y

Why sub-centers matter here specifically. A single prototype per identity assumes intra-class variation is unimodal. Lensless capture violates that assumption — pose, illumination, and calibration drift produce genuinely multi-modal clusters per subject. With K = 3, an outlying measurement can bind to its own sub-centroid instead of dragging the class prototype off-manifold. Empirically K = 3 maximizes Rank-1; K = 5 maximizes AUC and costs about one point of Rank-1.

The head is decorated with @torch.amp.custom_fwd(cast_inputs=torch.float32). This is not decoration — a plain .float() cast is silently overridden inside autocast, and the sqrt(1 − cos²) term produces NaNs in FP16 the moment cos exceeds 1 by one ULP.

Stage 4 — Inference: identification and verification from one embedding

_{Figure 4. A single encoder serves both protocols. Identification takes an argmax over ArcFace cosine scores; verification thresholds the cosine similarity between two embeddings.}

Five test-time views are embedded independently and averaged before re-normalization: the original input, its horizontal flip, a 3×3 Gaussian blur, a 90 % center crop resized back, and a +10 % brightness shift. TTA costs 5× inference compute and buys stability more than peak accuracy — it suppresses the epoch-to-epoch oscillation that raw single-view evaluation exhibits on this dataset.

---

Architecture Evolution: Eight Phases

This project is a research trajectory, not a single model. Each phase is a hypothesis, an experiment, and a verdict — including the ones that lost.

Phase	Hypothesis	Result	Verdict
1	A 5-branch VGG CNN with pixel attention can read DCT subbands	93.97 %	✅ Solid baseline; no cross-band interaction
2	A pure DCT-ViT will capture global structure the CNN misses	~77 %	❌ Failed — no inductive bias, no pretraining, 22 k samples
3	Pretrained CNN + Transformer + ArcFace beats both	94.40 %	✅ The core architecture
4	Sub-centers, 768-d, warm-up, freezing push past 96 %	94.33 %	⚠️ Mixed; NaN divergence exposed the eta_min bug
5	Tuned margins and longer training on real hardware help	96.67 %	✅ m = 0.4, s = 40, batch 64, 150 ep on A6000
6	200 epochs and a softer margin still have headroom	97.62 %	✅ m = 0.35 wins; the model was still learning at 150
7	Label smoothing + SWA convert a lucky peak into a stable floor	97.94 %	🏆 Best model. CutMix reveals the AUC trade-off
8	A deeper ResNet-34 backbone extracts richer features	96.63 %	❌ Failed — +10 M params overfit 22 k samples

Two negative results are worth as much as the positive ones. Phase 2 establishes that Transformers alone cannot substitute for convolutional priors at this data scale. Phase 8 establishes that the bottleneck is data, not capacity — adding parameters actively hurts.

→ Every run, every command, every verdict: docs/EXPERIMENTS.md

---

Repository Layout

LensLessFace/
│
├── models/
│   ├── transformer_model.py      HybridResNetTransformer — backbone, GeM, TTA, freezing
│   ├── arcface.py                ArcMarginProduct — sub-center ArcFace, AMP-safe
│   └── proposed_model.py         proposed_net — 5-branch VGG baseline with PALayer
│
├── fc_data_process/              Offline FlatCam → DCT preprocessing pipeline
│   ├── flatcam.py                Calibration, SVD, Bayer demultiplexing
│   ├── fc_utils.py               DCT subband decomposition and .npy export
│   ├── process_raw_fc_meas.py    Parallel driver over a raw capture tree
│   ├── prep_data_recog_complete.py       Train/test split by capture index
│   ├── train_test_split_classification.py
│   └── generate_verification_pairs.py    Verification protocol generator
│
├── train.py                      Training loop — Mixup, CutMix, SWA, warm-up, AMP, resume
├── test_face_recognition.py      Rank-1 / Top-5 evaluation with 5-view TTA
├── test_face_verification.py     Cosine-similarity verification, ROC, AUC, threshold
├── run_trinity.py                Orchestrator — single runs and four batch experiment tiers
├── generate_graphs.py            Full evaluation figures (requires weights + test data)
├── plot_results.py               Figures from committed JSON — no GPU, no data needed
├── my_data_class.py              Dataset loaders for .npy DCT tensors (+ noise variants)
├── utils.py                      Progress bar
│
├── data/
│   ├── verification_pairs.txt    The verification pair protocol
│   └── noise_locations/          Structured-occlusion masks for robustness studies
│
├── saved_models/t1_ls_swa/       Best-model metrics.json + results.json (weights not tracked)
├── assets/figures/               Publication figures (SVG + PNG)
├── assets/results/               Generated evaluation plots
├── paper/                        Elsevier CAS manuscript source (LaTeX + BibTeX)
│
└── docs/
    ├── ARCHITECTURE.md           Layer-by-layer model specification
    ├── DATASET.md                Capture protocol, DCT derivation, splits
    ├── TRAINING.md               Every flag, every schedule, every recipe
    ├── EXPERIMENTS.md            All runs with commands and verdicts
    ├── RESULTS.md                Metrics, figures, error analysis
    ├── REPRODUCIBILITY.md        Environment, seeds, hardware, known caveats
    ├── CODE_MAP.md               Module-by-module API reference
    └── FAQ.md                    Design questions, answered

---

Quickstart

1. Environment

git clone https://github.com/SatyamSingh-Git/LensLessFace.git
cd LensLessFace

python -m venv venv
source venv/bin/activate          # Windows: venv\Scripts\activate

pip install -r requirements.txt   # requirements_win.txt on Windows

_{PyTorch ≥ 1.10 with CUDA is strongly recommended. A full 250-epoch run takes roughly half a day on an RTX A6000; CPU training is possible but impractical.}

2. Reproduce the figures — no GPU, no dataset

The metrics of the best model are committed to this repository, so every evaluation figure in this README can be regenerated on any laptop in seconds:

python plot_results.py \
    --metrics_file saved_models/t1_ls_swa/metrics.json \
    --results_file saved_models/t1_ls_swa/results.json \
    --output_dir graphs/

3. Prepare the dataset

# Raw FlatCam captures → 15-channel DCT tensors
python fc_data_process/process_raw_fc_meas.py \
    --data_path  /path/to/fc_captures \
    --out_path   /path/to/lensless_data \
    --calib_file data/flatcam_calibdata.mat \
    --meas_size  128 \
    --num_of_cores -1

# Verification protocol
python fc_data_process/generate_verification_pairs.py \
    --data_path    /path/to/lensless_data/test/ymdct_npy \
    --output_file  data/verification_pairs.txt \
    --num_of_pairs 10092

→ Dataset access, directory conventions and the exact split: docs/DATASET.md

4. Train the best model

python train.py \
    --model transformer \
    --train_data /path/to/lensless_data/train/ymdct_npy \
    --test_data  /path/to/lensless_data/test/ymdct_npy \
    --batch_size 64  --num_epoch 250 \
    --arcface_m 0.35 --arcface_s 40 --arcface_k 3 \
    --label_smoothing 0.1 --use_swa \
    --seed 42 --eta_min 1e-6

Or reproduce the entire Tier-1 study — every run, sequentially, with checkpoints and dashboards:

python run_trinity.py --batch_tier1

5. Evaluate

# Rank-1 / Top-5, with 5-view TTA
python test_face_recognition.py \
    --model transformer \
    --test_data /path/to/lensless_data/test/ymdct_npy \
    --weights   logs/<run>/swa_best.pth \
    --batch_size 128

# Verification — ROC, AUC, operating threshold
python test_face_verification.py \
    --model transformer \
    --test_data /path/to/lensless_data/test/ymdct_npy \
    --pairs     data/verification_pairs.txt \
    --weights   logs/<run>/swa_best.pth

→ Full flag reference and training recipes: docs/TRAINING.md

---

Documentation

Document	What it answers
ARCHITECTURE.md	What is every layer, tensor shape, and parameter? Why pre-norm? Why 49 tokens?
DATASET.md	How was the data captured, calibrated, transformed, and split?
TRAINING.md	What does each flag do, and what recipe reproduces each result?
EXPERIMENTS.md	What was tried, what worked, what failed, and with which command?
RESULTS.md	What are the final numbers, and where does the model still fail?
REPRODUCIBILITY.md	What is fixed, what is stochastic, and what will not reproduce exactly?
CODE_MAP.md	Which module owns which responsibility, and what is its API?
FAQ.md	Why this design and not the obvious alternative?

The manuscript source (Elsevier CAS double-column format) lives in paper/.

---

Ethical Considerations

This work is biometric research. Three positions are worth stating explicitly.

On privacy. Lensless sensing raises the cost of casual visual surveillance — a stolen frame is not a stolen face — but it is not anonymization. The system is built to recognize people, and it does so at 97.94 % accuracy. Optical encoding protects against a passive observer of the sensor stream; it protects no one from the operator of the system.

On consent and data. The 87-identity dataset was captured through a research FlatCam prototype. Raw captures are not distributed in this repository. Anyone reproducing this work on new subjects should obtain informed consent covering biometric template storage, not merely image capture.

On deployment. These results come from a single hardware prototype, one capture methodology, and 87 subjects. Demographic performance parity has not been evaluated. Nothing here is validated for access control, law enforcement, or any consequential decision about a person.

---

Citation

If this work is useful in your research, please cite it:

@software{singh_lenslessface_2026,
  author  = {Singh, Satyam},
  title   = {{LensLessFace}: A Hybrid ResNet--Transformer Framework with
             Sub-Center ArcFace for Privacy-Preserving Face Recognition
             from Lensless Camera Measurements},
  year    = {2026},
  url     = {https://github.com/SatyamSingh-Git/LensLessFace},
  version = {1.0.0}
}

A machine-readable CITATION.cff is provided; GitHub renders it as a "Cite this repository" button in the sidebar.

---

References

1. Asif, Ayremlou, Sankaranarayanan, Veeraraghavan, Baraniuk. FlatCam: Thin, Lensless Cameras Using Coded Aperture and Computation. IEEE Transactions on Computational Imaging, 2017.
2. Deng, Guo, Xue, Zafeiriou. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. CVPR, 2019.
3. Deng, Guo, Liu, Gong, Zafeiriou. Sub-center ArcFace: Boosting Face Recognition by Large-Scale Noisy Web Faces. ECCV, 2020.
4. He, Zhang, Ren, Sun. Deep Residual Learning for Image Recognition. CVPR, 2016.
5. Dosovitskiy et al. An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. ICLR, 2021.
6. Zhang, Cisse, Dauphin, Lopez-Paz. mixup: Beyond Empirical Risk Minimization. ICLR, 2018.
7. Yun, Han, Oh, Chun, Choe, Yoo. CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. ICCV, 2019.
8. Izmailov, Podoprikhin, Garipov, Vetrov, Wilson. Averaging Weights Leads to Wider Optima and Better Generalization. UAI, 2018.
9. Loshchilov, Hutter. Decoupled Weight Decay Regularization. ICLR, 2019.
10. Loshchilov, Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR, 2017.

Full BibTeX: paper/lensless_refs.bib

---

Documentation · Experiments · Results · Manuscript

Released under the MIT License.

_{Research code. Interfaces may change between experiment tiers.}