A Curated Arsenal of Advanced Deep Learning Techniques

In the rapidly evolving landscape of AI research, groundbreaking techniques are often scattered across countless repositories, disparate implementations, and dense academic papers. dl_techniques emerges as a unified, curated, and production-ready arsenal for advanced deep learning. Pioneered and sponsored by Electi Consulting, this library is more than a collection of components—it is the definitive toolkit for researchers and engineers to design, train, and dissect state-of-the-art neural networks.

We bridge the chasm between theoretical innovation and practical application, providing faithful, efficient, and enterprise-validated components. From next-generation attention mechanisms and graph neural networks to information-theoretic loss functions and an unparalleled model analysis suite, dl_techniques is your strategic advantage for pushing the boundaries of deep learning.

Key Features: A Glimpse into the Arsenal.
Why dl_techniques?: Your Unfair Advantage in AI R&D.
Installation: Get Up and Running in Minutes.
Quick Start: See the Library in Action.
In-Depth Documentation: From Theory to Tensors.
Project Structure: A Tour of the Repository.
Contributing: Join Our Research & Development Efforts.
License: Understanding the GPL-3.0 License.
Acknowledgments: Recognition and Support.
Citations & References: The Research That Inspired This Library.

Key Features

This library is a comprehensive suite of tools organized into five key pillars, developed through rigorous research and validated in real-world enterprise applications:

1. State-of-the-Art Architectures & Models (70+ Architectures)

Next-Generation Language Models: Production-ready implementations of Gemma 3, Qwen3, Mamba, ModernBERT, DistilBERT and GPT-2, alongside the Byte Latent Transformer (BLT) and the Hierarchical Reasoning Model (HRM).
Vision & Multimodal Powerhouses: A comprehensive suite including CLIP, MobileCLIP and MobileCLIP2 (on a faithful FastViT image tower), FastVLM, NanoVLM (including a score-based World Model), DINOv1/v2/v3, SigLIP-ViT, BEiT, Swin Transformer, Video-JEPA, object detection with DETR and YOLOv12, keypoints with SuperPoint, and Segment Anything (SAM 1, 2 and 3).
Advanced CNNs: A rich collection of advanced convolutional architectures such as ConvNeXtV1/V2, ConvUNeXt, MobileNetV1-V4, ResNet, the recursively-defined FractalNet, the complex shearlet-based CoShNet, and the ultra-efficient SqueezeNet family.
Time Series & Forecasting: State-of-the-art forecasting models including the probabilistic TiRex with quantile prediction, an enhanced implementation of N-BEATS, autoregressive DeepAR, PRISM, and the novel xLSTM.
Generative Modeling & Image Restoration: Diffusion and flow-matching transformers (SD3 MMDiT, Ideogram4), a complete Variational Autoencoder (VAE) framework with VQ-VAE variants, arbitrary-scale super-resolution (THERA, PFT-SR), and restoration/denoising backbones (DarkIR, SCUNet, ACC-UNet, and a family of bias-free denoisers).
Specialized Models: Task-specific models like DepthAnything for monocular depth estimation, full Capsule Networks (CapsNet) with dynamic routing, TabM for tabular data, and unsupervised aligners like Mini-Vec2Vec.
Experimental Frontiers: Explore novel concepts including a wide range of Graph Neural Networks (GNNs) in RELGT, the Energy Transformer family (image and graph domains), geometric-algebra networks in CliffordNet, Kolmogorov-Arnold Networks (KAN), Neural Turing Machines (NTM), and bio-mimetic models like MothNet.

2. A Modular Arsenal of Advanced Layers (290+ Components)

Pioneering Attention Mechanisms: Go beyond standard attention with DifferentialMultiHeadAttention, modern HopfieldAttention, GroupQueryAttention, CapsuleRoutingAttention, and efficient alternatives like FNetFourierTransform and RingAttention.
Unified Factory Architecture: A consistent, powerful factory system for creating and validating 32 attention mechanisms, 18 normalization variants (including BandRMS, LogitNorm), and 21 Feed-Forward Network (FFN) types (SwiGLU, GeGLU, OrthoGLU) with a single line of code — plus registries for 22 activations, 13 embeddings, and task heads dispatched by domain via create_head.
Graph & Structural Primitives: Configurable GNN layers with multiple aggregation strategies (GCN, GAT, GraphSAGE), Relational Graph Transformer (RELGT) blocks, and Entity-Graph Refinement for learning hierarchical relationships.
Mixture of Experts (MoE) System: A complete MoE implementation with configurable FFN experts, multiple gating strategies (including SoftMoE and Cosine Gating), and integrated training utilities.
Probabilistic & Statistical Layers: Build models that reason about uncertainty with Mixture Density Networks (MDN), Normalizing Flows, and time series analysis layers for residual autocorrelation (ResidualACFLayer).
Advanced Embeddings: A full suite of positional and semantic embeddings including standard, sinusoidal, Rotary Position Embedding (RoPE), and its modern variants like DualRotaryPositionEmbedding.

3. A Unified Command Center for Model Analysis & Introspection

Holistic Model Analysis: A powerful ModelAnalyzer to benchmark models across five critical dimensions: training dynamics, weight health, prediction calibration, information flow, and advanced spectral analysis. Its modular design includes specialized analyzers like CalibrationAnalyzer, WeightAnalyzer, and InformationFlowAnalyzer.
Publication-Ready Visualizations: Automatically generate insightful visualizations, interactive summary dashboards, and comparative analysis plots with integrated statistical significance testing.
Predictive Generalization with Spectral Analysis: Integrate the power of WeightWatcher through our SpectralAnalyzer to assess generalization potential by analyzing the spectral properties (eigenvalues) of weight matrices—often without needing test data.
Deep Diagnostic Toolkit: Move beyond accuracy to diagnose overconfidence (CalibrationAnalyzer), information bottlenecks (InformationFlowAnalyzer), weight decay and similarity (WeightAnalyzer), and learning efficiency (TrainingDynamicsAnalyzer).

4. Next-Generation Loss Functions & Optimization (40+ Specialized Losses)

Optimize What Matters with AnyLoss: A groundbreaking framework that transforms any confusion-matrix-based metric (e.g., F1-score, Balanced Accuracy, Matthews Correlation Coefficient) into a differentiable loss function for direct optimization on imbalanced data.
Information-Theoretic & Robust Losses: Train more robust models with GoodhartAwareLoss to combat spurious correlations, calibration-focused losses like BrierScoreLoss, the uncertainty-aware FocalUncertaintyLoss, and DINO's self-distillation loss.
Domain-Specific Loss Functions: Specialized losses for vision-language (CLIPContrastiveLoss, SigLIPLoss), segmentation (Dice, Focal, Tversky), time series (MASELoss, SMAPELoss), and generative modeling (WassersteinLoss with gradient penalty).
Advanced Optimization Suite: Leverage smart learning rate schedulers like WarmupSchedule, utilities for DeepSupervision in multi-scale architectures, and a suite of advanced regularizers (SoftOrthogonal, SRIP).

5. Enterprise-Grade Training & Deployment Infrastructure

Accelerated Development with Training Pipelines: 70 ready-to-use train_*.py entry points across 48 pipeline directories (src/train/), establishing standardized and reproducible workflows for training, validation, and testing across domains like NLP, Vision, and Time Series.
Production-Ready Utilities: A suite of tools including advanced data loaders, augmentation pipelines, a structured visualization and logging manager (VisualizationManager), and enhanced model serialization with custom object support.
Assured Reliability: An extensive 600+ module test suite (tests/) ensures the correctness and stability of every component, with dedicated fixtures for mixed-precision and TF32-sensitive regressions.
Verified Against the Source: Where a reference implementation exists, ports are checked against it numerically — several packages commit the reference itself as an executable oracle (e.g. src/dl_techniques/layers/fastvit/reference.py) rather than asserting parity in prose.

Why `dl_techniques`?

From Theory to Tensors, Instantly: We consolidate the fragmented landscape of AI research. Instead of hunting down dozens of disparate GitHub repos, you get a single, cohesive framework with faithful implementations of cutting-edge research.
Built for Battle, Validated in the Enterprise: This is not a toy library. Every component has been hardened and validated in demanding enterprise applications, ensuring robustness, efficiency, and production-readiness.
Unprecedented Introspection: Move beyond accuracy scores. Our first-class analysis toolkit is designed to answer the why behind your model's behavior, providing deep insights that are critical for building trustworthy and reliable AI.
Engineered for Experimentation: Our innovative factory patterns and modular design are built for rapid prototyping. Swap attention mechanisms, normalization layers, or even entire architectural blocks with a single line of code.
Modern, Maintainable, and Future-Proof: Built from the ground up for Keras 3 and modern Python, dl_techniques adheres to the highest standards of software engineering, ensuring it's easy to use, extend, and maintain.

Installation

Note: This library requires Python 3.11+ and Keras 3.8.0 with the TensorFlow 2.18.0 backend.

Clone the repository:

git clone https://github.com/nikolasmarkou/dl_techniques.git
cd dl_techniques

Install dependencies: For standard usage, install the library and its core dependencies:
```
pip install .
```
Editable Install (for developers): If you plan to contribute or modify the library's source code, install it in editable mode with development tools:
```
pip install -e ".[dev]"
```
This installs additional tools such as pytest, pytest-cov, pylint, and pre-commit.
Optional Extras: The Streamlit applications under src/applications/ need an extra that is not part of a core install:
```
pip install ".[apps]"
```

Verify Installation:

python -c "import dl_techniques; print('Installation successful!')"

No trained weights ship with this repository. Training outputs live under results/, which is gitignored, so a fresh clone contains source, tests and prose only. Any results/<run>/final_model.keras path quoted in a trainer default, a paper or an application is a local artifact — you train it yourself first. See REPO_MAP.md § What a fresh clone actually contains.

Quick Start

1. Compose a State-of-the-Art Transformer Block

Effortlessly construct and experiment with modern transformer components using our unified factory system.

import keras
from dl_techniques.layers.attention.factory import create_attention_layer
from dl_techniques.layers.norms.factory import create_normalization_layer
from dl_techniques.layers.ffn.factory import create_ffn_layer

inputs = keras.Input(shape=(1024, 512))

# Use factories for consistent, validated component creation
attention = create_attention_layer(
    'differential',
    dim=512,
    num_heads=8,
    head_dim=64
)
norm = create_normalization_layer('rms_norm', epsilon=1e-6)
ffn = create_ffn_layer('swiglu', output_dim=512)

# Build a modern transformer block
x = attention(inputs)
x = norm(x)
x = ffn(x)

model = keras.Model(inputs, x)
model.summary()

2. Deploy a Probabilistic Time Series Forecaster

Instantiate a state-of-the-art univariate time series model capable of generating robust, uncertainty-aware forecasts.

from dl_techniques.models.time_series.tirex.model import create_tirex_model
from dl_techniques.losses.quantile_loss import QuantileLoss

# TiRex: probabilistic univariate forecasting with quantile prediction
model = create_tirex_model(
    input_length=100,            # length of the input context window
    prediction_length=24,        # forecast horizon
    quantile_levels=[0.1, 0.5, 0.9],  # 80% prediction interval + median
)

# Train directly for calibrated uncertainty with a quantile loss.
# Predictions are (batch, prediction_length, n_quantiles), so a plain
# point-forecast metric like 'mae' would broadcast across the quantile axis.
model.compile(
    optimizer='adamw',
    loss=QuantileLoss(quantiles=[0.1, 0.5, 0.9]),
)

3. Contrastive Vision-Language Learning with CLIP

Map images and text into a single shared embedding space with a CLIP dual encoder, unlocking zero-shot classification and cross-modal retrieval.

import keras
from dl_techniques.models.clip.model import create_clip_variant

# CLIP ViT-B/32: a dual encoder mapping images and text into one space
model = create_clip_variant("ViT-B/32")
model.build({"image": (None, 224, 224, 3), "text": (None, 77)})

images = keras.random.normal((4, 224, 224, 3))
tokens = keras.ops.cast(keras.random.uniform((4, 77), 0, 49408), "int32")

# Full contrastive forward pass
outputs = model({"image": images, "text": tokens}, training=False)
# outputs["image_features"]   -> (4, 512), L2-normalised
# outputs["text_features"]    -> (4, 512), L2-normalised
# outputs["logits_per_image"] -> (4, 4),  similarity matrix

# Or encode each modality independently for retrieval
image_features = model.encode_image(images)
text_features = model.encode_text(tokens)

4. Dissect Model Behavior with the Analysis Engine

Go beyond surface-level metrics and gain deep, actionable insights into your models' performance and behavior.

from dl_techniques.analyzer import ModelAnalyzer, AnalysisConfig, DataInput

# Compare multiple models across a suite of deep diagnostics
models = {'TiRex_Model': tirex_model, 'Baseline_LSTM': lstm_model}
histories = {'TiRex_Model': tirex_history, 'Baseline_LSTM': lstm_history}
test_data = DataInput(x_test, y_test)

# Configure a comprehensive analysis run
config = AnalysisConfig(
    analyze_training_dynamics=True,
    analyze_calibration=True,
    analyze_weights=True,
    analyze_spectral=True, # Unleash spectral analysis for generalization insights
    save_plots=True,
    plot_style='publication'
)

analyzer = ModelAnalyzer(models, config=config, training_history=histories)

# Execute the complete analysis and generate a full suite of visualizations
results = analyzer.analyze(test_data)

# Access detailed, structured metrics programmatically for automated reporting
print(f"Best calibrated model (by ECE): {min(results.calibration_metrics.items(), key=lambda x: x[1]['ece'])}")
print(f"Training efficiency ranking (epochs to converge): {results.training_metrics.epochs_to_convergence}")

5. Optimize Directly for F1-Score on Imbalanced Data

Stop tuning class weights and start optimizing your target metric directly with the AnyLoss framework.

from dl_techniques.losses.any_loss import F1Loss, BalancedAccuracyLoss

# For imbalanced datasets, optimize F1-score directly
model.compile(
    optimizer='adamw',
    loss=F1Loss(from_logits=True),
    metrics=['accuracy', 'precision', 'recall']
)

# Alternatively, optimize for balanced accuracy
model.compile(
    optimizer='adamw',
    loss=BalancedAccuracyLoss(from_logits=True),
    metrics=['accuracy']
)

6. Harness Graph Neural Networks for Relational Data

Unlock insights from graph-structured data with our powerful and configurable GNN implementations.

import keras
from dl_techniques.layers.graphs.graph_neural_network import GraphNeuralNetworkLayer

# Create a Graph Attention Network (GAT) to process relational data
gnn = GraphNeuralNetworkLayer(
    concept_dim=256,
    num_layers=3,
    message_passing='gat',  # Use attention for message passing
    aggregation='attention',
    dropout_rate=0.1
)

# Apply the layer to graph-structured inputs
node_features = keras.Input(shape=(None, 256))  # Variable number of nodes
adjacency_matrix = keras.Input(shape=(None, None))

node_embeddings = gnn([node_features, adjacency_matrix])

In-Depth Documentation

This library is engineered to be a living knowledge base, bridging the gap between academia and industry. Our documentation is split into three primary resources: a navigational map of the repository, in-depth theoretical guides, and a comprehensive API reference.

Start Here: `REPO_MAP.md`

Before anything else, read REPO_MAP.md — a path-verified router for the repository. It answers where code of a given kind lives, how the registry and factory dispatch is wired, which trainer trains which model, and which of the many in-tree docs answers your question. Every number in it sits beside the command that re-derives it, and it carries a ledger of claims the repo's own older docs get wrong.

Tutorials & Deep Dives (`research/`)

Our research/ directory contains over 120 articles providing the theoretical foundations, implementation details, and best practices behind key components. Highlights include:

Complete Transformer Guide (2025): A production-focused guide to implementing state-of-the-art Transformer architectures.
Model Analyzer Guide: A comprehensive tutorial for the advanced model analysis toolkit.
AnyLoss Framework: A deep dive into the theory and practice of direct metric optimization.
Chronological Neural Architectures: An extensive chronological guide to influential architectures with implementation notes.
Band-Constrained Normalization: A novel normalization technique that preserves magnitude information within bounded constraints.
Mixture Density Networks: The theory and best practices for implementing probabilistic models.

Papers (`research/papers/`)

Five LaTeX manuscripts written against this codebase live under research/papers/, several with built PDFs: band_rms (band-constrained RMS normalization), bfunet (bias-free denoisers as image priors), cliffordnet_extensions, correlations, and logical_net.

API Reference (per-module docs)

For detailed documentation on every module, class, and function, browse the source tree directly: each subpackage ships a focused README.md (e.g. src/dl_techniques/analyzer/README.md) and a per-package CLAUDE.md describing its conventions, patterns, and components. The research/ guides above complement these with the underlying theory.

Project Structure

The repository is organized for clarity, maintainability, and ease of contribution.

src/ is the import root: the library is imported as dl_techniques.*, the training pipelines as train.*, and the applications as applications.*.

dl_techniques/
├── src/dl_techniques/         # THE LIBRARY — 13 subpackages
│   ├── models/                # 70+ complete, ready-to-train model implementations
│   │   ├── qwen/              # Qwen3 (Base, Next, Embeddings)
│   │   ├── dino/              # DINO v1, v2, v3
│   │   ├── SAM/               # Segment Anything v1, v2, v3
│   │   ├── time_series/       # 7 nested forecasting models (TiRex, N-BEATS, xLSTM, ...)
│   │   └── ...
│   ├── layers/                # 290+ modules across 21 themed subpackages
│   │   ├── attention/         # 32 modern attention mechanisms with factory
│   │   ├── norms/             # 18 advanced normalization layers with factory
│   │   ├── ffn/               # 21 feed-forward networks with factory
│   │   ├── embedding/         # 13 positional/semantic embeddings with factory
│   │   ├── heads/             # Task heads dispatched by domain (nlp/vision/vlm)
│   │   ├── graphs/            # Graph neural network components
│   │   ├── moe/               # Mixture of Experts (MoE) system
│   │   ├── statistics/        # Statistical and probabilistic layers (MDN, Flows)
│   │   └── ...
│   ├── losses/                # 40+ specialized loss modules (AnyLoss, Goodhart, etc.)
│   ├── metrics/               # Custom Keras metrics (PSNR, SSIM, perplexity, Brier)
│   ├── optimization/          # Optimizers (Muon, VSGD, SGLD), LR schedules
│   ├── analyzer/              # Comprehensive model analysis toolkit and visualizers
│   ├── visualization/         # Plotting helpers for training and evaluation
│   ├── datasets/              # Dataset loaders and synthetic generators
│   ├── callbacks/             # Reusable Keras callbacks
│   ├── regularizers/          # Advanced regularization techniques (SRIP, Orthogonal)
│   ├── initializers/          # Structured initializers (Gabor, Haar, orthonormal, KAN)
│   ├── constraints/           # Weight constraints
│   └── utils/                 # Core utilities, loggers, masking, and data handlers
├── src/train/                 # 70 training entry points across 48 pipeline directories
├── src/applications/          # Deployable applications built on the library
├── research/                  # 120+ in-depth articles, guides, and LaTeX papers
├── tests/                     # 600+ test modules ensuring component reliability
└── REPO_MAP.md                # Path-verified router — read this first

Contributing

We welcome contributions from the research community. Whether you are implementing a new technique, improving documentation, or fixing bugs, your input is valuable.

Getting Started

Fork & Clone the repository.
Set up the development environment: pip install -e ".[dev]".
Create a new branch for your feature: git checkout -b feature/new-technique.
Adhere to our development standards: Use type hints, write comprehensive tests, and document your code thoroughly.

Development Standards

Code Quality: Follow PEP 8, use type hints, and rely on centralized logging via dl_techniques.utils.logger (no print).
Testing: Develop in the .venv environment and write comprehensive tests using pytest, scoped to the modules you change (make test runs the full ~1.5h suite). Set MPLBACKEND=Agg when running training scripts on headless machines.
Documentation: Document every public symbol, and update relevant guides in the research/ directory. Docstring style is not uniform across this repo — layers/ is predominantly Sphinx/reST (:param:), models/ is measurably mixed with the Sphinx exemplar models/bert/bert.py as the model for new packages, and losses//metrics//utils//optimization//analyzer//visualization/ are Google-Args:-majority. Match the package you are editing and never convert a file wholesale; the measured counts and the greps that re-derive them are in src/dl_techniques/CLAUDE.md § Core Conventions → Code Style.
Validation: Include benchmarks or comparisons against reference implementations where applicable.

Contribution Types

New Architectures: Implementations of recent, impactful research papers.
Performance Improvements: Optimizations that maintain numerical accuracy.
New Analysis Tools: Additional analyzers or visualizations for the ModelAnalyzer toolkit.
Enhanced Documentation: Tutorials, guides, or improved API documentation.

License

This project is licensed under the GNU General Public License v3.0.

Important Considerations:

Copyleft License: Any derivative works must also be licensed under GPL-3.0.
Enterprise Use: Please contact us for commercial licensing options that may be better suited for enterprise environments.
Research Use: The library is fully open for academic and non-commercial research applications.

See the LICENSE file for complete details.

Acknowledgments

This library is proudly sponsored and pioneered by Electi Consulting, a premier AI consultancy specializing in enterprise artificial intelligence, blockchain technology, and cryptographic solutions. The practical validation and enterprise-ready nature of these components has been made possible through Electi's extensive experience deploying state-of-the-art AI solutions across diverse industries:

Financial Services: High-frequency trading, risk assessment, and fraud detection.
Maritime Industry: Route optimization, predictive maintenance, and cargo management.
Healthcare: Diagnostic assistance, treatment optimization, and clinical decision support.
Manufacturing: Predictive maintenance, quality control, and supply chain optimization.

Special recognition is extended to the open-source community and the many researchers whose groundbreaking work forms the foundation of this library.

Citations & References

This library stands on the shoulders of giants. Our implementations are grounded in a rigorous study of the source papers that have defined the field of modern deep learning.

Transformers & Language Models

Attention Is All You Need (Transformer): Vaswani, A., et al. (2017). NeurIPS.
BERT: Pre-training of Deep Bidirectional Transformers: Devlin, J., et al. (2018). NAACL.
RoFormer: Enhanced Transformer with Rotary Position Embedding: Su, J., et al. (2021). ACL.
Differential Transformer: Ye, T., et al. (2024). ICLR 2025.
Mamba: Linear-Time Sequence Modeling: Gu, A., & Dao, T. (2023).
Gemma 3 Technical Report: Google DeepMind (2025).
Qwen Technical Report: Bai, J., et al. (2023).
Byte Latent Transformer: Patches Scale Better Than Tokens: Pagnoni, A., et al. (2024).
ModernBERT: Smarter, Better, Faster, Longer: Warner, B., et al. (2024).
DistilBERT, a distilled version of BERT: Sanh, V., et al. (2019).

Vision & Multimodal Models

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT): Dosovitskiy, A., et al. (2020). ICLR.
A ConvNet for the 2020s (ConvNeXt): Liu, Z., et al. (2022). CVPR.
ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders: Woo, S., et al. (2023). CVPR.
MobileNetV2: Inverted Residuals and Linear Bottlenecks: Sandler, M., et al. (2018). CVPR.
Searching for MobileNetV3: Howard, A., et al. (2019). ICCV.
MobileNetV4: Universal Inverted Bottleneck and Mobile MQA: Li, Y., et al. (2024).
DINOv2: Learning Robust Visual Features without Supervision: Oquab, M., et al. (2023).
Sigmoid Loss for Language Image Pre-Training (SigLIP): Zhai, X., et al. (2023). ICCV.
Learning Transferable Visual Models From Natural Language Supervision (CLIP): Radford, A., et al. (2021). ICML.
End-to-End Object Detection with Transformers (DETR): Carion, N., et al. (2020). ECCV.
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization: Vasu, P. K. A., et al. (2023).
BEiT: BERT Pre-Training of Image Transformers: Bao, H., et al. (2021). ICLR.
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows: Liu, Z., et al. (2021). ICCV.
MobileCLIP2: Improving Multi-Modal Reinforced Training: Faghri, F., et al. (2025).
Segment Anything: Kirillov, A., et al. (2023). ICCV.
SAM 2: Segment Anything in Images and Videos: Ravi, N., et al. (2024).
SuperPoint: Self-Supervised Interest Point Detection and Description: DeTone, D., et al. (2018). CVPR Workshops.
V-JEPA: Revisiting Feature Prediction for Learning Visual Representations from Video: Bardes, A., et al. (2024).
Masked Autoencoders Are Scalable Vision Learners (MAE): He, K., et al. (2022). CVPR.

Generative Modeling & Image Restoration

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (SD3 / MMDiT): Esser, P., et al. (2024). ICML.
Flow Matching for Generative Modeling: Lipman, Y., et al. (2023). ICLR.
Auto-Encoding Variational Bayes (VAE): Kingma, D. P., & Welling, M. (2013).
Neural Discrete Representation Learning (VQ-VAE): van den Oord, A., et al. (2017). NeurIPS.
Robust and Interpretable Blind Image Denoising via Bias-Free Convolutional Neural Networks: Mohan, S., et al. (2020). ICLR.
Thera: Aliasing-Free Arbitrary-Scale Super-Resolution with Neural Heat Fields: Becker, A., et al. (2025). CVPR.
DarkIR: Robust Low-Light Image Restoration: Feijoo, D., et al. (2025). CVPR.

Advanced Attention & FFN Mechanisms

Modern Hopfield Networks is All You Need: Ramsauer, H., et al. (2020). ICML.
GQA: Training Generalized Multi-Query Transformer Models: Ainslie, J., et al. (2023).
Ring Attention with Blockwise Transformers for Near-Infinite Context: Liu, H., et al. (2024).
Rethinking Attention with Performers: Choromanski, K., et al. (2020).
FNet: Mixing Tokens with Fourier Transforms: Lee-Thorp, J., et al. (2021).
GLU Variants Improve Transformer: Shazeer, N. (2020).
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer: Shazeer, N., et al. (2017).
Pay Attention to MLPs (gMLP): Liu, H., et al. (2021). NeurIPS.

Graph Neural Networks & Special Architectures

Graph Neural Networks: A Review: Wu, Z., et al. (2020).
Graph Attention Networks: Veličković, P., et al. (2018). ICLR.
Semi-Supervised Classification with Graph Convolutional Networks: Kipf, T. N., & Welling, M. (2016).
Dynamic Routing Between Capsules: Sabour, S., et al. (2017). NeurIPS.
KAN: Kolmogorov-Arnold Networks: Liu, Z., et al. (2024).
FractalNet: Ultra-Deep Neural Networks without Residuals: Larsson, G., et al. (2016).
Energy Transformer: Hoover, B., et al. (2023). NeurIPS.
Neural Turing Machines: Graves, A., et al. (2014).
Geometric Clifford Algebra Networks: Ruhe, D., et al. (2023). ICML.

Time Series & Forecasting

N-BEATS: Neural basis expansion analysis for interpretable time series forecasting: Oreshkin, B. N., et al. (2019). ICLR.
Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting: Lim, B., et al. (2021).
DeepAR: Probabilistic forecasting with autoregressive recurrent networks: Salinas, D., et al. (2020).
xLSTM: Extended Long Short-Term Memory: Beck, M., et al. (2024). NeurIPS.
Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift: Kim, T., et al. (2021). ICLR.

Loss Functions, Optimization, & Regularization

AnyLoss: A General and Differentiable Framework for Classification Metric Optimization: Han, D., et al. (2024).
Focal Loss for Dense Object Detection: Lin, T. Y., et al. (2017). ICCV.
On calibration of modern neural networks: Guo, C., et al. (2017). ICML.
Wasserstein GAN: Arjovsky, M., et al. (2017). ICML.
Root Mean Square Layer Normalization: Zhang, B., & Sennrich, R. (2019). NeurIPS.
Can We Gain More from Orthogonality Regularizations in Training Deep Networks?: Bansal, N., et al. (2018). NeurIPS.
Predicting the Generalization Gap in Deep Networks with Margin Distributions: Martin, C., & Mahoney, M. W. (2019). ICLR.

Complete bibliographic information is available in the documentation for individual modules.

Name		Name	Last commit message	Last commit date
imgs		imgs
plans		plans
research		research
scripts		scripts
src		src
tests		tests
tools		tools
.dockerignore		.dockerignore
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
.pylintrc		.pylintrc
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
REPO_MAP.md		REPO_MAP.md
generate_docs.py		generate_docs.py
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Navigation Menu

Repository files navigation

A Curated Arsenal of Advanced Deep Learning Techniques

Table of Contents

Key Features

Why `dl_techniques`?

Installation

Quick Start

1. Compose a State-of-the-Art Transformer Block

2. Deploy a Probabilistic Time Series Forecaster

3. Contrastive Vision-Language Learning with CLIP

4. Dissect Model Behavior with the Analysis Engine

5. Optimize Directly for F1-Score on Imbalanced Data

6. Harness Graph Neural Networks for Relational Data

In-Depth Documentation

Start Here: `REPO_MAP.md`

Tutorials & Deep Dives (`research/`)

Papers (`research/papers/`)

API Reference (per-module docs)

Project Structure

Contributing

Getting Started

Development Standards

Contribution Types

License

Acknowledgments

Citations & References

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

A Curated Arsenal of Advanced Deep Learning Techniques

Table of Contents

Key Features

Why dl_techniques?

Installation

Quick Start

1. Compose a State-of-the-Art Transformer Block

2. Deploy a Probabilistic Time Series Forecaster

3. Contrastive Vision-Language Learning with CLIP

4. Dissect Model Behavior with the Analysis Engine

5. Optimize Directly for F1-Score on Imbalanced Data

6. Harness Graph Neural Networks for Relational Data

In-Depth Documentation

Start Here: REPO_MAP.md

Tutorials & Deep Dives (research/)

Papers (research/papers/)

API Reference (per-module docs)

Project Structure

Contributing

Getting Started

Development Standards

Contribution Types

License

Acknowledgments

Citations & References

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

Why `dl_techniques`?

Start Here: `REPO_MAP.md`

Tutorials & Deep Dives (`research/`)

Papers (`research/papers/`)