Lab 11 submission - Advanced Microservice Patterns - #358
Open
N1qro wants to merge 35 commits into
Open
Conversation
…eful degradation and resource analysis
Lab 1 submission - deploy, break, understand
Lab 2 submission - inspect, optimize, trace
Lab 3 submission - monitor, observe, define SLOs
Lab 4 submission - deploy, probe, helm
Lab 5 submission - CI/CD, GitOps, rollback
Lab 6 submission - alerting, responding
Fix CI manifest update push race
Lab 7 submission - Progressive Delivery
Lab 8 submission - Chaos Engineering
Lab 9 submission - Stateful Services and DB Reliability
Lab 10 submission - SRE Portfolio and Reliability Review
There was a problem hiding this comment.
Pull request overview
This pull request delivers the Lab 11 “Advanced Microservice Patterns” work for the QuickTicket training project by adding a new notifications microservice, wiring it through the gateway, and introducing resilience patterns (retry, circuit breaker, rate limiting), alongside supporting Kubernetes, monitoring, CI, and lab-submission documentation updates.
Changes:
- Added a new
notificationsservice (app + Kubernetes manifest) and wired the gateway to notify asynchronously (fire-and-forget). - Implemented retry with exponential backoff/jitter, a CLOSED/OPEN/HALF_OPEN circuit breaker, and a per-endpoint sliding-window rate limiter in the gateway.
- Expanded operational artifacts (Prometheus rules, Grafana dashboard, runbook, migrations, CI, and lab writeups) to support reliability and observability labs.
Reviewed changes
Copilot reviewed 49 out of 50 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| submissions/runbooks/quickticket-handbook.md | Adds an SRE handbook/runbook for the QuickTicket deployment and operations. |
| submissions/lab1.md | Adds Lab 1 writeup documenting baseline behavior, failures, and graceful degradation. |
| submissions/lab2.md | Adds Lab 2 writeup on containerization/inspection/optimization. |
| submissions/lab3.md | Adds Lab 3 writeup on monitoring/observability/SLOs and Prometheus setup. |
| submissions/lab4.md | Adds Lab 4 writeup on Kubernetes deployment/manifests and k3d evidence. |
| submissions/lab5.md | Adds Lab 5 writeup on CI/CD + GitOps (ArgoCD) evidence and workflows. |
| submissions/lab6.md | Adds Lab 6 writeup on alerting, runbooks, and incident response/postmortem. |
| submissions/lab7.md | Adds Lab 7 writeup on canary deployments with Argo Rollouts and analysis. |
| submissions/lab8.md | Adds Lab 8 writeup on chaos engineering experiments and observations. |
| submissions/lab9.md | Adds Lab 9 writeup on stateful services, migrations, backups, and DR. |
| submissions/lab10.md | Adds Lab 10 reliability review with load testing, DORA metrics, and risks. |
| submissions/lab11.md | Adds Lab 11 writeup covering notifications + retry/CB/rate limiting behaviors. |
| monitoring/prometheus/prometheus.yml | Defines Prometheus scrape config for gateway/events/payments. |
| monitoring/prometheus/rules.yml | Adds recording rules for SLO/SLI calculations used by dashboards. |
| monitoring/grafana/dashboards/golden-signals.json | Replaces placeholder panels with latency/saturation/SLO visualizations. |
| migrations/env.py | Adds Alembic migration environment configuration. |
| migrations/README | Adds Alembic migrations README. |
| migrations/script.py.mako | Adds Alembic revision template. |
| migrations/versions/0001_baseline_pre_existing_schema.py | Adds baseline Alembic revision. |
| migrations/versions/0002_add_email_column_to_events.py | Adds migration introducing nullable email on events. |
| alembic.ini | Adds Alembic configuration (script location + DB URL). |
| locustfile.py | Adds in-cluster Locust scenario used for load testing. |
| k8s/redis.yaml | Defines Redis Deployment/Service for Kubernetes. |
| k8s/postgres.yaml | Defines Postgres Deployment/Service and adds PVC-backed storage. |
| k8s/events.yaml | Defines Events Deployment/Service (including scaling to 2 replicas). |
| k8s/payments.yaml | Defines Payments Deployment/Service using GHCR image + probes/resources. |
| k8s/gateway.yaml | Defines gateway as an Argo Rollouts Rollout and sets NOTIFICATIONS_URL env. |
| k8s/notifications.yaml | Adds notifications Deployment/Service for Kubernetes. |
| k8s/analysis-template.yaml | Adds an Argo Rollouts AnalysisTemplate for canary error-rate analysis. |
| k8s/backup-cronjob.yaml | Adds CronJob for periodic Postgres custom-format backups to PVC. |
| k8s/chart/Chart.yaml | Adds Helm chart metadata for the QuickTicket stack. |
| k8s/chart/values.yaml | Adds Helm chart values for services/resources configuration. |
| k8s/chart/templates/gateway.yaml | Adds Helm template for gateway Deployment/Service. |
| k8s/chart/templates/events.yaml | Adds Helm template for events Deployment/Service. |
| k8s/chart/templates/payments.yaml | Adds Helm template for payments Deployment/Service. |
| k8s/chart/templates/postgres.yaml | Adds Helm template for postgres Deployment/Service. |
| k8s/chart/templates/redis.yaml | Adds Helm template for redis Deployment/Service. |
| docker-compose.monitoring.yaml | Updates monitoring compose config to mount Prometheus rules file. |
| app/gateway/main.py | Implements retry, circuit breaker, rate limiter, and async notifications call. |
| app/gateway/Dockerfile | Updates gateway container to run as non-root (and adjusts EXPOSE). |
| app/gateway/.dockerignore | Adds dockerignore patterns for the gateway image build context. |
| app/events/Dockerfile | Updates events container to run as non-root (and adjusts EXPOSE). |
| app/events/.dockerignore | Adds dockerignore patterns for the events image build context. |
| app/payments/Dockerfile | Updates payments container to run as non-root (and adjusts EXPOSE). |
| app/payments/.dockerignore | Adds dockerignore patterns for the payments image build context. |
| app/notifications/main.py | Adds notifications FastAPI service with fault injection + Prometheus metrics. |
| app/notifications/requirements.txt | Adds notifications service dependency pins. |
| app/notifications/Dockerfile | Adds notifications service image build definition (non-root runtime). |
| .github/workflows/ci.yml | Adds CI workflow to build/push images and update k8s manifests on merges. |
| .gitignore | Updates ignore rules to allow committing submission and Kubernetes content. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+385
to
395
| except httpx.RequestError: | ||
| return JSONResponse( | ||
| status_code=503, | ||
| content={ | ||
| "error": "payments_unavailable", | ||
| "message": "Payment service is temporarily down. Your reservation is held — try again in a few minutes.", | ||
| "reservation_id": reservation_id, | ||
| }, | ||
| ) | ||
| except httpx.TimeoutException: | ||
| raise HTTPException(504, "Payment service timeout") |
Comment on lines
+104
to
+107
| last_error = None | ||
| base_delay = RETRY_BASE_DELAY_MS / 1000 | ||
|
|
||
| for attempt in range(max_retries): |
Comment on lines
+8
to
12
| EXPOSE 8080 | ||
| RUN addgroup --system app && adduser --system --ingroup app app | ||
| USER app | ||
|
|
||
| EXPOSE 8080 |
Comment on lines
+8
to
12
| EXPOSE 8081 | ||
| RUN addgroup --system app && adduser --system --ingroup app app | ||
| USER app | ||
|
|
||
| EXPOSE 8081 |
Comment on lines
+8
to
12
| EXPOSE 8082 | ||
| RUN addgroup --system app && adduser --system --ingroup app app | ||
| USER app | ||
|
|
||
| EXPOSE 8082 |
Comment on lines
+19
to
+24
| ( | ||
| sum(rate(gateway_requests_total{rs_hash="{{args.canary-hash}}",status=~"5.."}[60s])) | ||
| or on() vector(0) | ||
| ) | ||
| / | ||
| sum(rate(gateway_requests_total{rs_hash="{{args.canary-hash}}"}[60s])) |
Comment on lines
+6
to
+16
| Locust / users | ||
| | | ||
| v | ||
| gateway Rollout, 5 replicas | ||
| | | ||
| +--> events Deployment, 2 replicas ----> PostgreSQL Deployment + PVC | ||
| | | | ||
| | +---------------------> Redis holds | ||
| | | ||
| +--> payments Deployment, 1 replica | ||
|
|
Comment on lines
+22
to
+25
| - Gateway owns the public HTTP API and routes to events and payments. | ||
| - Events owns event listing, reservations, confirmations, Redis holds, and Postgres writes. | ||
| - Payments is intentionally simple and supports failure/latency injection for labs. | ||
| - PostgreSQL is PVC-backed after Lab 9; backups are handled by the `postgres-backup` CronJob. |
Comment on lines
+34
to
+38
| - name: Build and push QuickTicket images | ||
| env: | ||
| IMAGE_OWNER: n1qro | ||
| SHA: ${{ github.sha }} | ||
| run: | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lab 11 - Advanced Microservice Patterns
Summary
Retry-After: 1submissions/lab11.md