-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathuber-xp-framework.html
More file actions
200 lines (181 loc) · 23.2 KB
/
uber-xp-framework.html
File metadata and controls
200 lines (181 loc) · 23.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Uber XP — staged rollout, universal holdout, marketplace experiments</title>
<link rel="stylesheet" href="framework.css">
<style>
/* Page-accent — overrides framework.css fallback */
:root{--page-accent:var(--blue);--page-accent-soft:var(--blue-soft)}
/* three pieces */
.three{display:grid;grid-template-columns:repeat(3,1fr);gap:14px;margin:14px 0}
.concept{background:#fff;border:1px solid var(--line);border-radius:12px;padding:18px 20px;box-shadow:var(--shadow);border-top:5px solid var(--page-accent)}
.concept .ab{font-family:Georgia,serif;font-size:13px;color:var(--page-accent);font-weight:700;letter-spacing:.05em;text-transform:uppercase;margin-bottom:4px}
.concept h3{margin:0 0 8px;font-size:17px;font-family:Georgia,serif}
.concept p{margin:0;font-size:13.5px;color:var(--ink-soft);line-height:1.5}
.concept p b{color:var(--ink)}
/* ramp visualization */
.ramp{background:#fff;border:1px solid var(--line);border-radius:12px;padding:22px;box-shadow:var(--shadow);margin:14px 0;text-align:center}
.ramp .stages{display:flex;justify-content:space-around;gap:8px;margin-top:14px;flex-wrap:wrap}
.ramp .stage{background:var(--page-accent-soft);border:1px solid #cdd9ee;border-radius:8px;padding:10px 14px;flex:1 1 100px;min-width:90px}
.ramp .stage .pct{font-family:Georgia,serif;font-size:20px;color:var(--page-accent);font-weight:700;line-height:1}
.ramp .stage .lb{font-size:12px;color:var(--ink-soft);margin-top:4px}
.ramp .arrow{display:inline-block;color:var(--page-accent);font-size:18px;font-weight:700;align-self:center}
/* stats */
.stats{display:flex;gap:10px;flex-wrap:wrap;margin:14px 0}
.stat{flex:1 1 150px;background:#f6f4ee;border:1px solid var(--line);border-radius:8px;padding:12px 14px;text-align:center}
.stat .n{font-family:Georgia,serif;font-size:22px;color:var(--page-accent);font-weight:700;line-height:1.1}
.stat .l{font-size:11.5px;color:var(--ink-soft);margin-top:4px;line-height:1.35}
@media(max-width:680px){.three{grid-template-columns:1fr}}
</style>
</head>
<body>
<nav class="sitenav">
<details>
<summary>📑 Jump to</summary>
<div class="navmenu">
<div class="navgrp"><h4>Start here</h4>
<a href="index.html"><b>← Home (goal & map)</b></a>
<a href="impact-saas-companies.html">SaaS / B2B field study</a>
<a href="impact-consumer-companies.html">Consumer-tech field study</a>
<a href="methodologies-comparison.html"><b>All methods compared →</b></a>
<a href="experiment-trustworthiness.html">How 40k tests actually work →</a>
<a href="jargon.html">Jargon (glossary)</a>
</div>
<div class="navgrp"><h4>Scoring & Input modeling</h4>
<a href="rice-framework.html">RICE (Intercom)</a>
<a href="north-star-framework.html">North Star (Amplitude / Slack)</a>
</div>
<div class="navgrp"><h4>Goal-laddering / Define first</h4>
<a href="v2mom-framework.html">V2MOM (Salesforce)</a>
<a href="pyramid-of-clarity-framework.html">Pyramid of Clarity (Asana)</a>
<a href="pr-faq-framework.html">PR-FAQ / Working Backwards (Amazon)</a>
<a href="heart-framework.html">HEART (Google)</a>
<a href="dibb-framework.html">DIBB (Spotify)</a>
</div>
<div class="navgrp"><h4>Experimentation (SaaS)</h4>
<a href="microsoft-exp-framework.html">Microsoft ExP / CUPED</a>
<a href="linkedin-xlnt-framework.html">LinkedIn T-REX</a>
</div>
<div class="navgrp"><h4>Experimentation (Consumer)</h4>
<a href="netflix-experimentation.html">Netflix · ABlaze</a>
<a href="booking-experimentation.html">Booking.com</a>
<a href="airbnb-erf-framework.html">Airbnb ERF</a>
<a class="cur" href="uber-xp-framework.html">Uber XP</a>
<a href="doordash-switchback-framework.html">DoorDash switchback</a>
<a href="lyft-experimentation.html">Lyft</a>
<a href="pinterest-ab-framework.html">Pinterest</a>
</div>
<div class="navgrp"><h4>AI labs</h4>
<a href="anthropic-pm-on-ai-exponential.html">Anthropic · PM on AI exponential</a>
<a href="google-customer-zero-2026.html">Google · "Customer zero" 2026</a>
</div>
<div class="navgrp"><h4>Written discipline</h4>
<a href="stripe-shaping-framework.html">Stripe shaping</a>
</div>
</div>
</details>
</nav>
<div class="wrap">
<header class="masthead">
<p class="kicker">Methods · Deep-dive · Experimentation</p>
<h1>Uber XP — staged rollout + universal holdout for long-term effects <span class="srcyr">2017</span></h1>
<p class="sub">Uber's experimentation platform (XP) combines <strong>flag-based <a class="j" href="jargon.html#ramp">staged rollout</a></strong> with a distinctive <strong><a class="j" href="jargon.html#universal-holdout">universal holdout</a></strong> — a small group permanently excluded from all new features to measure cumulative long-term impact.</p>
<p class="sub">First publicly described by Zhenyu Zhao & Eva Feng in <a class="cite" href="https://www.uber.com/blog/experimentation-platform/">"Building an Intelligent Experimentation Platform with Uber Engineering" (May 4, 2017)</a>. A separate 2023 engineering post documents a 100× speedup in the experiment-evaluation engine.</p>
<div class="goal"><span>Goal</span><br>Decide features by data-backed expected impact — choose by outcome, not by to-do list or opinion.</div>
</header>
<div class="eli">
<div class="lbl">🎓 8th-grade version</div>
Uber ships features carefully: first to <b>1%</b> of users to make sure nothing crashes, then 5%, then 25%, then 50%, then everyone. At each step the system checks: did the number we cared about go up, and did nothing important break? If something breaks, the system rolls back automatically — no human needed. Their cleverest move: they keep <b>a small group of users who never get any new features for a whole quarter</b>. Then they compare those users to everyone else. If everyone else's product isn't actually better after 50 features ship, then those 50 "wins" were lies. This catches a sneaky failure: each feature looks like it helps a little, but together they made the product worse.
</div>
<nav class="toc">
<a href="#headline">Honest headline</a>
<a href="#anatomy">Three pieces</a>
<a href="#mechanism">How it picks work</a>
<a href="#ramp">The ramp pattern</a>
<a href="#apply">Apply to a sheet</a>
<a href="methodologies-comparison.html" style="color:var(--blue);font-weight:700">Comparison table →</a>
</nav>
<div class="finding" id="headline">
<h2>The honest headline: the long-run holdout is the underrated idea</h2>
<p>Any platform can run an A/B test. Uber's contribution: <b>a "universal holdout" — a permanently-excluded slice of users who never see new features</b> — to measure the <em>cumulative</em> impact of everything shipped over a quarter or year. It catches the failure mode where every individual test "wins" but the product gets worse overall (death by a thousand local optima).</p>
<p>Same decision rule as the rest (OEC + ramp). The holdout is the audit layer that the rest of the experimentation set tends to skip.</p>
</div>
<!-- ANATOMY -->
<h2 class="sec" id="anatomy">Three pieces — XP, staged rollout, universal holdout</h2>
<div class="three">
<div class="concept">
<div class="ab">XP</div>
<h3>The platform</h3>
<p>Uber's internal experimentation platform — flag service, randomization, metrics, dashboards. Defined in the 2017 origin post as a system to "ensure that new features roll out successfully and then return actionable analysis." The headline engineering win came later: a 2023 evaluation-engine rewrite cut experiment-evaluation latency 100×.</p>
</div>
<div class="concept">
<div class="ab">Ramp</div>
<h3>Staged rollout as the default</h3>
<p>Every feature ships behind a flag and ramps 1% → 5% → 25% → 50% → 100%. At each step the platform reads the OEC and guardrails; a regression auto-rolls-back.</p>
</div>
<div class="concept">
<div class="ab">Holdout</div>
<h3>Universal holdout — cumulative long-term effect</h3>
<p>A small % of users is <b>permanently excluded</b> from all new features for a period (e.g. a quarter). Comparing their behaviour to the rest measures the <em>aggregate</em> effect of the entire shipped quarter — not just per-feature lifts.</p>
</div>
</div>
<div class="src">Sources: <a class="cite" href="https://www.uber.com/us/en/blog/experimentation-platform/">Zhenyu Zhao & Eva Feng — "Building an Intelligent Experimentation Platform with Uber Engineering" (May 4, 2017)</a> · <a class="cite" href="https://www.uber.com/us/en/blog/xp/">"Under the Hood of Uber's Experimentation Platform"</a> · <a class="cite" href="https://www.uber.com/blog/making-ubers-experiment-evaluation-engine-100x-faster/">"Making Uber's Experiment Evaluation Engine 100x Faster"</a> (the 100× speedup post, 2023).</div>
<!-- MECHANISM -->
<h2 class="sec" id="mechanism">How an XP-style decision actually gets made</h2>
<div class="step"><div class="num">1</div><div><h3>Define the experiment in the platform UI</h3><p>Hypothesis, OEC, guardrails, segments. Standard config so the platform can auto-monitor.</p></div></div>
<div class="step"><div class="num">2</div><div><h3>Ramp behind a flag, with auto-rollback</h3><p>If guardrail metrics breach during a ramp step, the platform rolls the flag back automatically. Engineers don't have to be watching.</p></div></div>
<div class="step"><div class="num">3</div><div><h3>Read the OEC at each step; promote or kill</h3><p>Each ramp step is itself an experiment. Most features die at the 1% or 5% step.</p></div></div>
<div class="step"><div class="num">4</div><div><h3>Compare against the universal holdout, quarterly</h3><p>The audit: did the quarter's shipped work actually improve the product vs the held-out users? If not, individual wins were probably local — a cue to re-examine the OECs.</p></div></div>
<div class="step"><div class="num">5</div><div><h3>Loop the holdout learning back into metric design</h3><p>If many quarters show no aggregate lift despite per-feature wins, the OECs are wrong (or the metric is gameable). The holdout forces this honesty.</p></div></div>
<!-- RAMP -->
<h2 class="sec" id="ramp">The ramp pattern — what shipping behind XP looks like</h2>
<p class="secsub">Five steps. Every feature, every time. The ramp is the experiment.</p>
<div class="ramp">
<div class="stages">
<div class="stage"><div class="pct">1%</div><div class="lb">Smoke test — does anything crash?</div></div>
<div class="stage"><div class="pct">5%</div><div class="lb">Early read of OEC + guardrails</div></div>
<div class="stage"><div class="pct">25%</div><div class="lb">Statistical significance check</div></div>
<div class="stage"><div class="pct">50%</div><div class="lb">Confirm at scale</div></div>
<div class="stage"><div class="pct">100%</div><div class="lb">Ship to everyone (except holdout)</div></div>
</div>
<p style="font-size:12.5px;color:var(--ink-soft);margin:14px 0 0">Auto-rollback at any step if guardrails breach. Most features that don't ship to 100% never make it past the 5% step — exactly when killing is cheapest.</p>
</div>
<div class="stats">
<div class="stat"><div class="n">1,000+</div><div class="l">experiments running concurrently (Uber-published estimate, exact figure varies across posts)</div></div>
<div class="stat"><div class="n">100×</div><div class="l">faster experiment evaluation (2023 engine rewrite)</div></div>
<div class="stat"><div class="n">~1–5%</div><div class="l">typical universal-holdout size</div></div>
</div>
<!-- APPLY TO A SHEET -->
<h2 class="sec" id="apply">Apply to a feature sheet</h2>
<p class="secsub">Uber XP adds two Uber-specific columns to the standard experiment ledger: <em>two-sided OEC</em> (a feature must clear rider <em>and</em> driver metrics) and <em>universal-holdout check</em> (the quarterly causal confirmation that the cumulative product is better, not just each individual test). The staged 1%→5%→25%→50%→100% ramp is the standard layer underneath.</p>
<div class="note" style="background:var(--teal-soft);border-left-color:var(--teal)"><b>Try it Monday morning (30 minutes).</b> You probably don't have a "universal holdout" running. Set up a primitive version: list every shipped feature from last quarter on a doc. For each one, write the OEC delta it claimed. Now ask: if all those wins were real, would the headline business metric (<a class="j" href="jargon.html#wau-dau-mau">DAU</a>, revenue, <a class="j" href="jargon.html#nps">NPS</a>) be visibly better this quarter vs last? If yes — great, your wins compound. If no — you have a holdout-shaped problem: your per-feature OECs are picking up local noise that doesn't aggregate. The universal holdout is the institutional version of this exercise.</div>
<div class="note" style="background:var(--blue-soft);border-left-color:var(--blue);font-size:13.5px"><b>Quick glossary for the columns below.</b> <b>Two-sided OEC</b> = a marketplace metric pair (rider + driver) where a feature must help one side without harming the other. <b>Universal holdout</b> = a small slice of users (typically 1–5%) who never see <em>any</em> new feature for a fixed period (e.g. a quarter); comparing them to everyone else measures the cumulative product change in a causal way no roll-up of individual tests can. <b>Auto-rollback</b> = the platform unwinds a flag automatically if a guardrail breaches during ramp — no human required.</div>
<h3 style="font-family:Georgia,serif;font-size:18px;margin:18px 0 8px">Worked example — an experiment ledger snapshot (Uber-style)</h3>
<p style="font-size:13.5px;color:var(--ink-soft);margin:0 0 12px">Eight tests. Note the two-sided OEC: features that lift one side at the other side's expense are killed at the guardrail.</p>
<div style="overflow-x:auto;margin:14px 0">
<table style="border-collapse:collapse;width:100%;font-size:13px;background:#fff;border:1px solid var(--line);border-radius:10px;overflow:hidden">
<thead><tr style="background:var(--ink);color:#f3efe6;font-size:11.5px;letter-spacing:.05em;text-transform:uppercase"><th style="padding:9px 10px;text-align:left">Feature</th><th style="padding:9px 10px;text-align:left">Rider OEC</th><th style="padding:9px 10px;text-align:left">Driver OEC</th><th style="padding:9px 10px;text-align:left">Two-sided guardrails</th><th style="padding:9px 10px;text-align:left">Ramp stage</th><th style="padding:9px 10px;text-align:left">Result</th><th style="padding:9px 10px;text-align:left">Decision</th></tr></thead>
<tbody>
<tr style="background:#e6ecf6"><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-weight:600">New ETA algorithm</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Wait time</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Utilization</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Cancellation, ETA accuracy</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">100%</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Wait −8s; util +0.3pp</td><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-family:Georgia,serif;color:var(--blue);font-weight:700">Ship</td></tr>
<tr style="background:#e6ecf6"><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-weight:600">Dynamic surge mapping</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Trips completed</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Earnings/hr</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Rider price sensitivity</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">50%</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Trips +0.7%, earn +1.1%</td><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-family:Georgia,serif;color:var(--blue);font-weight:700">Ship</td></tr>
<tr style="background:#e6ecf6"><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-weight:600">New driver-allocation algo</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Wait time</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Utilization</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Both-sided cancel</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">100%</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Wait −4s; util +0.5pp</td><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-family:Georgia,serif;color:var(--blue);font-weight:700">Ship</td></tr>
<tr style="background:#e6ecf6"><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-weight:600">In-app tipping UI</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">—</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Tip rate</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Rider friction</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">100%</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Tips +1.2pp, friction OK</td><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-family:Georgia,serif;color:var(--blue);font-weight:700">Ship</td></tr>
<tr><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-weight:600">Pool variant — bigger detour budget</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Trips/$</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Earnings/hr</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Rider satisfaction</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">25%</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Trips OK BUT satisfaction −3pts</td><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-family:Georgia,serif;color:var(--gold);font-weight:700">Iterate — softer detour</td></tr>
<tr><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-weight:600">UI redesign (home tab)</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Trip-request rate</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">N/A</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Bounce, load time</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">25%</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Flat</td><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-family:Georgia,serif;color:var(--gold);font-weight:700">Iterate</td></tr>
<tr><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-weight:600">Aggressive pre-booking</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Wait time</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Utilization</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Driver cancellations</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">5% halted</td><td style="padding:9px 10px;border-bottom:1px solid var(--line)">Wait −10s BUT driver cancels +12%</td><td style="padding:9px 10px;border-bottom:1px solid var(--line);font-family:Georgia,serif;color:var(--accent);font-weight:700">Kill</td></tr>
<tr><td style="padding:9px 10px;font-weight:600">Quarterly universal-holdout readout</td><td style="padding:9px 10px">Trips, wait</td><td style="padding:9px 10px">Earnings, util</td><td style="padding:9px 10px">All</td><td style="padding:9px 10px">1–5% holdout</td><td style="padding:9px 10px">Holdout confirms +2.1% trips vs last Q's product</td><td style="padding:9px 10px;font-family:Georgia,serif;color:var(--blue);font-weight:700">Confirm</td></tr>
</tbody>
</table>
</div>
<div class="note" style="background:var(--accent-soft);border-left-color:var(--accent)"><b>The most important reading skill on this page.</b> Read the bottom two rows together. The pre-booking row (Kill) shows the <em>per-test</em> failure mode — one feature, one A/B, one Kill verdict for breaching a guardrail. The universal-holdout row shows the <em>aggregate</em> failure mode — a check that's not about any single feature, but about whether the entire quarter's product is actually better than the previous quarter's. Both layers are needed. A team with great per-test discipline can still drift backward over time if it never compares its current product to its past one. The holdout is how you spot that drift before it compounds.</div>
<div class="note"><b>Decision rule.</b> A feature must move <em>both</em> sides (or one side without hurting the other) AND clear two-sided guardrails. The pre-booking row is the textbook Uber kill: shorter rider wait is wonderful, until driver cancellations spike 12% because they bailed on locked-in fares. The universal-holdout row is the quarterly confirmation — even after shipping 50 individual wins, the holdout answers the only question that matters: <em>is this quarter's product actually better than last quarter's?</em></div>
<div class="note"><b>The transferable insight: the holdout, not the ramp.</b> Staged ramps are now a standard pattern across the industry (LinkedIn, Booking, Microsoft all do it). The universal holdout is rarer and more powerful — it answers the question <em>"is the product better this quarter than it was last quarter?"</em> in a causal way no roll-up of individual tests can. The honest cost: a small slice of users gets a deliberately worse product for a quarter.</div>
<footer>
Companion to <a href="impact-consumer-companies.html#measure">← Consumer case studies · Measure don't estimate</a> · <a href="methodologies-comparison.html">All methods compared</a> · siblings: <a href="microsoft-exp-framework.html">Microsoft ExP</a> · <a href="doordash-switchback-framework.html">DoorDash switchback</a> (other marketplace-specific design)<br>
<b>Grounded in</b> <a href="https://www.uber.com/us/en/blog/experimentation-platform/">Zhao & Feng — "Building an Intelligent Experimentation Platform with Uber Engineering" (May 4, 2017)</a>, <a href="https://www.uber.com/us/en/blog/xp/">"Under the Hood of Uber's Experimentation Platform"</a>, and <a href="https://www.uber.com/blog/making-ubers-experiment-evaluation-engine-100x-faster/">"Making Uber's Experiment Evaluation Engine 100x Faster" (2023)</a>. <b>Verbatim from Uber:</b> the XP definition (a system to "ensure that new features roll out successfully and then return actionable analysis"), the staged-rollout framing ("controlling user exposure in the early stages"), the universal-holdout concept, and the 100× evaluation speedup figure. <b>Attribution-timing note:</b> the universal holdout and staged rollout are 2017+ concepts; the 100× engine speedup is from a separate 2023 post — the page now keeps these distinct. <b>Added by us, not in Uber's posts:</b> the specific ramp percentages (1%/5%/25%/50%/100% is a common industry pattern, not necessarily Uber's exact published default), the 8-row worked example with rider/driver OEC pairs, the verdict labels, the in-page glossary, and the "Try it Monday" exercise. The "1,000+ concurrent" figure is a reasonable estimate from across Uber's posts but should be re-verified before quoting.
</footer>
</div>
</body>
</html>