Repository navigation
docs: add a page to estimate datacenter proxy IP addresses #3020
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
7a17bcf
440045e
7470001
745a870
8247dc5
1ed6efe
6ad1553
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| --- | ||
| title: Estimate how many datacenter IP addresses you need | ||
| sidebar_label: Estimate IPs | ||
| description: Estimate how many datacenter proxy IP addresses a job needs, compare the result with your plan, and get more with an add-on or a plan upgrade. | ||
| sidebar_position: 10.25 | ||
| slug: /proxy/estimate-ip-addresses | ||
| --- | ||
|
|
||
| Your plan includes a fixed number of [datacenter proxy](./datacenter_proxy.md) IP addresses. Before you run a large job through the proxy, estimate how many IP addresses it needs and compare the result with your plan. | ||
|
|
||
| The estimate applies to datacenter proxies only. [Residential proxies](./residential_proxy.md) are billed by traffic, the [Google SERP proxy](./google_serp_proxy.md) by request, and [Unblocker](./unblocker.md) in Unblocker units, so their IP count doesn't limit you. For Google Search results, use the Google SERP proxy instead of sizing datacenter IP addresses for them. | ||
|
|
||
| ## Calculate the estimate | ||
|
|
||
| The number of IP addresses a job needs depends on how many pages it visits on a single domain, how long it can take, and how often it repeats. Major websites need 10 times as many IP addresses as other websites. | ||
|
|
||
| | The job can take up to | Divide by | | ||
| | ---------------------- | --------- | | ||
| | A month | 10,000 | | ||
| | A day | 1,000 | | ||
| | An hour | 100 | | ||
|
|
||
| | The job repeats every | Multiply by | | ||
| | --------------------- | ----------- | | ||
| | Month | 1 | | ||
| | Day | 10 | | ||
| | Hour | 100 | | ||
|
|
||
| To estimate the number of IP addresses: | ||
|
|
||
| 1. Take the number of pages the job visits on a single domain. | ||
| 1. Divide it by the value for how long the job can take. | ||
| 1. Multiply the result by the value for how often the job repeats. | ||
| 1. If the target is a major website, such as CNN, multiply by 10. | ||
|
|
||
| ### Examples | ||
|
|
||
| | Scenario | Calculation | IP addresses | | ||
| | ------------------------------------------------------------------------------------- | ----------------------------- | ------------ | | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do these lengthy tables follow our style guide? I mean all these hyphens and spaces.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I actually don't think we have any for tables 🤔 rendered they both look the same, but it might be good opportunity to unify this. We have mdlint accepts both styles but requires unity within a page, not project |
||
| | 10,000 pages from a local e-commerce website, once a month, finished within a day | `10,000 / 1,000 * 1` | 10 | | ||
| | 5M product pages from a major online marketplace, once a month, spread over the month | `5,000,000 / 10,000 * 1 * 10` | 5,000 | | ||
| | 100 pages from each of 1,000 websites, every day, finished within a day | `100 / 1,000 * 10` | 1 | | ||
|
|
||
| Each estimate depends on its assumptions: | ||
|
|
||
| - The e-commerce estimate assumes you spread the job across the whole day. A faster job needs more IP addresses. | ||
| - The marketplace estimate seems high, but the job visits the same marketplace every day. [IP address rotation](./index.md#ip-address-rotation) spreads those visits across your IP addresses, so more IP addresses mean fewer visits from each one. Long-term reliable access to a major website needs about this many. | ||
| - The 1,000-websites estimate uses the number of pages per domain, not the total. Its one IP address covers all 1,000 websites. It assumes you spread the job and the websites don't use Cloudflare or a similar distributed IP protection system. | ||
|
|
||
| ## Compare the estimate with your plan | ||
|
|
||
| To see how many datacenter IP addresses your plan includes, go to **Billing** > **Subscription** in Apify Console and look under **Datacenter proxies**. To see the count in each proxy group, open the [Proxy](https://console.apify.com/proxy/groups) page and check the **Available IPs** column. | ||
|
|
||
| ## Get more IP addresses | ||
|
|
||
| If you need more IP addresses than your plan includes: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think that info deserves its own heading. |
||
|
|
||
| - On a paid plan, go to **Billing** > **Subscription**, select **Buy add-ons**, and enter the number of IP addresses in **Shared datacenter proxies**. | ||
| - Upgrade to a plan that includes more IP addresses. For the IP addresses and add-on price of each plan, see the [Apify pricing page](https://apify.com/pricing). | ||
| - For more than 1,999 additional IP addresses, [contact support](https://apify.com/contact). | ||
| - For IP addresses that only you use, ask support about [dedicated proxy groups](./datacenter_proxy.md#dedicated-proxy-groups). | ||
|
|
||
| ## Adjust the estimate | ||
|
|
||
| The tables give rough estimates. On some websites, they can be far off. If the job repeats without being blocked, try fewer IP addresses. If you see blocking, add more. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This process is hard to follow. Maybe an explicit example in each step?