Skip to content

Commit 7f0881e

Browse files
cpanarescpanares
authored
Merge pull request #202 from DomainTools/IDEV-2563-ip-feeds
IDEV-2563: Add IP Feeds to DT Python Wrapper
2 parents 445384b + 1e743f1 commit 7f0881e

9 files changed

Lines changed: 559 additions & 3 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 64 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,68 @@
11
# Changelog
22

3+
### 2.9.0
4+
- [NEW] Add support for querying real-time IP feeds (`iphotlist` and `iprisk`).
5+
6+
### 2.8.1
7+
- [FIX] Update python wrapper with a patch that allows the wrapper to proceed with Iris Enrich call even if `account_information` returns a 503 error only (rate limit error).
8+
9+
### 2.8.0
10+
- [NEW] Add IrisQL Support in `iris_investigate` function.
11+
- [NEW] Add `domain_history` command/function in the wrapper.
12+
13+
### 2.7.4
14+
- [FIX] Handle validation error for none value arguments.
15+
16+
### 2.7.3
17+
- [FIX] Issue on missing `risk_score` when passing it on `iris_investigate` function.
18+
19+
### 2.7.2
20+
- [FIX] Issue on importing OpenAPI spec.
21+
22+
### 2.7.0
23+
- [FIX] `iris_investigate` API parity updates (update help documentation)
24+
- [FIX] `iris_investigate` `active` should raise an error as it requires a boolean not a string
25+
26+
### 2.6.0
27+
- [NEW] Implement streaming request for RTTF endpoints to handle the feeds properly and yield the feed line by line and not storing all of it in memory.
28+
- [UPDATE] Test cases.
29+
- [FIX] `available_api_calls` function for RTTF products
30+
31+
### 2.5.3
32+
- [UPDATE] Add IrisQL Support in `iris_investigate` function.
33+
- [UPDATE] Change default Feeds authentication behavior
34+
35+
### 2.5.2
36+
- [FIX] Total count bug for Iris Investigate and Iris Enrich results.
37+
38+
### 2.5.1
39+
- [FIX] Fix for bug found in `realtime_domain_risk` endpoint. Changed product name from `domain-risk-<source>` to `domain-risk-feed-<source>`
40+
41+
### 2.5.0
42+
- [NEW] Add support for Real Time Domain Risk Feed
43+
- [NEW] Add support for Domain Hotlist Feed
44+
- [NEW] Add e2e tests for proxy and ssl
45+
- [UPDATE] Integrate e2e tests in CI pipeline.
46+
- [FIX] Bugs found in `/domainrisk` and `/domainhotlist` endpoints
47+
48+
### 2.4.1
49+
- [UPDATE] Remove support for MD5 based signing.
50+
51+
### 2.4.0
52+
- [NEW] Integrate the NOH Feed to be supportable with the Python Wrapper.
53+
- [NEW] Improve worfklow to automate publishing the package to PyPI.
54+
- [UPDATE] Remove PhishEye.
55+
- [FIX] Improvements on help texts.
56+
- [FIX] Pegged httpxdependency to v.0.28.1 to prevent proxy key error.
57+
58+
### 2.3.0
59+
- [NEW] Integrate the Domain RDAP Feed to be supportable with the Python Wrapper.
60+
- [NEW] Integrate the Domain Discovery Feed to be supportable with the Python Wrapper.
61+
- [UPDATE] Enhancements to RTUF endpoints to support the following: download API, header authentication, csv format.
62+
- [UPDATE] Processing of Iterative Response (HTTP 206) from RTUF endpoints.
63+
- [UPDATE] Help Text and Information Using New Documentation.
64+
- [FIX] Simplification of using the new `proxy` param of httpx.Client. Before we’re using proxy mounts equivalent which we used `proxies` but this was deprecated on httpx v.0.28.x and onward causing errors in the python_wrapper
65+
366
### 2.0.0
467
- [NEW] Modernize package - migrate package settings to pyproject.toml
568
- [NEW] Migrate CLI wrapper to use `typer` library. (CLI comes now with new interface.)
@@ -11,12 +74,11 @@
1174
- Filtering of results based on `updated_after` field.
1275
- Filtering of results based on a missing field. (include_domains_with_missing_field` or `exclude_domains_with_missing_field`).
1376
- [NEW] Add support on removing/stripping colon in when passing a value in `--ssl_hash` in `iris_investigate` cli command.
14-
- [UPDATE] replace use of upcoming deprecated `datetime.uctnow()` to `timezone.utc`
77+
- [UPDATE] replace use of upcoming deprecated `datetime.utcnow()` to `timezone.utc`
1578
- [UPDATE] Improve help text in CLI commands.
1679
- [UPDATE] Remove `dateparser` dependency and use native python `datetime` library.
1780
- [FIX] Fix error in `-o` or `--out-file` parameter.
1881

19-
2082
### 1.0.1
2183

2284
- Adds support for the hourly query limit on the Account API endpoint

‎domaintools/api.py‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1412,3 +1412,67 @@ def domainhotlist(self, **kwargs) -> FeedsResults:
14121412
cls=FeedsResults,
14131413
**kwargs,
14141414
)
1415+
1416+
def iphotlist(self, **kwargs) -> FeedsResults:
1417+
"""Returns back list of ip hotlist feed.
1418+
Captures IP addresses that meet strict criteria for both risk level and recent activity, making it ideal for immediate blocking and threat response.
1419+
1420+
before: str: Filter for records before the given time value inclusive or time offset relative to now
1421+
1422+
after: str: Filter for records after the given time value inclusive or time offset relative to now
1423+
1424+
headers: bool: Use in combination with Accept: text/csv headers to control if headers are sent or not
1425+
1426+
sessionID: str: A custom string to distinguish between different sessions
1427+
1428+
fromBeginning: bool: Requires a sessionID. When used with a new session ID, returns the first hour of data in the time window (rather than the last). Returns an error if the session ID already exists
1429+
"""
1430+
validate_feeds_parameters(kwargs)
1431+
endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
1432+
source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
1433+
if (
1434+
endpoint == Endpoint.DOWNLOAD.value
1435+
or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
1436+
):
1437+
# headers param is allowed only in Feed API and CSV format
1438+
kwargs.pop("headers", None)
1439+
1440+
return self._results(
1441+
f"real-time-ip-hotlist-({source})",
1442+
f"v1/{endpoint}/iphotlist/",
1443+
response_path=(),
1444+
cls=FeedsResults,
1445+
**kwargs,
1446+
)
1447+
1448+
def iprisk(self, **kwargs) -> FeedsResults:
1449+
"""Returns back list of domain hotlist feed.
1450+
Captures all IP addresses that actively host one or more domains, providing risk assessment and enrichment data for each IP address.
1451+
1452+
before: str: Filter for records before the given time value inclusive or time offset relative to now
1453+
1454+
after: str: Filter for records after the given time value inclusive or time offset relative to now
1455+
1456+
headers: bool: Use in combination with Accept: text/csv headers to control if headers are sent or not
1457+
1458+
sessionID: str: A custom string to distinguish between different sessions
1459+
1460+
fromBeginning: bool: Requires a sessionID. When used with a new session ID, returns the first hour of data in the time window (rather than the last). Returns an error if the session ID already exists
1461+
"""
1462+
validate_feeds_parameters(kwargs)
1463+
endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
1464+
source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
1465+
if (
1466+
endpoint == Endpoint.DOWNLOAD.value
1467+
or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
1468+
):
1469+
# headers param is allowed only in Feed API and CSV format
1470+
kwargs.pop("headers", None)
1471+
1472+
return self._results(
1473+
f"real-time-ip-risk-({source})",
1474+
f"v1/{endpoint}/iprisk/",
1475+
response_path=(),
1476+
cls=FeedsResults,
1477+
**kwargs,
1478+
)

‎domaintools/cli/commands/feeds.py‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -554,3 +554,153 @@ def feeds_realtime_domain_risk(
554554
),
555555
):
556556
DTCLICommand.run(name=c.FEEDS_REALTIME_DOMAIN_RISK, params=ctx.params)
557+
558+
559+
@dt_cli.command(
560+
name=c.FEEDS_IPHOTLIST,
561+
help=get_cli_helptext_by_name(command_name=c.FEEDS_IPHOTLIST),
562+
)
563+
def feeds_iphotlist(
564+
ctx: typer.Context,
565+
user: str = typer.Option(None, "-u", "--user", help="Domaintools API Username."),
566+
key: str = typer.Option(None, "-k", "--key", help="DomainTools API key"),
567+
creds_file: str = typer.Option(
568+
"~/.dtapi",
569+
"-c",
570+
"--credfile",
571+
help="Optional file with API username and API key, one per line.",
572+
),
573+
no_verify_ssl: bool = typer.Option(
574+
False,
575+
"--no-verify-ssl",
576+
help="Skip verification of SSL certificate when making HTTPs API calls",
577+
),
578+
no_sign_api_key: bool = typer.Option(
579+
False,
580+
"--no-sign-api-key",
581+
help="Skip signing of api key",
582+
),
583+
no_header_authentication: bool = typer.Option(
584+
False,
585+
"--no-header-auth",
586+
help="Don't use header authentication",
587+
),
588+
output_format: str = typer.Option(
589+
"jsonl",
590+
"-f",
591+
"--format",
592+
help=f"Output format in [{OutputFormat.JSONL.value}, {OutputFormat.CSV.value}]",
593+
callback=DTCLICommand.validate_feeds_format_input,
594+
),
595+
endpoint: str = typer.Option(
596+
Endpoint.FEED.value,
597+
"-e",
598+
"--endpoint",
599+
help=f"Valid endpoints: [{Endpoint.FEED.value}, {Endpoint.DOWNLOAD.value}]",
600+
callback=DTCLICommand.validate_endpoint_input,
601+
),
602+
sessionID: str = typer.Option(
603+
None,
604+
"--session-id",
605+
help="Unique identifier for the session",
606+
),
607+
after: str = typer.Option(
608+
None,
609+
"--after",
610+
help="Start of the time window, relative to the current time in seconds, for which data will be provided",
611+
callback=DTCLICommand.validate_after_or_before_input,
612+
),
613+
before: str = typer.Option(
614+
None,
615+
"--before",
616+
help="The end of the query window in seconds, relative to the current time, inclusive",
617+
callback=DTCLICommand.validate_after_or_before_input,
618+
),
619+
fromBeginning: bool = typer.Option(
620+
None,
621+
"-fb",
622+
"--frombeginning",
623+
help="Requires a sessionID. When used with a new session ID, returns the first hour of data in the time window (rather than the last). Returns an error if the session ID already exists",
624+
),
625+
headers: bool = typer.Option(
626+
False,
627+
"--headers",
628+
help="Adds a header to the first line of response when text/csv is set in header parameters",
629+
),
630+
):
631+
DTCLICommand.run(name=c.FEEDS_IPHOTLIST, params=ctx.params)
632+
633+
634+
@dt_cli.command(
635+
name=c.FEEDS_IPRISK,
636+
help=get_cli_helptext_by_name(command_name=c.FEEDS_IPRISK),
637+
)
638+
def feeds_iprisk(
639+
ctx: typer.Context,
640+
user: str = typer.Option(None, "-u", "--user", help="Domaintools API Username."),
641+
key: str = typer.Option(None, "-k", "--key", help="DomainTools API key"),
642+
creds_file: str = typer.Option(
643+
"~/.dtapi",
644+
"-c",
645+
"--credfile",
646+
help="Optional file with API username and API key, one per line.",
647+
),
648+
no_verify_ssl: bool = typer.Option(
649+
False,
650+
"--no-verify-ssl",
651+
help="Skip verification of SSL certificate when making HTTPs API calls",
652+
),
653+
no_sign_api_key: bool = typer.Option(
654+
False,
655+
"--no-sign-api-key",
656+
help="Skip signing of api key",
657+
),
658+
no_header_authentication: bool = typer.Option(
659+
False,
660+
"--no-header-auth",
661+
help="Don't use header authentication",
662+
),
663+
output_format: str = typer.Option(
664+
"jsonl",
665+
"-f",
666+
"--format",
667+
help=f"Output format in [{OutputFormat.JSONL.value}, {OutputFormat.CSV.value}]",
668+
callback=DTCLICommand.validate_feeds_format_input,
669+
),
670+
endpoint: str = typer.Option(
671+
Endpoint.FEED.value,
672+
"-e",
673+
"--endpoint",
674+
help=f"Valid endpoints: [{Endpoint.FEED.value}, {Endpoint.DOWNLOAD.value}]",
675+
callback=DTCLICommand.validate_endpoint_input,
676+
),
677+
sessionID: str = typer.Option(
678+
None,
679+
"--session-id",
680+
help="Unique identifier for the session",
681+
),
682+
after: str = typer.Option(
683+
None,
684+
"--after",
685+
help="Start of the time window, relative to the current time in seconds, for which data will be provided",
686+
callback=DTCLICommand.validate_after_or_before_input,
687+
),
688+
before: str = typer.Option(
689+
None,
690+
"--before",
691+
help="The end of the query window in seconds, relative to the current time, inclusive",
692+
callback=DTCLICommand.validate_after_or_before_input,
693+
),
694+
fromBeginning: bool = typer.Option(
695+
None,
696+
"-fb",
697+
"--frombeginning",
698+
help="Requires a sessionID. When used with a new session ID, returns the first hour of data in the time window (rather than the last). Returns an error if the session ID already exists",
699+
),
700+
headers: bool = typer.Option(
701+
False,
702+
"--headers",
703+
help="Adds a header to the first line of response when text/csv is set in header parameters",
704+
),
705+
):
706+
DTCLICommand.run(name=c.FEEDS_IPRISK, params=ctx.params)

‎domaintools/cli/constants.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,5 @@
4848
FEEDS_DOMAINRDAP = "domainrdap"
4949
FEEDS_DOMAINDISCOVERY = "domaindiscovery"
5050
FEEDS_REALTIME_DOMAIN_RISK = "realtime_domain_risk"
51+
FEEDS_IPHOTLIST = "iphotlist"
52+
FEEDS_IPRISK = "iprisk"

‎domaintools/cli/utils.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,12 @@ def _iris_investigate_helptext():
6565
c.FEEDS_NAD: "Returns back newly active domains feed.",
6666
c.FEEDS_NOD: "Returns back newly observed domains feed.",
6767
c.FEEDS_NOH: "Returns back newly observed hosts feed.",
68-
c.FEEDS_DOMAINHOTLIST: "Returns domaint hotlist feed.",
68+
c.FEEDS_DOMAINHOTLIST: "Returns domain hotlist feed.",
6969
c.FEEDS_DOMAINRDAP: "Returns changes to global domain registration information, populated by the Registration Data Access Protocol (RDAP).",
7070
c.FEEDS_DOMAINDISCOVERY: "Returns new domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties.",
7171
c.FEEDS_REALTIME_DOMAIN_RISK: "Returns realtime domain risk information for apex-level domains, regardless of observed traffic.",
72+
c.FEEDS_IPHOTLIST: "Returns ip hotlist feed.",
73+
c.FEEDS_IPRISK: "Returns ip risk feed.",
7274
}
7375

7476

‎domaintools/constants.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,10 @@ class OutputFormat(Enum):
3838
"real-time-domain-risk-(s3)",
3939
"real-time-domain-discovery-feed-(api)",
4040
"real-time-domain-discovery-feed-(s3)",
41+
"real-time-ip-hotlist-(api)",
42+
"real-time-ip-hotlist-(s3)",
43+
"real-time-ip-risk-(api)",
44+
"real-time-ip-risk-(s3)",
4145
]
4246

4347
RTTF_PRODUCTS_CMD_MAPPING = {
@@ -55,6 +59,10 @@ class OutputFormat(Enum):
5559
"real-time-domain-risk-(s3)": "realtime_domain_risk",
5660
"real-time-domain-discovery-feed-(api)": "domaindiscovery",
5761
"real-time-domain-discovery-feed-(s3)": "domaindiscovery",
62+
"real-time-ip-hotlist-(api)": "iphotlist",
63+
"real-time-ip-hotlist-(s3)": "iphotlist",
64+
"real-time-ip-risk-(api)": "iprisk",
65+
"real-time-ip-risk-(s3)": "iprisk",
5866
}
5967

6068
SPECS_MAPPING = {

0 commit comments

Comments
 (0)