Automatically Generating URLs to Public API Documents & Reducing Hallucination (Invalid URLs)

[in-c0]

Issue

The current method involves manually prompting ChatGPT with the following:

"Generate the URLs pointing to the appropriate page of popular public APIs based on this table"
(Refer to api-docs-urls.csv for the table format)

Then, I repeatedly request "10 more" results and merge the tables together. However, this approach often leads to hallucinations (invalid URLs) or incorrect data. This reduces the utility of the generated table and creates additional overhead in validation and correction.

🤔

Proposed Solution

1 - One agent generates and maintains a list of "names" of APIs - hopefully this will lead to less hallucination

2 - One agent tries to find all candidate URLs to the API document page - based on the API names from step 1.

3 - One agent validates the URLs (head requests) so that only live and reachable URLs make it into the final table

4 - One agent connects these candidate URLs to their "best matches" (e.g. Privacy, TOS, etc) (chances are that several APIs exist under a single known name, e.g. OpenAI API -> OpenAI GPT-4 API or OpenAI Embedding API ... in that case, we might have to generate new columns for the sub-APIs)

It's a vague pipeline yet, I'll try to update it as I begin to work on it sometime soon!

[pradhanhitesh]

@in-c0, any updates on the agents? I may have found a lighter and simpler way to navigate around this.

Proposed workflow

1. Using Custom Search API (by Google), we can get results (max 10 items per page, index through multiple pages) from Google in JSON format.
2. Save each item's Search Title and Search Link. Using requests, fetch the robots.txt file and implement the same logic of isScrapAllowed here.
3. Then fetch sitemap.xml for each link and look for other associated links, such as Terms and Conditions, using the Gemini-1.5-flash model, which is free.

I have worked on this up until now. I am in the process of making it lighter and more scalable. Let me know your thoughts?

[in-c0]

@in-c0, any updates on the agents? I may have found a lighter and simpler way to navigate around this.

Proposed workflow

1. Using Custom Search API (by Google), we can get results (max 10 items per page, index through multiple pages) from Google in JSON format.
2. Save each item's Search Title and Search Link. Using requests, fetch the robots.txt file and implement the same logic of isScrapAllowed here.
3. Then fetch sitemap.xml for each link and look for other associated links, such as Terms and Conditions, using the Gemini-1.5-flash model, which is free.

I have worked on this up until now. I am in the process of making it lighter and more scalable. Let me know your thoughts?

I haven't had the time to work on the agents yet. This sounds like a promising suggestion! Let's see where we can get upto with this pipeline.
It seems we're allowed to search upto 100 queries per day for free with Custom Search API. I think that is enough for a proof of concept, but I'm not sure if we'll be able to afford the price at scale.

[in-c0]

Update: I've found a pretty useful tool that packs your entire repository into a single, AI-friendly file - https://github.com/yamadashy/repomix

I've tested with ChatGPT and it works pretty well. I'm not sure if it's able to expose all known endpoints like postman does,
but It already achieves what I was hoping to achieve with UpdAPI for public repositories hosted publically on github. Though there's still a plenty of public API docs out there that aren't hosted on github.

just adding FYI @pradhanhitesh @ishaangrewal

[pradhanhitesh]

I will take a look at this, maybe on the weekend. @in-c0