feat: Add skipRequest context helper

[janbuchar]

• closes Decide the public status of ContextPipelineInterruptedError before 4.0 #4141

[B4nan]

few things found from staff review, I'd address the first one at least

[barjin]

Thank you @janbuchar !

I agree with Martin's notes above, otherwise lgtm 👍

[barjin]

Thank you @janbuchar , few more questions, but otherwise lgtm.

Once these are resolved (or considered wontfix), it's good to go by me 👍

[janbuchar]

@B4nan @barjin thanks guys, I fixed the last batch. Give it a quick look please.

[barjin]

thank you @janbuchar , just one last edge case ⬇️

Once that's dealt with, let's merge

[barjin]

When onSkippedRequest times out, err here is still the ContextPipelineInterruptedError, right? Does it mean this ⬇️ will keep looping?

import { BasicCrawler, Configuration, serviceLocator } from '@crawlee/basic';
import { setTimeout } from 'node:timers/promises';

serviceLocator.setConfiguration(new Configuration({ internalTimeoutMillis: 100 }));
const crawler = new BasicCrawler({
    requestHandlerTimeoutSecs: 0.05,
    requestHandler: async ({ skipRequest }) => skipRequest(),
    onSkippedRequest: () => setTimeout(300),
});
await crawler.run(['https://example.com']);

Something like:

• onSkippedRequest timeouts
• we do not increment the counter because of the change here
• we process the request again, skip it again
• back to step one - infinite loop?

Maybe a timed-out onSkippedRequest should go the same way as a throwing one, i.e. become an ordinary failure that uses up the retry budget?

[janbuchar]

Wow, thanks! fedcf11