Where you call the PDF path, can I give the code a directory with many files in it?
The end goal is still one dataset.
Would it be better or worse to combine the files into a single file? In other words, combining the data as 1. pre-input, or 2. post-output.
EDIT: I'm already exceeding a token count with one file. I think it's limiting to 2048 per transformers limits. I'm curious how you were able to run a novel through it which would have ~100k tokens.
Where you call the PDF path, can I give the code a directory with many files in it?
The end goal is still one dataset.
Would it be better or worse to combine the files into a single file? In other words, combining the data as 1. pre-input, or 2. post-output.
EDIT: I'm already exceeding a token count with one file. I think it's limiting to 2048 per transformers limits. I'm curious how you were able to run a novel through it which would have ~100k tokens.