Lnav initialization is much slower since addition of the digestible library

[errge]

lnav version
Bisected and the commit causing the issue is: ed9bf70

Describe the bug
Since addition of the digestible library, the startup time of lnav became much slower with the initial indexing.

I don't really need the advanced histogram feature, could we make this indexing optional and can we add an option to disable the usage of the library?

To Reproduce
Have a directory with a couple log files (3K lines altogether).

Start up a musl static binary built from ed9bf70 -> initialization time is around 5-10 seconds

Start up a musl static binary built from 14a1082 (one commit before) -> initialization is instant.

[tstack]

Have a directory with a couple log files (3K lines altogether).

A 3GB web access log file with 10M lines loads in less than 10 seconds on my systems. If 3K lines is taking this long, it seems like there is something wildly wrong. Can you please share the log format JSON file, run lnav with -d /tmp/lnav-debug.log, and then attach the debug log file. Thanks!

[errge]

Please find the complete reproduction toolchain here: https://github.com/CodeExpertETH/lnav/tree/repro-1738

Sorry, I made a typo in my original report, I meant 3M lines. It's on a slow remote VM, where lnav was fast enough before, and now it's not bearable without downgrading back before to the bisected commit.

To clearly reproduce the situation please use the attached toolchain the following way:

• check out the linked repo and switch to the repro-1738 branch
• use 00-generate.sh to generate the test log files
• use 01-fast.sh to reproduce the fast startup
• use 02-slow.sh to reproduce the slow startup

On my machine slow is ~27 seconds, while fast ~3 seconds until the final UI with all messages shown. That's a 9x increase in loading time, that prompted my log viewing colleagues to report to me that something is actually wrong.

It would be amazing if this new extra data structure build could be moved to the background (so UI is usable asap) or could be opted out from.

[tstack]

I can replicate the slowdown. Things seem a little better on the top-of-tree, but still slow. Going to try a couple things and get back.

[tstack]

There were a bunch of extra allocations that were dragging things down when there were a lot of files. Avoiding those was simple enough and seems to have restored the original performance. Thanks for the report!

[tstack]

I think this is fixed now, closing.