[merces]

See #70, but uses regex::bytes (@glazari's idea).

The translator is still basic. It supports ?? only.

[glazari]

Very nice!

just out of curiosity, can you share how it compares in terms of the numbers you mentioned on the other MR:

• compilation time
• binary size
• search time

[merces]

I'm glad you asked because I forgot to consider kernel caching when running previous tests with it. =/

I've tested again and realized it's not much faster than yara-x, but still achieves the same performance without the burden of longer compilation times and binary size.

Search methodology

I've added this code surrounding the call to search():

let before = Instant::now();
// search()
eprintln!("Search took {:.2?}", before.elapsed());

Then ran dz6 using this command to drop the kernel cache:

$ echo 3 | sudo tee -a /proc/sys/vm/drop_caches && cargo r --release -- ~/vmware/Windows\ 11\ x64/Windows\ 11\ x64-s001.vmdk 2>err.log

Search pattern: FEC01F803F??07
File size: 3.8G
Target sequence: FEC01F803FF007 at F1DAD60B (a few bytes from EOF)

Results

Branch	Compilation time	Binary size (release)	Search time (3 runs)
feature/search-with-regex	15.43s	5.9MB	8.32s, 8.28s, 8.37s (avg: 8.32s)
feature/search-with-yara	1m10s	25MB	8.70s, 8.32s, 8.41s (avg: 8.47s)

I'm not sure we can get any faster than this. I tried to search for FEC01F803FF007 (dz6 uses memchr::memmem::find() since there is no pattern) and results were similar (8s range).

Ideas for the future: could we spawn two threads, each one searching over half the file? Just something that came to mind. :)

[glazari]

If kernel caching made a big difference then that indicates the search is to some degree bottlenecked by disk IO.

If that is really the case, then it's not a given that multithreading would help.

But the strange part is that your numbers say ~8s for ~4Gb, so roughly 500Mb/s, and ssd these days can do something like 3-5Gb/s so it's a bit odd to have it saturated at 500Mb/s

Maybe there is something else going on.

Im not very experienced with this kind of optimization, so just doing some napkin math :🙃

But I think it's a good idea to try the paralelization, some care is needed about finding matches in the boarder of where we split the file though

[merces]

I've dropped the support for [] because they were defining ranges instead of jumps.

[merces]

Im not very experienced with this kind of optimization, so just doing some napkin math :🙃

me neither, but you raised an interesting point. I'll run more tests when I have time.

But I think it's a good idea to try the paralelization, some care is needed about finding matches in the boarder of where we split the file though
Cool. We can try that in the future.

Thank you!