[merces]

This PR is bold. :)

It adds an option enable case-insensitive search (:set ignorecase, like in Vim) and adds to support hex pattern matches. This first version uses YARA-X for pattern matching, but will check how bmatcher performs before merging.

YARA-X pros: negation, unlimited jumps, can use string modifiers with text matching if needed (wide, nocase, fullword, xor, etc). cons: release binary went from 5.9MB (15.37s to compile in my machine) to 25MB (1m13s).

bmatcher pros: both binary size and compilation time are nearly the same. However, a simple FEC01F803F??07 search on a 3.8GB file took ~22s while YARA-X completed in ~2s.

PS.: the case-insensitive search is done with the new find_nocase and rfind_nocase functions that use memchr. So, no pattern matching is needed.

[glazari]

Wow this is surprising, both the results (10x speed-up) and the price (5x compile time and 5x binary size).

I did a little investigation and apparently the reason yara-x is so big is because it bundles in a wasm-runtime that has a JIT compiler. And that is a good chunk of the binary.

The regex crate has a bytes mode regex::bytes. We already use it for searching strings, so if we use it for the bytes it likely wouldn't increase the binary size much.

Not sure about the speed though. regex is pretty fast, so it could be competitive.
Do you have some examples of the kind of queries you were testing with? and how you got your 3.8Gb file to test with?

As for syntax, I have never used yara, but it seems to have a pretty clean syntax for searching bytes, the regex crate seems to require using \x escape on all bytes of the pattern, which seems pretty annoying. But if that is the only downside of regex, maybe we could implement a translation layer between yara syntax and regex syntax.

Just an idea

[merces]

Wow this is surprising, both the results (10x speed-up) and the price (5x compile time and 5x binary size).
I did a little investigation and apparently the reason yara-x is so big is because it bundles in a wasm-runtime that has a JIT compiler. And that is a good chunk of the binary.

Thanks for checking! \o/

The regex crate has a bytes mode regex::bytes. We already use it for searching strings, so if we use it for the bytes it likely wouldn't increase the binary size much.
Not sure about the speed though. regex is pretty fast, so it could be competitive.

This is a great idea. If we can do this without introducing a new crate, it'd be really cool.

Do you have some examples of the kind of queries you were testing with? and how you got your 3.8Gb file to test with?

I opened a VM disk and found a byte sequence near the end of the file (G, then O as the end of the was filled with zeroes):

then went back to the beginning of the file with g and searched for FEC01F803F??07 in SEARCH/HEX mode (forward).

As for syntax, I have never used yara, but it seems to have a pretty clean syntax for searching bytes, the regex crate seems to require using \x escape on all bytes of the pattern, which seems pretty annoying. But if that is the only downside of regex, maybe we could implement a translation layer between yara syntax and regex syntax.

Yeah, yara's syntax is clean and it's used everywhere in the cybersecurity field, so it'd be nice if we can use something similar. :)

I like the translation layer idea. If the performance of regex::bytes is acceptable, I think it's a promising path. :)

Also tried a few different configurations here (yara-x):

Config	Release binary size	Compilation time
lto = "fat" + strip = "symbols"	16MB	2m36s
lto = "thin" + strip = "symbols"	18MB	1m20s

[merces]

I tested regex::bytes and it looks very promising indeed! /(?s-u)\xFE\xC0\x1F\x80\x3F.\x07 was faster than yara. :)

We'd lose a few things such as nibble-level matching, but it sounds much better than increasing both compilation time and binary size a lot.

EDIT: I forgot to disable caching. regex actually performed similar to yara-x. See #71.

[merces]

I have something :) #71

[glazari]

I tested regex::bytes and it looks very promising indeed! /(?s-u)\xFE\xC0\x1F\x80\x3F.\x07 was faster than yara. :)

We'd lose a few things such as nibble-level matching, but it sounds much better than increasing both compilation time and binary size a lot.

Faster than yara? very nice then!

[merces]

#71 prevailed.