use a different msgpack lib?

[justinmk]

@chemzqm noticed that msgpack-lite is very slow.

msgpack-js msgpack5 might be a good choice instead of msgpack-lite.

According to this it's comparable in performance to the official lib. That comparison doesn't benchmark msgpack-lite. Edit: However, that post is 3 years old, and this discussion and the commit log suggest that msgpack-js is out of date.

msgpack5 looks like the only maintained msgpack lib for JS.

Can someone try out msgpack-js msgpack5 ?

[justinmk]

cc @bryphe

[chemzqm]

Looks like there's some number exceeded and cause V8 slow, it only takes 184ms for transfer 10k lines of code, but 60+ seconds for 200k lines of code.

PS: my processor is 2.6 GHz Intel Core i7

[billyvg]

@chemzqm if you can submit a PR with a benchmark, that would help

[billyvg]

I used these benchmarks when deciding to use msgpack-lite https://github.com/endel/msgpack-benchmark, but using a more real world benchmark would be better.

[chemzqm]

I did a benchmark for 330k lines of JSON, but I found that msgpack is fast enough,
there's other thing that block.

The result:

Name:msgpack Times:100 Average:140ms
Name:msgpack-javascript Times:100 Average:296ms
Name:msgpack-js-v5 Times:100 Average:142ms
Name:msgpack-lite Times:100 Average:246ms
Name:msgpack5 Times:100 Average:264ms
Name:notepack Times:100 Average:103ms
Name:notepack.io Times:100 Average:105ms

notepack.io could be 2 ~ 4 times faster than msgpack-lite on decode.

[chemzqm]

I think I got the reason, the chunk in nodeJS stream is too small, only 8kb.

2018-06-01T13:59:33.804Z - error: 8192
2018-06-01T13:59:33.832Z - error: 8192
2018-06-01T13:59:33.859Z - error: 8192
2018-06-01T13:59:33.884Z - error: 8192
2018-06-01T13:59:33.912Z - error: 8192
2018-06-01T13:59:33.937Z - error: 8192
2018-06-01T13:59:33.966Z - error: 8192
2018-06-01T13:59:33.991Z - error: 8192
2018-06-01T13:59:34.019Z - error: 8192
2018-06-01T13:59:34.044Z - error: 8192
2018-06-01T13:59:34.073Z - error: 8192
2018-06-01T13:59:34.098Z - error: 8192
2018-06-01T13:59:34.127Z - error: 8192
2018-06-01T13:59:34.155Z - error: 8192
2018-06-01T13:59:34.181Z - error: 8192
2018-06-01T13:59:34.209Z - error: 8192
2018-06-01T13:59:34.234Z - error: 8192
2018-06-01T13:59:34.264Z - error: 8192
2018-06-01T13:59:34.289Z - error: 8192
2018-06-01T13:59:34.319Z - error: 8192
2018-06-01T13:59:34.343Z - error: 8192
2018-06-01T13:59:34.372Z - error: 8192
2018-06-01T13:59:34.397Z - error: 8192
2018-06-01T13:59:34.429Z - error: 8192
2018-06-01T13:59:34.458Z - error: 8192
2018-06-01T13:59:34.489Z - error: 8192
2018-06-01T13:59:34.516Z - error: 8192
2018-06-01T13:59:34.545Z - error: 8192

[chemzqm]

It's a problem of piping a buffer stream with low highWaterMark to a object mode stream.
It could be solved if we can change the hightWaterMark of the buffer stream, but it's not possible to change hightWaterMark of socket stream, so we can make use of a transform stream to did the trick #79

[chemzqm]

Another solution it make neovim send content-length header on request, like what tsserver https://github.com/Microsoft/TypeScript/wiki/Standalone-Server-(tsserver) did.
But that would require lots of work.

[bryphe]

Sorry I'm late to the party here, definitely interested in the results. We use msgpack-lite for Oni today, and the msgpack decode is our critical path for performance / responsiveness - would definitely be interested if there is a faster option 👍

[billyvg]

Closed by #79