Skip to content

fix(compat/uniqWith): performance improvements and lodash compatibility fixes - #2115

Closed
ssi02014 wants to merge 7 commits into
toss:mainfrom
ssi02014:perf/uniqWith
Closed

ssi02014 wants to merge 7 commits into
toss:mainfrom
ssi02014:perf/uniqWith

Conversation

@ssi02014

@ssi02014 ssi02014 commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

es-toolkit/compat's uniqWith delegated to es-toolkit's uniqWith. That delegation existed only so compat could reuse the dedupe loop, and it cost an Array.from copy of the input plus a wrapper closure whose sole job was to swap the comparator arguments into lodash's documented (candidate, kept) order.

This PR inlines an indexed for loop into compat's uniqWith that breaks on the first match and calls the comparator in lodash's order directly, so both the copy and the closure disappear. es-toolkit's own uniqWith is left untouched — the change is confined to es-toolkit/compat.

Dropping Array.from also fixes a lodash incompatibility, because Array.from prefers Symbol.iterator when the input defines one while lodash's baseUniq always reads array[index]. Every path through compat's uniq / uniqWith went through Array.from, so all three — uniq, uniqWith without a comparator, and uniqWith with a comparator — disagreed with lodash on an array-like whose iterator yields something other than its indexed elements. Reading by index fixes all three at once, and uniq now fills its Set by index for the same reason, which also removes the intermediate array.

Four new tests cover this, two in uniq.spec.ts and two in uniqWith.spec.ts; all four fail without the change.


Change phases & benchmark

The screenshots below come from the original measurement run. only the es-toolkit/compat/* rows are relevant to this PR.
benchmark environments differ from machine to machine, and the numbers carry some measurement noise.

Before

스크린샷 2026-09-23 오후 2 55 16

STEP 1 — compat/uniqWith stops delegating

compat's uniqWith inlines the dedupe loop instead of calling es-toolkit's. Three costs go away at once:

  1. the Array.from(arr) copy of the input,
  2. the wrapper closure that reordered the comparator arguments,
  3. the per-item closure that Array.prototype.every allocated inside the shared loop.

The inlined loop reads arr[i] directly, calls the comparator as lodash documents it — (candidate, kept) — and breaks on the first match.

Reading arr[i] instead of Array.from(arr) is also a behavior fix for the comparator path: Array.from prefers Symbol.iterator, lodash does not. See STEP 2 for the full explanation — the two paths share the same root cause.

// src/compat/array/uniqWith.ts — before
import { uniqWith as uniqWithToolkit } from '../../array/uniqWith.ts';
// ...

export function uniqWith<T>(arr: ArrayLike<T> | null | undefined, comparator?: Comparator<T>): T[] {
  if (!isArrayLike(arr)) {
    return [];
  }

  if (typeof comparator !== 'function') {
    return uniqToolkit(Array.from(arr));
  }

  // `es-toolkit`'s `uniqWith` invokes the comparator as `(kept, candidate)`, but
  // lodash documents and invokes it as `(candidate, kept)`. Swap the arguments so
  // that asymmetric comparators behave the same as in lodash.
  return uniqWithToolkit(Array.from(arr), (kept, candidate) => comparator(candidate, kept));
}
// src/compat/array/uniqWith.ts — after
export function uniqWith<T>(arr: ArrayLike<T> | null | undefined, comparator?: Comparator<T>): T[] {
  if (!isArrayLike(arr)) {
    return [];
  }

  if (typeof comparator !== 'function') {
    return uniqToolkit(arr);
  }

  const result: T[] = [];

  for (let i = 0; i < arr.length; i++) {
    const item = arr[i];
    let isUniq = true;

    for (let j = 0; j < result.length; j++) {
      if (comparator(item, result[j])) {
        isUniq = false;
        break;
      }
    }

    if (isUniq) {
      result.push(item);
    }
  }

  return result;
}

benchmark

compat/uniqWith now improves substantially on small arrays(+138%), coming out fastest of the three. Large arrays also gained roughly +59%.

스크린샷 2026-09-23 오후 2 56 53

Improvement rate

compat/uniqWith(small array): 14,633,424.21 -> 34,854,729,44 (+138.19%)
compat/uniqWith(large array): 5,9887 -> 9.5256 (+59.06%)

STEP 2 — compat/uniq reads array-likes by index

STEP 1 fixed the comparator path, but left the no-comparator path behind: it delegates to compat's uniq, which still went through Array.from.

The difference is observable, because Array.from prefers Symbol.iterator while lodash's baseUniq always reads array[index]:

const arrayLike = {
  length: 3,
  0: 1,
  1: 1,
  2: 2,
  *[Symbol.iterator]() {
    yield 9;
    yield 9;
    yield 8;
  },
};

_.uniq(arrayLike);                          // [1, 2]
_.uniqWith(arrayLike, (a, b) => a === b);   // [1, 2]

compat.uniq(arrayLike);                          // [9, 8]  <- before
compat.uniqWith(arrayLike);                      // [9, 8]  <- before (no comparator, fixed in STEP 2)
compat.uniqWith(arrayLike, (a, b) => a === b);   // [9, 8]  <- before (comparator, fixed in STEP 1)

So uniq fills its Set by index as well. This removes the intermediate array that Array.from allocated, and makes both uniq and the no-comparator path of uniqWith agree with lodash.

// src/compat/array/uniq.ts — after
export function uniq<T>(arr: ArrayLike<T> | null | undefined): T[] {
  if (!isArrayLike(arr)) {
    return [];
  }

  const set = new Set<T>();

  for (let i = 0; i < arr.length; i++) {
    set.add(arr[i]);
  }

  return [...set];
}

An Array subclass that overrides Symbol.iterator diverged in the same way and is now fixed too. Two tests were added to src/compat/array/uniq.spec.ts and two to src/compat/array/uniqWith.spec.ts (each covering both the no-comparator and the comparator path); all four fail without these changes.

benchmark

  • as-is
스크린샷 2026-09-23 오후 3 02 50 - to-be 스크린샷 2026-09-23 오후 3 01 50

Improvement rate

compat/uniq(small array): 13,503,607.29 -> 18,061,005.64 (+33.75%)
compat/uniq(large array): 4,018.09 -> 8,942.56  (+122.56%)

Compatibility check

Verified against lodash@4.18.1

// input1
const arrayLike = {
  0: 1,
  1: 1,
  2: 2,
  length: 3,
  *[Symbol.iterator]() {
    yield 9;
    yield 9;
    yield 8;
  },
};

// input2
class OverriddenIterator extends Array {
  *[Symbol.iterator](): Generator<string, undefined> {
    yield 'x';
    yield 'x';
    yield 'y';
    return undefined;
  }
}

const array = new OverriddenIterator();
array.push(1, 1, 2);
call lodash compat (before) compat (after)
uniq(arrayLike) [1, 2] [9, 8] [1, 2]
uniqWith(arrayLike) [1, 2] [9, 8] [1, 2]
uniqWith(arrayLike, (a, b) => a === b) [1, 2] [9, 8] [1, 2]
uniq(Array subclass overriding the iterator) [1, 2] ['x', 'y'] [1, 2]
uniqWith(Array subclass overriding the iterator) [1, 2] ['x', 'y'] [1, 2]
comparator arguments for uniqWith([1, 2], fn) (2, 1) (2, 1) (2, 1)

The comparator is still invoked in lodash's (candidate, kept) order, so asymmetric comparators are unaffected.


Conclusion

compat/uniqWith(small array): 14,633,424.21 -> 34,854,729,44 (+138.19%)
compat/uniqWith(large array): 5,9887 -> 9.5256 (+59.06%)

compat/uniq(small array): 13,503,607.29 -> 18,061,005.64 (+33.75%)
compat/uniq(large array): 4,018.09 -> 8,942.56  (+122.56%)

Dropping Array.from is what makes both the performance and the lodash-compatibility fix possible, so the two changes share one root cause rather than being independent. es-toolkit's own uniqWith is unchanged, so no bundle-size snapshot moves.

@vercel

vercel Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
es-toolkit Error Error Oct 8, 2026 9:32am UTC

Request Review

@ssi02014 ssi02014 changed the title perf(uniqWith): performance improvements to uniqWith perf(compat/uniqWith): performance improvements to uniqWith Sep 22, 2026
@ssi02014 ssi02014 changed the title perf(compat/uniqWith): performance improvements to uniqWith fix(compat/uniqWith): performance improvements and lodash compatibility fixes Sep 22, 2026
@raon0211

raon0211 commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the thorough write-up and for verifying everything against lodash 4.18.1. The numbers and the behavior difference both check out on my side.

We're closing this one, though. The lodash-compatibility part of this PR falls under the rule for compat behavior fixes, which we updated on Sep 30 (#2125), after this PR was opened: https://github.com/toss/es-toolkit/blob/main/.github/CONTRIBUTING.md#fixing-behavior-that-differs-from-lodash

A compat difference is accepted only when both of the following hold:

  1. The reproduction compiles against @types/lodash in strict mode. Yours does.
  2. Real code in a public repository makes the call. This is what's missing. An array-like whose Symbol.iterator yields something other than its indexed elements, or an Array subclass that overrides its iterator to do the same, isn't an input we've seen any real project pass to uniq or uniqWith.

As the document says, we close when either condition is missing, even when the fix itself is correct.

If you do find a public project that passes such an input, please reopen with the permalink. Alternatively, the performance side of this change stands on its own: if you'd like to resubmit it as a pure perf PR for compat/uniq and compat/uniqWith with the benchmark results and without the iterator-specific tests, we'd be happy to review that.

@raon0211 raon0211 closed this Oct 9, 2026
@ssi02014

Copy link
Copy Markdown
Contributor Author

@raon0211 #2139 I have separated the performance-related tasks into a separate file and resubmitted them.

This branch had an error being deployed

1 failed deployment
Preview — ddd7ae10 Deployed Oct 8, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants