Skip to content

[SPARK-48685][ML] Fix MinHashLSH to handle empty sparse vectors without crashing - #58417

Open
ngtanvan wants to merge 5 commits into
apache:masterfrom
ngtanvan:SPARK-48685
Open

[SPARK-48685][ML] Fix MinHashLSH to handle empty sparse vectors without crashing#58417
ngtanvan wants to merge 5 commits into
apache:masterfrom
ngtanvan:SPARK-48685

Conversation

@ngtanvan

Copy link
Copy Markdown

What changes were proposed in this pull request?

Fix MinHashLSHModel.hashFunction to return Array.empty[Vector] when encountering empty sparse vectors (0 non-zero entries) instead of throwing an unhandled IllegalArgumentException.

Why are the changes needed?

When pipelines combining CountVectorizer and MinHashLSH process empty sparse vectors (e.g. text containing terms outside vocabulary), MinHashLSH threw java.lang.IllegalArgumentException: requirement failed: Must have at least 1 non zero entry., crashing the entire Spark job.
Returning an empty vector array allows posexplode in approxSimilarityJoin to cleanly skip empty vectors without crashing.

Does this PR introduce any user-facing change?

No breaking change. It fixes an unexpected crash when transforming or joining empty vectors with MinHashLSH.

How was this patch tested?

Updated unit test MinHashLSHSuite ("hashFunction: empty vector") to verify returning empty vector array, and verified all 15 tests in MinHashLSHSuite pass cleanly.

@ngtanvan

Copy link
Copy Markdown
Author

Hi @HyukjinKwon , @srowen

Could you please help take a look or trigger CI for this PR when you have a moment?
This PR fixes [SPARK-48685] where MinHashLSHModel.hashFunction throws an unhandled IllegalArgumentException when encountering empty sparse vectors. All 15 unit tests in MinHashLSHSuite have been updated and pass cleanly.
Thank you very much for your time and assistance!

hashValues.map(Vectors.dense(_))
}


Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove the unnecessary whitespace only change.

@tdcmeehan

tdcmeehan commented Sep 1, 2026

Copy link
Copy Markdown

Would it make sense to follow other Spark ML transformers and expose this through handleInvalid with error (default), skip, and keep? keep could produce the proposed empty hash array. This preserves existing behavior while letting users opt into tolerant handling, consistent with VectorSizeHint and VectorAssembler.

@ngtanvan

ngtanvan commented Sep 1, 2026

Copy link
Copy Markdown
Author

Thank you @tdcmeehan for the great suggestion! Adding handleInvalid (error, skip, keep) makes a lot of sense to maintain backward compatibility while providing flexibility consistent with other Spark ML transformers.

I will work on updating the PR to implement HasHandleInvalid. Thanks again for the review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants