[SPARK-48685][ML] Fix MinHashLSH to handle empty sparse vectors without crashing - #58417
[SPARK-48685][ML] Fix MinHashLSH to handle empty sparse vectors without crashing#58417ngtanvan wants to merge 5 commits into
Conversation
|
Hi @HyukjinKwon , @srowen Could you please help take a look or trigger CI for this PR when you have a moment? |
| hashValues.map(Vectors.dense(_)) | ||
| } | ||
|
|
||
|
|
There was a problem hiding this comment.
Please remove the unnecessary whitespace only change.
|
Would it make sense to follow other Spark ML transformers and expose this through |
|
Thank you @tdcmeehan for the great suggestion! Adding I will work on updating the PR to implement |
…ror, keep, and skip
What changes were proposed in this pull request?
Fix
MinHashLSHModel.hashFunctionto returnArray.empty[Vector]when encountering empty sparse vectors (0 non-zero entries) instead of throwing an unhandledIllegalArgumentException.Why are the changes needed?
When pipelines combining
CountVectorizerandMinHashLSHprocess empty sparse vectors (e.g. text containing terms outside vocabulary),MinHashLSHthrewjava.lang.IllegalArgumentException: requirement failed: Must have at least 1 non zero entry., crashing the entire Spark job.Returning an empty vector array allows
posexplodeinapproxSimilarityJointo cleanly skip empty vectors without crashing.Does this PR introduce any user-facing change?
No breaking change. It fixes an unexpected crash when transforming or joining empty vectors with
MinHashLSH.How was this patch tested?
Updated unit test
MinHashLSHSuite("hashFunction: empty vector") to verify returning empty vector array, and verified all 15 tests inMinHashLSHSuitepass cleanly.