-
Notifications
You must be signed in to change notification settings - Fork 1.8k
perf: batch duplicate marking in batch deduplication#14458
Open
valentijnscholten wants to merge 3 commits intoDefectDojo:devfrom
Open
perf: batch duplicate marking in batch deduplication#14458valentijnscholten wants to merge 3 commits intoDefectDojo:devfrom
valentijnscholten wants to merge 3 commits intoDefectDojo:devfrom
Conversation
Member
Summary
Deduplication matching happens in batches. But the "marking" of findings as duplicates still happened one by one. For duplicate heavy instances this is a major slow down and resource hog. This PR batches the "marking" of duplicates to reduce the complexity from O(2N) to O(1).
set_duplicatenow accepts asave=Falsekeyword argument; when false, field changes are applied to the model instance in memory without hitting the database- Added
_flush_duplicate_changeshelper that bulk-updates all modified duplicate findings in a singleFinding.objects.bulk_updatecall -- one round-trip regardless of batch size, no saves on original findings - All four
_dedupe_batch_*algorithm functions (hash_code,unique_id,uid_or_hash,legacy) now collect duplicate pairs and flush via_flush_duplicate_changesinstead of saving per-pair - Transitive duplicates (findings that previously pointed to
new_findingas their original, now re-pointed to the true original) are captured throughset_duplicate's return value and included in the bulk update - Added
Finding.DEDUPLICATION_FIELDSconstant (allowlist) used with.only()inget_finding_models_for_deduplicationto skip loading unused columns for the findings being processed - Added
Finding.DEDUPLICATION_DEFERRED_FIELDSconstant (denylist of large text columns) used with.defer()inbuild_candidate_scope_querysetto skip loading unused columns for the candidate pool - Performance test query count updated: second import with duplicates drops from 236 - 183 queries (-53)
Instead of saving each duplicate finding individually, collect all
modified findings during a batch deduplication run and flush them in
a single bulk_update call. Original (existing) findings are still
saved individually to preserve auto_now timestamp updates and
post_save signal behavior, but are deduplicated by id so each is
saved at most once per batch.
Reduces DB writes from O(2N) individual saves to 1 bulk_update +
O(unique originals) saves for a batch of N duplicates.
Performance test shows -23 queries on a second import with duplicates.
modified findings during a batch deduplication run and flush them in
a single bulk_update call. Original (existing) findings are still
saved individually to preserve auto_now timestamp updates and
post_save signal behavior, but are deduplicated by id so each is
saved at most once per batch.
Reduces DB writes from O(2N) individual saves to 1 bulk_update +
O(unique originals) saves for a batch of N duplicates.
Performance test shows -23 queries on a second import with duplicates.
Add Finding.DEDUPLICATION_FIELDS -- the union of all Finding fields
needed across every deduplication algorithm -- and apply it as an
only() clause in get_finding_models_for_deduplication.
This avoids loading large text columns (description, mitigation,
impact, references, steps_to_reproduce, severity_justification, etc.)
when loading findings for the batch deduplication task, reducing
data transferred from the database without affecting query count.
build_candidate_scope_queryset is intentionally excluded: it is also
used for reimport matching (which accesses severity, numerical_severity
and other fields outside this set) and applying only() there would
cause deferred-field extra queries.
needed across every deduplication algorithm -- and apply it as an
only() clause in get_finding_models_for_deduplication.
This avoids loading large text columns (description, mitigation,
impact, references, steps_to_reproduce, severity_justification, etc.)
when loading findings for the batch deduplication task, reducing
data transferred from the database without affecting query count.
build_candidate_scope_queryset is intentionally excluded: it is also
used for reimport matching (which accesses severity, numerical_severity
and other fields outside this set) and applying only() there would
cause deferred-field extra queries.
- Add Finding.DEDUPLICATION_DEFERRED_FIELDS constant listing large text
columns (description, mitigation, impact, references, etc.) that are
never read during deduplication or candidate matching.
- Apply .defer(*Finding.DEDUPLICATION_DEFERRED_FIELDS) in
build_candidate_scope_queryset to avoid loading those columns for the
potentially large candidate pool fetched per dedup batch.
Reduces deduplication second-import query count from 213 to 183 (-30).
columns (description, mitigation, impact, references, etc.) that are
never read during deduplication or candidate matching.
- Apply .defer(*Finding.DEDUPLICATION_DEFERRED_FIELDS) in
build_candidate_scope_queryset to avoid loading those columns for the
potentially large candidate pool fetched per dedup batch.
Reduces deduplication second-import query count from 213 to 183 (-30).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.