Summary

Major upgrade migrating from NEST (Elasticsearch.Net 7.x) to Elastic.Clients.Elasticsearch (official ES 8.x/9.x client). This is a breaking change release (v8.0) that replaces the entire Elasticsearch client layer and serialization stack.

Depends on FoundatioFx/Foundatio.Parsers#84

Core Changes

• Elasticsearch Client: IElasticClient (NEST) -> ElasticsearchClient (Elastic.Clients.Elasticsearch)
• Serialization: Newtonsoft.Json (JToken/JObject) -> System.Text.Json (JsonNode/JsonElement) throughout
• JsonPatch: Full port from Newtonsoft to System.Text.Json with RFC 6902 compliance
• Query DSL: All query builders ported from QueryContainer/SearchDescriptor to Query/SearchRequestDescriptor
• Index Configuration: All mapping/settings APIs ported to new descriptor types
• Docker: Elasticsearch/Kibana images updated to 9.3.1

Breaking API Changes

All breaking changes are direct 1:1 type replacements required by the client migration:

Known Limitation: PartialPatch Cannot Set Fields to Null

This is a behavioral change from NEST. The DefaultSourceSerializer in Elastic.Clients.Elasticsearch sets JsonIgnoreCondition.WhenWritingNull by default, which means any property explicitly set to null in a PartialPatch document is silently stripped from the serialized JSON before it reaches Elasticsearch. The field retains its original value instead of being set to null.

Root cause: Decompiled source of DefaultSourceSerializerOptionsProvider.MutateOptions in Elastic.Clients.Elasticsearch 9.3.1 confirms this is intentional:

Why we cannot work around it globally: Setting JsonIgnoreCondition.Never on the SourceSerializer (the obvious fix) is not viable because:

1. It causes all null properties on all documents to be serialized on every index/update operation, bloating payloads and potentially overwriting fields in multi-writer scenarios
2. Some Elastic request types with object-typed fields get routed through the SourceSerializer and fail with Never (as reported in the upstream issue)
3. Foundatio.Repositories is a library -- changing a global serializer default would silently affect all consumer documents

Upstream tracking: elastic/elasticsearch-net#8763 -- still open. We have commented on the issue with our specific use case.

Workarounds for consumers who need to set a field to null:

• Use ScriptPatch: new ScriptPatch("ctx._source.companyName = null;")
• Use JsonPatch (RFC 6902): new JsonPatch(new PatchDocument(new ReplaceOperation { Path = "companyName", Value = null }))

A regression test (PartialPatchAsync_WithNullField_RetainsOriginalValue) documents this behavior and will flip automatically once the upstream issue is resolved.

Quality & Robustness Improvements

• Zero build warnings (down from 190): Fixed all xUnit1051, CS8002, AsyncFixer02 warnings
• 441 total tests passing (96 core + 345 ES integration, 0 skipped, 0 failed)
• All previously-skipped tests unskipped and passing (7 tests that were skipped on main have been fixed and re-enabled)
• Fixed query timeout format to use ES duration syntax (30s) instead of .NET TimeSpan format
• Fixed DoubleSystemTextJsonConverter.Read() (was throwing NotImplementedException)
• Added NotSupportedException for JSONPath ($-prefixed) paths in patch operations with clear error message
• Added null guard in Operation.Build() for malformed patch JSON
• Strict validation in PatchDocument.Load() -- now throws on non-object elements
• Fixed null-conditional constant conditions in JsonDiffer, LoggerExtensions, ElasticReindexer
• Fixed potential null dereferences in test assertions (split Assert.NotNull(x?.Y) into proper two-step assertions)
• Fixed GetAliasIndexCount to only treat HTTP 404 as "no aliases" and throw on real failures
• Added PatchDocument.Add/Replace convenience overloads for string, int, long, double, bool
• Improved RuntimeFieldsQueryBuilder test to verify builder output, not just input state
• Removed all dead/commented-out code
• Passed TestContext.Current.CancellationToken to all test async calls for proper cancellation support
• Clean dotnet format -- no unused usings or style issues
• Made ConfigureFoundatioRepositoryDefaults() idempotent to prevent duplicate converter registration
• Fixed TopHitsAggregate._hits field to initialize with empty list in parameterless constructor
• Fixed ElasticReindexer.CreateRangeQuery to use startTime.Value after HasValue guard for explicit intent

Review Feedback Addressed

All github-code-quality and Copilot review comments have been resolved:

• partsToCreate unused container removed (was already fixed in prior commit)
• Constant condition warnings in JsonDiffer.GetHashCode fixed with explicit null guard
• Redundant null-conditional in LoggerExtensions.LogRequest fixed
• startTime.Value changed in ElasticReindexer for explicit intent after null guard
• as casts replaced with direct casts in test assertions
• Query timeout format fixed from .ToString() to ES duration format
• Operation.Build null guard added
• PatchDocument.Load strict validation added
• GetAliasIndexCount error handling improved
• Stale NEST-style mapping syntax in docs updated to ES9 API
• Silent bucket type skip in GetKeyedBuckets replaced with explicit error