-
Notifications
You must be signed in to change notification settings - Fork 29.1k
[SPARK-55708][SQL] TransformingEncoder with primitive inside OptionEncoder#54506
[SPARK-55708][SQL] TransformingEncoder with primitive inside OptionEncoder#54506eejbyfeldt wants to merge 1 commit intoapache:masterfrom
Conversation
What changes were proposed in this pull request?
Make it possible to use TransformingEncoder from a primitive type and have that inside an OptionEncoder.
Supports for primitives using TransformingEncoder was initially improved in #51313 but it was missed that it did not work inside an OptionEncoder. The fix here also cleans up how javaClassToPrimitiveType is computed, it was missed that the values in typeJavaMapping are not unique and the previous code would assign wrong types for some classes.
Why are the changes needed?
To make the AgnosticEncoders as flexible as Expression based encoders were, this will allow libraries providing custom encoders to move to Spark 4.0.0 and beyond.
Does this PR introduce any user-facing change?
Yes, libraries trying to derive custom encoders with have more flexibility.
How was this patch tested?
Existing and new unittests.
Was this patch authored or co-authored using generative AI tooling?
No.
OptionEncoder.
|
@hvanhovell @chris-twiner here is another issue I ran into trying to move to the AgnosticEncoders. Would be great if someone could have review. |