Details
-
Sub-Task
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Importing part table from a 500gb dbt3 dataset. When the target table is compressed, observed behavior is expected and good (19k calls to write()). When uncompressed, the opposite (750k calls to write()).
Small writes make the overhead parts in SM dominate. In particular, what rises to the top of the profile are functions related to json parsing.
The first thing I will do is implement a parsed-json cache, then reevaluate.