24 Pull requests merged by 12 people

• Workaround numpy2 failed fastparquet compatibility tests [databricks]

  #11072 merged Jun 17, 2024

• Calculate parallelism to speed up pre-merge CI

  #11046 merged Jun 14, 2024

• fix flaky array_item test failures

  #11054 merged Jun 14, 2024

• [FEA] Increase parallelism of deltalake test on databricks

  #11051 merged Jun 14, 2024

• `binary-dedupe` changes for Spark 4.0.0 [databricks]

  #10993 merged Jun 13, 2024

• Add in the ability to fingerprint JSON columns [databricks]

  #11060 merged Jun 13, 2024

• Revert "Add in the ability to fingerprint JSON columns (#11002)" [skip ci]

  #11059 merged Jun 13, 2024

• Merge branch-24.06 into main [skip ci]

  #11058 merged Jun 13, 2024

• [auto-merge] branch-24.06 to branch-24.08 [skip ci] [bot]

  #11057 merged Jun 13, 2024

• Update latest changelog [skip ci]

  #11056 merged Jun 13, 2024

• [auto-merge] branch-24.06 to branch-24.08 [skip ci] [bot]

  #11055 merged Jun 13, 2024

• Add spark343 shim for scala2.13 dist jar

  #11052 merged Jun 13, 2024

• Concat() Exception bug fix

  #11039 merged Jun 13, 2024

• Add in the ability to fingerprint JSON columns

  #11002 merged Jun 12, 2024

• Rewrite multiple literal choice regex to multiple contains in rlike

  #10977 merged Jun 12, 2024

• [auto-merge] branch-24.06 to branch-24.08 [skip ci] [bot]

  #11034 merged Jun 11, 2024

• Fix auto merge conflict 11034 [skip ci]

  #11035 merged Jun 11, 2024

• Append new authorized user to blossom-ci whitelist [skip ci]

  #11040 merged Jun 11, 2024

• Update blossom-ci ACL to secure format [skip ci]

  #11036 merged Jun 11, 2024

• Fix a hive write test failure for Spark 350

  #11032 merged Jun 11, 2024

• Merge branch-24.06 into main

  #10982 merged Jun 10, 2024

• Improve log to print more lines in build [skip ci]

  #10998 merged Jun 10, 2024

• Update latest changelog [skip ci]

  #10981 merged Jun 10, 2024

• [DOC] Update docs for 24.06.0 release [skip ci]

  #10984 merged Jun 10, 2024

7 Pull requests opened by 4 people

• Drop spark-3.1.x support for spark-rapids

  #11041 opened Jun 11, 2024

• Dataproc serverless test fixes

  #11043 opened Jun 11, 2024

• Fixed Failing tests in arithmetic_ops_tests for Spark 4.0.0 [databricks]

  #11044 opened Jun 12, 2024

• Fixed array_tests for Spark 4.0.0 [databricks]

  #11048 opened Jun 12, 2024

• Fix some cast_tests for Spark 4.0.0 [databricks]

  #11049 opened Jun 13, 2024

• fix duplicate counted metrics like op time for GpuCoalesceBatches

  #11062 opened Jun 14, 2024

• Replaced spark3xx-common references to spark-shared

  #11066 opened Jun 14, 2024

9 Issues closed by 8 people

• [BUG] array_item test failures on Spark 3.3.x

  #8652 closed Jun 14, 2024

• [FEA] Improve delta test parallelism on databricks.

  #11050 closed Jun 14, 2024

• [BUG] Build on Databricks 330 fails

  #11053 closed Jun 13, 2024

• Concat cannot accept no parameter

  #10925 closed Jun 13, 2024

• [BUG] regex `^.*literal` cannot be rewritten as `contains(literal)` for multiline strings

  #10975 closed Jun 12, 2024

• Rewrite `pattern1|pattern2|pattern3` to multiple contains in `rlike`

  #10976 closed Jun 12, 2024

• [BUG] hive_parquet_write_test.py: test_write_compressed_parquet_into_hive_table integration test failures

  #10956 closed Jun 11, 2024

• [DOC] v2402 already support spark342

  #10840 closed Jun 10, 2024

• [BUG] [log] Improve log to print more lines in build/buildall.sh

  #10967 closed Jun 10, 2024

12 Issues opened by 10 people

• [BUG] numpy2 fail fastparquet cases: numpy.dtype size changed

  #11070 opened Jun 17, 2024

• [AUDIT][SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts

  #11069 opened Jun 14, 2024

• [AUDIT][SPARK-48168][SQL][FOLLOWUP] Match expression strings of shift operators & functions with user inputs

  #11068 opened Jun 14, 2024

• [AUDIT][SPARK-41049][SQL][FOLLOW-UP] Mark map related expressions as stateful expressions

  #11067 opened Jun 14, 2024

• [BUG] Spark Connect Server (3.5.1) Can Not Running Correctly

  #11065 opened Jun 14, 2024

• GPU OutOfMemory while DISTINCT a partitionedBy column on DeltaTable ?

  #11064 opened Jun 14, 2024

• [BUG] op time for GpuCoalesceBatches is more than actual

  #11063 opened Jun 14, 2024

• [BUG] Shims inadvertently missed for various artifacts

  #11061 opened Jun 13, 2024

• Revisit the ANSI tests for Spark 4.0.0

  #11047 opened Jun 12, 2024

• [BUG] test `json_test.py::test_from_json_struct_decimal` fails

  #11045 opened Jun 12, 2024

• [FEA] explore using hyper-log-log to estimate the if we should continue with a partial aggregation.

  #11042 opened Jun 11, 2024

• Support multiple ranges in PrefIxRange regex rewrite such as `prefix[a-zA-Z0-9]`

  #11037 opened Jun 11, 2024

65 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Support bucketing write for GPU

  #10957 commented on Jun 17, 2024 • 56 new comments

• [FEA] Introduce low shuffle merge.

  #10979 commented on Jun 14, 2024 • 44 new comments

• Add a heuristic to skip second or third agg pass

  #10950 commented on Jun 13, 2024 • 13 new comments

• Fix some test issues in Spark UT and keep RapidsTestSettings update-to-date

  #10997 commented on Jun 14, 2024 • 10 new comments

• Fallback non-UTC TimeZoneAwareExpression with zoneId [databricks]

  #10996 commented on Jun 12, 2024 • 5 new comments

• POM Changes for Spark 4.0.0 [databricks]

  #10994 commented on Jun 14, 2024 • 4 new comments

• [BUG] Failures in Integration Tests on Dataproc Serverless

  #10347 commented on Jun 11, 2024 • 3 new comments

• Fix tests failures in parquet_write_test.py

  #11024 commented on Jun 12, 2024 • 2 new comments

• [FEA] Have a repartition fallback for hash aggregates instead of sort

  #10370 commented on Jun 12, 2024 • 2 new comments

• [BUG] CICD failed a case: cmp_test.py::test_empty_filter[>]

  #11033 commented on Jun 11, 2024 • 2 new comments

• [FEA] Rework GpuSubstringIndex to use cudf::slice_strings

  #8750 commented on Jun 14, 2024 • 2 new comments

• [BUG] Slow/no progress with cascaded pandas udfs/mapInPandas in Databricks

  #10770 commented on Jun 17, 2024 • 2 new comments

• [AUDIT][SPARK-48215][SQL] Extending support for collated strings on date_format expression

  #11003 commented on Jun 11, 2024 • 2 new comments

• Fix tests failures in url_test.py

  #11017 commented on Jun 13, 2024 • 1 new comment

• Fix tests failures in date_time_test.py

  #11025 commented on Jun 12, 2024 • 1 new comment

• Fix tests failures in dpp_test.py

  #11023 commented on Jun 12, 2024 • 1 new comment

• Fix tests failures in join_test.py

  #11022 commented on Jun 12, 2024 • 1 new comment

• Fix tests failures in orc_cast_test.py

  #11021 commented on Jun 12, 2024 • 1 new comment

• Fix tests failures in grouping_sets_test.py

  #11020 commented on Jun 12, 2024 • 1 new comment

• Fix tests failures in subquery_test.py

  #11029 commented on Jun 12, 2024 • 1 new comment

• [BUG] test_regexp_choice failed

  #10641 commented on Jun 12, 2024 • 1 new comment

• Fix tests failures in multiple files

  #11031 commented on Jun 12, 2024 • 1 new comment

• Fix tests failures in map_test.py

  #11026 commented on Jun 12, 2024 • 1 new comment

• [FEA] Implement lore framework to support all operators.

  #10987 commented on Jun 11, 2024 • 1 new comment

• Fix tests failures in hash_aggregate_test.py

  #11018 commented on Jun 11, 2024 • 1 new comment

• [FEA] Support short mode of DayTimeIntervalType when cast string to daytime

  #10980 commented on Jun 11, 2024 • 1 new comment

• Fix tests failures in window_function_test.py

  #11019 commented on Jun 11, 2024 • 1 new comment

• Fix tests failures in csv_test.py

  #11016 commented on Jun 13, 2024 • 1 new comment

• Fix tests failures in sort_test.py

  #11027 commented on Jun 12, 2024 • 1 new comment

• Fix tests failures in ast_test.py

  #11008 commented on Jun 12, 2024 • 1 new comment

• [FEA] Remove support for Spark 3.1.x

  #10955 commented on Jun 14, 2024 • 1 new comment

• [AUDIT][SPARK-48148][CORE] JSON objects should not be modified when read as STRING

  #10990 commented on Jun 11, 2024 • 1 new comment

• Fix tests failures in string_test.py

  #11030 commented on Jun 12, 2024 • 1 new comment

• [BUG] GpuCoalesceBatches op time metric includes everything before it

  #7353 commented on Jun 17, 2024 • 1 new comment

• Fix test failures in arithmetic_ops_test.py

  #11006 commented on Jun 11, 2024 • 1 new comment

• [AUDIT][SPARK-48191][SQL] Support UTF-32 for string encode and decode

  #10991 commented on Jun 11, 2024 • 1 new comment

• Release Checklist v24.06

  #10638 commented on Jun 17, 2024 • 1 new comment

• Fix tests failures in cast_test.py

  #11009 commented on Jun 13, 2024 • 0 new comments

• [BUG] Issues found by Spark UT Framework on RapidsStringExpressionsSuite

  #10775 commented on Jun 13, 2024 • 0 new comments

• [FEA] Add Spark 3.5.2 snapshot support

  #10437 commented on Jun 13, 2024 • 0 new comments

• [FEA] Create Spark 4.0.0 shim and build env

  #9259 commented on Jun 13, 2024 • 0 new comments

• [FEA] Support single '$' or '^' on right side of regexp choice

  #10764 commented on Jun 13, 2024 • 0 new comments

• [FEA] [AUDIT] Support expressions to work with collated strings

  #10876 commented on Jun 14, 2024 • 0 new comments

• Profiler: Disable collecting async allocation events by default

  #10965 commented on Jun 12, 2024 • 0 new comments

• prototype for new design LORE (with LORE id)

  #10999 commented on Jun 12, 2024 • 0 new comments

• [BUG] Cast String to Decimal could return null incorrectly when scale = precision

  #10890 commented on Jun 10, 2024 • 0 new comments

• [BUG] test_decimal_round fails with DATAGEN_SEED=3

  #9847 commented on Jun 10, 2024 • 0 new comments

• [BUG] 5 tests will fail when enabling GPU serde for the normal Shuffle.

  #10823 commented on Jun 11, 2024 • 0 new comments

• from_json, when input = empty object, rapids throws an exception.

  #10910 commented on Jun 11, 2024 • 0 new comments

• Parsing a column containing invalid json into StructureType with schema throws an Exception.

  #10891 commented on Jun 11, 2024 • 0 new comments

• [FEA] Support collect_limited_list in windowing

  #10930 commented on Jun 11, 2024 • 0 new comments

• [AUDIT][SPARK-48019] Fix incorrect behavior in ColumnVector/ColumnarArray with dictionary and nulls

  #10988 commented on Jun 11, 2024 • 0 new comments

• [AUDIT][SPARK-48182][SQL] SQL (java side): Migrate `error/warn/info` with variables to structured logging framework

  #10989 commented on Jun 11, 2024 • 0 new comments

• Fallback TimeZoneAwareExpression that only support UTC with zoneId instead of timeZone config

  #10995 commented on Jun 11, 2024 • 0 new comments

• Fix test failures for Spark 4.0.0

  #11004 commented on Jun 11, 2024 • 0 new comments

• Fix tests failures in cache_test.py

  #11010 commented on Jun 11, 2024 • 0 new comments

• Fix tests failures in collection_ops_test.py

  #11011 commented on Jun 11, 2024 • 0 new comments

• Fix tests failures in conditionals_test.py

  #11012 commented on Jun 11, 2024 • 0 new comments

• Fix tests failures in orc_test.py

  #11013 commented on Jun 11, 2024 • 0 new comments

• Fix tests failures in json_test.py

  #11014 commented on Jun 11, 2024 • 0 new comments

• Fix tests failures in parquet_test.py

  #11015 commented on Jun 11, 2024 • 0 new comments

• Fix tests failures in qa_nightly_select_test.py

  #11028 commented on Jun 11, 2024 • 0 new comments

• [FEA] Rewrite some regular expressions `RLIKE` cases to faster expressions

  #10741 commented on Jun 12, 2024 • 0 new comments

• Fix test failures in aqe_test.py

  #11005 commented on Jun 12, 2024 • 0 new comments

• Fix tests failures in array_test.py

  #11007 commented on Jun 12, 2024 • 0 new comments