| 5 Dec 2023 |
 cheapsolutionarchitect | Try spark-sql-api instead of spark-sql. | 08:12:30 |
| 7 Dec 2023 |
| cheapsolutionarchitect | Sorry for the late answer, you will get the same behavior in Scala 2.13.x. The case class is probably defined within a method. I do not know enough about the internals of Scala, but it looks like the type tag can not be estimated. However, if you pull the case class in into the outer class, it works. So do this instead of the commented line class Spark1() {
case class A(b: String, c: String)
private def execute() : Unit = {
val spark = SparkSession.builder().getOrCreate()
import spark.implicits._
//case class A(b: String, c: String)
val as = Seq(
A("b", "c"),
).toDS()
as.show()
}
}
| 05:03:43 |
| cheapsolutionarchitect | * Sorry for the late answer, you will get the same behavior in Scala 2.13.x. The case class is probably defined within a method. I do not know enough about the internals of Scala, but it looks like the type tag can not be estimated. However, if you pull the case class into the outer class, it works. So do this instead of the commented line class Spark1() {
case class A(b: String, c: String)
private def execute() : Unit = {
val spark = SparkSession.builder().getOrCreate()
import spark.implicits._
//case class A(b: String, c: String)
val as = Seq(
A("b", "c"),
).toDS()
as.show()
}
}
| 05:04:54 |
| 9 Dec 2023 |
|  Salar Rahmanian (softinio) changed their profile picture. | 05:18:53 |
|  unarist | 07:17:18 |
| cheapsolutionarchitect | Not really, just the last to digits of a year are ambiguous. Spark decided on 2000 for the base value see https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html. Perhaps you could write your own udf making use of a certain feature of SimpleDateFormat, where you set a date and two digit years are interpreted 20 years into the past and 60 years into the future. https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/text/SimpleDateFormat.html | 12:22:53 |
| cheapsolutionarchitect | * Not really, just the last to digits of a year are ambiguous. Spark decided on 2000 for the base value, see YEAR in https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html. Perhaps you could write your own udf making use of a certain feature of SimpleDateFormat, where you set a date and two digit years are interpreted 20 years into the past and 60 years into the future. https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/text/SimpleDateFormat.html | 12:23:18 |
| cheapsolutionarchitect | * Not really, just the last two digits of a year are ambiguous. Spark decided on 2000 for the base value, see YEAR in https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html. Perhaps you could write your own udf making use of a certain feature of SimpleDateFormat, where you set a date and two digit years are interpreted 20 years into the past and 60 years into the future. https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/text/SimpleDateFormat.html | 12:24:04 |
| cheapsolutionarchitect | * Not really, just the last two digits of a year are ambiguous. Spark decided on 2000 for the base value, see YEAR in https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html. Perhaps you could write your own udf making use of a certain feature of SimpleDateFormat, where you set a date on the calender and a a result two digit years are interpreted 20 years into the past and 60 years into the future. https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/text/SimpleDateFormat.html | 12:24:51 |
| cheapsolutionarchitect | * Not really, just the last two digits of a year are ambiguous. Spark decided on 2000 for the base value, see YEAR in https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html. Perhaps you could write your own udf making use of a certain feature of SimpleDateFormat, where you set a date on the calender and as a result two digit years are interpreted 20 years into the past and 60 years into the future. https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/text/SimpleDateFormat.html | 12:27:53 |
| 11 Dec 2023 |
| cheapsolutionarchitect | Looks like the same kind of problem as this https://discord.com/channels/632150470000902164/839723072239566878/1182330580646506548 | 13:20:32 |
| cheapsolutionarchitect | * Looks like the same kind of problem as this https://discord.com/channels/632150470000902164/839723072239566878/1182330580646506548. In Scala 2 this works, maybe someone in #scala-users can help to shed some light onto the situation why this does not work in Scala 3. Or it is just an artifact of Spark not supporting Scala 3 directly at the current time. | 13:22:22 |
|  jungle_hacker69 changed their display name from jungle_hacker69#0 to jungle_hacker69. | 16:30:37 |
| cheapsolutionarchitect | I don't know, perhaps write a ticket to the spark project 🙂 | 16:59:39 |
| 16 Dec 2023 |
|  potatoef changed their profile picture. | 10:55:25 |
| 19 Dec 2023 |
|  steadhaven changed their display name from marouan28 to steadhaven. | 14:38:12 |
| 22 Dec 2023 |
|  mglvl changed their display name from miguelvilag to mglvl. | 16:28:53 |
| 24 Dec 2023 |
 ufodivebomb | I'll be thrilled once 2.13 is default on emr and databricks | 14:58:19 |
| 9 Jan 2024 |
 ekrich | All the rows in one array or an array per row? | 16:31:03 |
| 10 Jan 2024 |
| potatoef changed their profile picture. | 11:33:01 |
| ufodivebomb | I don't use databricks anymore but I recall being able to use spark UI and another per-query history interface with useful stats | 20:55:08 |
| 12 Jan 2024 |
|  zhao_nan changed their display name from Zhao Nan to zhao_nan. | 17:01:58 |
|  ragnar changed their display name from ragnar#4252 to ragnar. | 21:32:24 |
| 13 Jan 2024 |
| ekrich | I am a little confused about auto trim and extra padded spaces in the same context. Can you explain in a little more detail? | 00:14:28 |
|  ose_369 changed their display name from OSE to ose_369. | 00:42:10 |
| 14 Feb 2024 |
 nrktkt | !discord bridge 632150470000902164 839723072239566878 | 17:34:35 |
 Discord Bridge | I'm asking permission from the guild administrators to make this bridge. | 17:34:37 |
| nrktktchanged room power levels. | 17:34:45 |
| Discord Bridge | Timed out waiting for a response from the Discord owners.
| 17:39:38 |
| 28 Mar 2024 |
|  Dr. Dude changed their profile picture. | 11:18:03 |
