!CJoUbovqKaCGrFkbrY:matrix.org

Spark with Scala

305 Members
A place to discuss and ask questions about using Scala for Spark programming.2 Servers

Load older messages


SenderMessageTime
21 May 2022
@_discord_870283898070900766:t2bot.iosoim12 Hi guys 17:27:35
@_discord_870283898070900766:t2bot.iosoim12 is there any workaround for circular references in scala reflection? 17:27:52
22 May 2022
@_discord_948609927906226177:t2bot.iohr24official joined the room.12:05:51
@_discord_243951995709292544:t2bot.ioMajestic#1066 I don't think this is Spark specific, maybe try #scala-users ? 14:23:05
@_discord_870283898070900766:t2bot.iosoim12 Scalareflection is a Part of spark catalyst api..for datasets 14:23:47
@_discord_243951995709292544:t2bot.ioMajestic#1066 oh right 14:24:05
@_discord_243951995709292544:t2bot.ioMajestic#1066 sorry 14:24:19
@_discord_870283898070900766:t2bot.iosoim12 Ah it's fine.I know it's an issue in spark,so I thought someone here would have a workaround 14:25:04
@_discord_243951995709292544:t2bot.ioMajestic#1066 Just curious, is it because you have 2 case classes referencing each other, or is it coming from Java Protobuf/Avro generated classes..? 14:37:44
@_discord_787364650315939862:t2bot.iotiki joined the room.14:47:54
23 May 2022
@_discord_712743405628948531:t2bot.iomaxo joined the room.04:51:50
@_discord_868969255633322048:t2bot.ioBiancaMarcucci joined the room.11:32:04
@_discord_633521744258531331:t2bot.ioakshay joined the room.15:11:43
@_discord_832633653380907048:t2bot.ioraishad joined the room.16:42:55
@_discord_978209905590812732:t2bot.ioAGUIE joined the room.17:43:59
@_discord_692391819916345364:t2bot.ioDynam1co joined the room.19:51:42
@_discord_692391819916345364:t2bot.ioDynam1co Hello all, next week I'll have the exam for Databricks Apache Spark certification. I have been working with Scala-Spark from last 6 months. Any tip? 19:51:46
@_discord_692391819916345364:t2bot.ioDynam1co * Hello all, next week I'll have the exam for Databricks Apache Spark certification. I have been working with Scala-Spark from last 6 months. Any tip? 19:51:46
24 May 2022
@_discord_972210357034700821:t2bot.iosteve-comcast joined the room.22:53:41
25 May 2022
@_discord_871875254451044363:t2bot.ioMridulGupta | GithHubSRM joined the room.00:24:58
@_discord_717633587284803616:t2bot.iorhansen joined the room.12:53:09
@_discord_240989507208478722:t2bot.ioBillzabob joined the room.17:24:03
@_discord_712743405628948531:t2bot.iomaxo changed their display name from maxo to maxo#0417.21:24:42
@_discord_712743405628948531:t2bot.iomaxo changed their display name from maxo#0417 to maxo.21:24:52
@_discord_346552981510160384:t2bot.ioACE81 joined the room.23:08:53
26 May 2022
@_discord_317417873159094283:t2bot.iomilestobudapest joined the room.09:21:27
@_discord_547027845034278912:t2bot.iopaddenpaadje How do I join two datasets when one id is Long and other is Option[Long] 11:44:04
@_discord_554696665056149505:t2bot.ioNicolas joined the room.12:12:15
@_discord_243951995709292544:t2bot.ioMajestic#1066 I guess using joinWith() and describing the join condition with Column type it works. It just won't match on None ids.
Did you try it?
18:22:56
@_discord_547027845034278912:t2bot.iopaddenpaadjeunknown.png
Download unknown.png
20:57:26

Show newer messages


Back to Room List