Sender | Message | Time |
---|---|---|
20 Oct 2023 | ||
derya changed their display name from derya to derya#0. | 21:38:12 | |
derya changed their profile picture. | 21:38:46 | |
derya changed their display name from derya#0 to derya. | 21:43:39 | |
24 Oct 2023 | ||
whiteturq | 19:11:37 | |
25 Oct 2023 | ||
lafeychine changed their display name from Vincent Lafeychine to lafeychine. | 13:35:17 | |
26 Oct 2023 | ||
pgm changed their display name from pgm to progamermatt. | 15:15:17 | |
27 Oct 2023 | ||
txdv | 06:36:27 | |
cheapsolutionarchitect | I do not think Spark supports such a use case. Usually you would trigger the application via a cronjob, or some kind of cluster manager. And you could run into a timeout. If your driver does not run on the Spark-Cluster, you can however, try to wait in your driver application and then call start() on your stream. | 09:39:35 |
cheapsolutionarchitect | * I do not think Spark supports such a use case. Usually you would trigger the application via a cronjob, or some kind of cluster manager task. If your driver does not run on the Spark-Cluster, you can however, try to wait in your driver application and then call start() on your stream. | 09:45:33 |
cheapsolutionarchitect | * I do not think Spark supports such a use case. Usually you would trigger the application via a cronjob, or some kind of cluster manager task. If your driver does not directly run on the Spark-Cluster, you can however, try to wait in your driver application and then call start() on your stream. | 09:45:58 |
cheapsolutionarchitect | As far as I have understood, StreamingContext is old API and Structured Streaming is the successor. And I would not call the alignment on batch interval borders a feature. It breaks down really fast, e.g. how do you trigger the batch on every hour starting half past a defined hour? However, my experience is confined to my specific cluster architecture. I run Spark in standalone mode on a k8s cluster. So every Spark-driver-app is a running Pod. This allows me, for example to sleep-wait within the entry point script or within the driver app and so on. | 20:21:09 |
UFO#0678 | nice! I've been wanting to try a small k8s spark instance | 20:45:20 |
29 Oct 2023 | ||
cu changed their display name from cu4381#0 to cu4381. | 23:32:40 | |
1 Nov 2023 | ||
YandexTan changed their profile picture. | 16:21:38 | |
7 Nov 2023 | ||
softinio changed their profile picture. | 17:56:04 | |
12 Nov 2023 | ||
cheapsolutionarchitect | Take a look at the class comment here https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/KeyValueGroupedDataset.html. Your K is the first element of the tuple, and your V is the tuple, that results in the given return type. If you call .collect, you will see your expected result. If you want a DF of type String, Array[Int], you could probably make use of mapGroups and turn the second position of the tuple into an Array. | 06:09:12 |
cheapsolutionarchitect | * Take a look at the class comment here https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/KeyValueGroupedDataset.html. Your K is the first element of the tuple, and your V is the tuple, that results in the given return type. If you call .collect, you will probably see your expected result. If you want a DF of type String, Array[Int], you could probably make use of mapGroups and turn the second position of the tuple into an Array. | 06:13:26 |
13 Nov 2023 | ||
eje changed their display name from eje to eje4073. | 18:30:41 | |
14 Nov 2023 | ||
atk91 changed their display name from atk91#0 to atk91. | 19:22:01 | |
28 Nov 2023 | ||
Zone of proximal development changed their display name from marouan28#0 to marouan28. | 12:18:26 | |
kagaku2340 changed their display name from kagaku to kagaku2340. | 20:37:26 | |
30 Nov 2023 | ||
vazand | 08:41:11 | |
2 Dec 2023 | ||
derya changed their profile picture. | 00:07:35 | |
3 Dec 2023 | ||
YandexTan changed their profile picture. | 08:38:56 | |
4 Dec 2023 | ||
ekrich#7695 | Repost from Jobs. https://discord.com/channels/632150470000902164/632628675287973908/1181311320298098808 Excited to see the Release Notes for Spark here. https://spark.apache.org/releases/spark-release-3-5-0.html Excerpt: Removals, Behavior Changes and Deprecations Upcoming Removal The following features will be removed in the next Spark major release Support for Java 8 and Java 11, and the minimal supported Java version will be Java 17 Support for Scala 2.12, and the minimal supported Scala version will be 2.13 | 20:56:01 |
ekrich#7695 | Typically, Spark has 2 versions, current and next with the default to be 2.13 and next 3. | 20:56:49 |
ekrich#7695 | * Repost from Jobs. https://discord.com/channels/632150470000902164/632628675287973908/1181311320298098808 Excited to see the Release Notes for Spark here. https://spark.apache.org/releases/spark-release-3-5-0.html Excerpt: Removals, Behavior Changes and DeprecationsUpcoming Removal The following features will be removed in the next Spark major release Support for Java 8 and Java 11, and the minimal supported Java version will be Java 17 Support for Scala 2.12, and the minimal supported Scala version will be 2.13 | 20:57:25 |
ekrich#7695 | * Repost from Jobs. https://discord.com/channels/632150470000902164/632628675287973908/1181311320298098808 Excited to see the Release Notes for Spark here. https://spark.apache.org/releases/spark-release-3-5-0.html Excerpt: Removals, Behavior Changes and Deprecations | 20:58:02 |
5 Dec 2023 | ||
cheapsolutionarchitect | Try spark-sql-api instead of spark-sql. | 08:12:30 |
7 Dec 2023 | ||
cheapsolutionarchitect | Sorry for the late answer, you will get the same behavior in Scala 2.13.x. The case class is probably defined within a method. I do not know enough about the internals of Scala, but it looks like the type tag can not be estimated. However, if you pull the case class in into the outer class, it works. So do this instead of the commented line | 05:03:43 |