Spark Streaming Bug - Window of Windowed DStream does not work -

window of windowed dstream not work in spark streaming. seems scheduler bug inside spark streaming.

val layer0= // input data val layer1 = layer0.window(seconds(30), seconds(30)) // works layer1.foreachrdd(...)  val layer2 = layer1.window(seconds(60), seconds(60)) // not work layer2.foreachrdd(...) 

anyone met issue, , know how fix in spark.

add more detail driver log:

time 1433141250000 ：

2015-06-01 06:47:30 info mapvalueddstream - time 1433141250000 ms invalid zerotime 1433141240000 ms , slideduration 30000 ms , difference 10000 ms

2015-06-01 06:47:30 info mapvalueddstream - time 1433141250000 ms invalid zerotime 1433141240000 ms , slideduration 60000 ms , difference 10000 ms

time 1433141260000 :

2015-06-01 06:47:40 info mapvalueddstream - time 1433141260000 ms invalid zerotime 1433141240000 ms , slideduration 30000 ms , difference 20000 ms

2015-06-01 06:47:40 info mapvalueddstream - time 1433141260000 ms invalid zerotime 1433141240000 ms , slideduration 60000 ms , difference 20000 ms

time 1433141270000 : (30s)

2015-06-01 06:47:50 info filtereddstream - slicing 1433141250000 ms 1433141270000 ms (aligned 1433141250000 ms , 1433141270000 ms)

2015-06-01 06:47:50 info mapvalueddstream - time 1433141270000 ms invalid zerotime 1433141240000 ms , slideduration 60000 ms , difference 30000 ms

time 1433141280000 : 2015-06-01 06:48:00 info mapvalueddstream - time 1433141280000 ms invalid zerotime 1433141240000 ms , slideduration 30000 ms , difference 40000 ms

2015-06-01 06:48:00 info mapvalueddstream - time 1433141280000 ms invalid zerotime 1433141240000 ms , slideduration 60000 ms , difference 40000 ms

time 1433141290000 :

2015-06-01 06:48:10 info mapvalueddstream - time 1433141290000 ms invalid zerotime 1433141240000 ms , slideduration 30000 ms , difference 50000 ms

2015-06-01 06:48:10 info mapvalueddstream - time 1433141290000 ms invalid zerotime 1433141240000 ms , slideduration 60000 ms , difference 50000 ms

time 1433141300000 : (60s)

2015-06-01 06:48:20 info windoweddstream - slicing 1433141270000 ms 1433141300000 ms (aligned 1433141250000 ms , 1433141280000 ms)

2015-06-01 06:48:20 info windoweddstream - time 1433141250000 ms invalid zerotime 1433141240000 ms , slideduration 30000 ms , difference 10000 ms

2015-06-01 06:48:20 info windoweddstream - time 1433141280000 ms invalid zerotime 1433141240000 ms , slideduration 30000 ms , difference 40000 ms

it indeed bug , i've logged spark-7326. i've fixed myself. see my pull request, has been merged master. believe fix in 1.4.0 release.