python - Spark "reduceByKey" calls reducer more times than expected -


i have dataframe of 4 rows same key this:

row_no id  age  time 1      abc  70  1524299530 2      abc  69  1524299528 3      abc  68  1524299526 4      abc  67  1524299524

then tried call reducebykey on dataframe this:

new_rdd = df.rdd \         .map(lambda row: (row['id'], [row['age'], row['time']])) \         .reducebykey(some_reducer)

in some_reducer, purpose of testing return previous obj. used print trace invocation of reducer , found spark had invoked reducer 4 times, namely, (1, 2), (3, 4), (1, 3) , (1, 3). basically, reducer called on row 1 , 3 twice. running spark locally 4 processors. tried run spark 1 processor on job , reducer called 3 times expected on (1, 2), (3, 4) , (1, 3). must have how spark partitions data still hard make sense of behavior. can provide explanation behavior?

update: did more constructive test sticking integer column on each row , make reducer lambda a, b: + b. observed spark did 4 additions when run multi-processor mode in total: 1 + 1, 1 + 1, 2 + 2 , 2 + 2. however, final result still 4. in way, spark discarded duplicated reducing of 2 + 2. question why there duplicated reducing in first place , how spark deals them?


Comments

Post a Comment

Popular posts from this blog

javascript - WinJS appendTextAsync producing scheduler errors -

im trying implement logger in uwp app (windows 10) following code in both this , this answers. i'm afraid in both cases end getting scheduler ( winjs.utilities.scheduler ) errors @ random times (couldn't establish pattern). main problem the application freezes sometimes , suspect due log implementation because final line reads transitioning job (1) from: running to: blocked when happens. i messages like transitioning job (1) from: blocked_waiting to: cooperative_yield transitioning job (1) from: cooperative_yield to: scheduled in middle of execution, varying states. initialization method this.init = function () { let logger = new winjs.promise(function (complete) { let = new date(); let logfilename = `log-${now.getdate()}-${now.getmonth() + 1}-${now.getfullyear()}.log`; localfolder .createfolderasync("logs", windows.storage.creationcollisionoption.openifexists) .then(fun...
Read more

minify - Minimizing css files -

i want know how css files can minimized. currently, using external sites css minifier compress css files. however, after removing whitespaces, comments, etc, file sizes reduced 10-15%. how frameworks bootstrap manage make minimized versions kilobytes. as example, css is: body { background-color: #ff0; } the reduction goes 28 bytes 22 bytes other css files have lot more characters still smaller this. actually minifying code means it's remove thing code white space characters new line characters comments block delimiters it reduces amount of code has transferred on web , speed site load fastly.but unreadable format.
Read more

Sockets with kotlin -

i looking develop application in kotlin uses sockets, unfortunately kotlinx sockets broken , not far connect current kotlin 1.1 co-routines. (i see noted unanswered issue on github @sundwarf on july 16 already) specifically, sample code kotlinx.sockets uses connect in form: asocket().tcp().connect(inetsocketaddress(inetaddress.getbyname("google.com"), 80)) which fails noted trace here . i looking either sample of connect work, or recommendation of alternate package kotlix.sockets. there many kotlin apps making use of sockets, either there alternatives kotlinx.sockets or there way around connect problem.
Read more