How to set Map values in spark/scala -


i new spark-scala development. trying create map values in spark using scala getting nothing printed

def createmap() : map[string, int] = { var tmap:map[string, int] = map() val tdf = spark.sql("select a, b, c temp") (x <- tdf) {   val k = x.getas[long](0) + "|" + x.getas[long](1)   val v = x.getas[int](2)   tmap += ( k -> v )   println( k -> v ) ///----------this print values } println("hellllooooooooo1") ((k,v) <- tmap) println("key = " + k+ ", value= " + v) ////------this prints nothing println("hellllooooooooo2") return tmap }

please suggest.

user8598832 gives how (for value of properly). reason approach doesn't work you're adding (k, v) map in executor, println occurs in driver, won't see map(s) in executor(s) (to extent might, that's artifact of running in local mode not in distributed mode).


Comments

Post a Comment

Popular posts from this blog

php - Embed BBCode within itself not working -

so working on simple php bbcode system, though doing [center][b][i]text[/i][/b][/center] doesn't parse. leaves couple out. curious if give me tip? or point me in direction learn how fix issue. sample of code: function bbcode($str) { $patterns = array(); $patterns = array( '#\[b\](.*?)\[/b\]#', '#\[i\](.*?)\[/i\]#', '#\[u\](.*?)\[/u\]#', '#\[color=(.*?)\](.*?)\[/color\]#', '#\[img\](.*?)\[/img\]#', '#\[center\](.*?)\[/center\]#', '#\n#' ); $replacements = array(); $replacements = array( '<strong>$1</strong>', '<em>$1</em>', '<u>$1</u>', '<span style="color:$1">$2</span>', '<img src="$1" style="max-width:45%;"></img>', '<center>$1</center>', '<br/>' ); return preg_replace($patterns, $re...
Read more

Connecting Spring Batch to Remote Cassandra Database -

i'm hoping read data csv file, process data, , upload remote database. i'm using spring's starter repo base . code here i tried putting information in properties file: spring.data.cassandra.contact-points=ip, ip spring.data.cassandra.port=port spring.data.cassandra.keyspace-name=keyspace_name spring.data.cassandra.username=user spring.data.cassandra.password=pass spring.data.cassandra.ssl=true however, think keeps defaulting pushing local tomcat jdbc. i'm not sure start. appreciated! thanks. your code doesn't have use cassandra. doesn't have of dependencies , itemwriter implementation jdbcbatchitemwriter don't think work cassandra. need configure application use cassandra (the spring data starter itemwriter implementation can write cassandra).
Read more

minify - Minimizing css files -

i want know how css files can minimized. currently, using external sites css minifier compress css files. however, after removing whitespaces, comments, etc, file sizes reduced 10-15%. how frameworks bootstrap manage make minimized versions kilobytes. as example, css is: body { background-color: #ff0; } the reduction goes 28 bytes 22 bytes other css files have lot more characters still smaller this. actually minifying code means it's remove thing code white space characters new line characters comments block delimiters it reduces amount of code has transferred on web , speed site load fastly.but unreadable format.
Read more