hadoop - Best way to automatate getting data from Csv files to Datalake -


i need data csv files ( daily extraction différent business databasses ) hdfs move hbase , finaly charging agregation of data datamart (sqlserver ).

i know best way automate process ( using java or hadoops tools )

little no coding required? in no particular order

  • talend open studio
  • streamsets data collector
  • apache nifi

assuming can setup kafka cluster, can try kafka connect

if want program something, spark. otherwise, pick favorite language. schedule job via oozie

if don't need raw hdfs data, can load directly hbase


Comments

Post a Comment

Popular posts from this blog

php - Embed BBCode within itself not working -

so working on simple php bbcode system, though doing [center][b][i]text[/i][/b][/center] doesn't parse. leaves couple out. curious if give me tip? or point me in direction learn how fix issue. sample of code: function bbcode($str) { $patterns = array(); $patterns = array( '#\[b\](.*?)\[/b\]#', '#\[i\](.*?)\[/i\]#', '#\[u\](.*?)\[/u\]#', '#\[color=(.*?)\](.*?)\[/color\]#', '#\[img\](.*?)\[/img\]#', '#\[center\](.*?)\[/center\]#', '#\n#' ); $replacements = array(); $replacements = array( '<strong>$1</strong>', '<em>$1</em>', '<u>$1</u>', '<span style="color:$1">$2</span>', '<img src="$1" style="max-width:45%;"></img>', '<center>$1</center>', '<br/>' ); return preg_replace($patterns, $re...
Read more

neo4j - finding mutual friends in a cypher statement starting with three or more persons -

Image
i trying build cypher statement neo4j know 2-n starting nodes name , need find node (if any) can reached of starting nodes. at first tought similar "mutual friend" situation handled (start1)-[*..2]->(main)<-[*..2]-(start2) in case have more 2 starting points around 6 know name. so puzzled how can include third, fourth , on node cypher able find commmon root amongst them. in above example neo4j website need path starting 'dilshad', 'becky' , 'cesar' check if have common friend (anders) excluding 'filipa' , 'emil' not friends of three. so far create statement programmatically looks like match (start1 {name:'person1'}), (start2 {name:'person2'}), (start3 {name: 'person3'}), (main) (start1)-[*..2]->(main) , (start2)-[*..2]->(main) , (start3)-[*..2]->(main) return distinct main but wondering if there more elegant / efficient way in cypher possibly use list of names parameter ...
Read more

linux - Why does bash short curcuit fail in crontab? -

if in terminal do [ $(/usr/bin/date +"%y%m%d") -le 20171209 ] && echo aaa >> /tmp/a it works. if in crontab -e * * * * * echo aaa >> /tmp/a then works. if in crontab -e * * * * * [ $(/usr/bin/date +"%y%m%d") -le 20171209 ] && echo aaa >> /tmp/a then nothing happens. looking @ /var/log/cron see sep 12 08:36:01 enote crond[8809]: (root) cmd ([ $(/usr/bin/date +") question can explain issue is? percent sign has special meaning in crontab, must escaped \% man 5 crontab /percent percent-signs (%) in command, unless escaped backslash (\), changed newline characters, , data after first % sent command standard input.
Read more