java - How to extract "infobox company" data from wiki dumps -


i have downloaded big wiki dump xml file https://dumps.wikimedia.org/enwiki/20170520/

i want extract metadata company name , parent company wikidumps. company data located in xml template below:

{{infobox company | name = | logo =  | type =  | industry =  | fate =  | predecessor = <!-- or: | predecessors = --> | successor = <!-- or: | successors = --> | founded = <!-- if known: {{start date , age|yyyy|mm|dd}} in [[city]], [[state]], [[country]] --> | founder = <!-- or: | founders = --> | defunct = <!-- {{end date|yyyy|mm|dd}} --> | hq_location_city =  | hq_location_country =  | area_served = <!-- or: | areas_served = --> | key_people =  | products =  | owner = <!-- or: | owners = --> | num_employees =  | num_employees_year = <!-- year of num_employees data (if known) --> | parent =  | website = <!-- {{url|example.com}} --> }}

i did research , found mediawiki parser. reference: https://github.com/dkpro/dkpro-jwpl/blob/master/de.tudarmstadt.ukp.wikipedia.parser/src/main/java/de/tudarmstadt/ukp/wikipedia/parser/tutorial/t1_simpleparserdemo.java

https://dkpro.github.io/dkpro-jwpl/jwplparser/

i tried use parser. requires file converted in string. wiki dump xml file 60 gb in size. can't convert big file in string , keep in memory. also, there no description mediawiki parser on how find specific element infobox company, go inside , extract name , other fields. below sample code mediawiki parser:

public static void main(string[] args) throws ioexception {      file file = new file("c:/users/njaiswal/downloads/accenture_data_from_wikidumps.xml");     string str = fileutils.readfiletostring(file);      // parsedpage object     mediawikiparserfactory pf = new mediawikiparserfactory();     mediawikiparser parser = pf.createparser();     parsedpage pp = parser.parse(str);     // sections       (section section : pp.getsections()) {         system.out.println("section : " + section.gettitle());         system.out.println(" nr of paragraphs      : " + section.nrofparagraphs());         system.out.println(" nr of tables          : " + section.nroftables());         system.out.println(" nr of nested lists    : " + section.nrofnestedlists());         system.out.println(" nr of definition lists: " + section.nrofdefinitionlists());         (link link : section.getlinks(link.type.internal)) {           system.out.println("  " + link.gettarget());       } }  }

is there other parser can solve problem? or can use same mediawiki parser "inbox company" , extract fields? appreciated. thanks

update: tried use wikixmlj parser khalil suggested. able "infobox" data, want limit "infobox company" data. below code , output:

import edu.jhu.nlp.wikipedia.*;     public class test {      public static void main(string[] args) throws exception{         wikixmlparser parser = wikixmlparserfactory.getsaxparser("c:/users/njaiswal/downloads/enwiki-20170520-pages-articles-multistream.xml/enwiki-20170520-pages-articles-multistream.xml");             parser.setpagecallback(new pagecallbackhandler() {                 public void process(wikipage page) {                   try {                     infobox infobox=page.getinfobox();                     system.out.println(infobox.dumpraw());                 } catch (wikitextparserexception e) {                     // todo auto-generated catch block                     e.printstacktrace();                 }                    //do info box                 }             });             parser.parse();     }  }

o/p:

{{infobox monarch | name            = attila | title           = [[list of hunnic rulers|ruler]] of [[hunnic empire]] | place of burial =  }} {{infobox sea | name = aegean sea | image = aegean sea map.png | caption = map of aegean sea | pushpin_map = world | pushpin_map_alt = world | pushpin_label_position = right }} {{infobox company | name             = audi ag  | logo             = audi-logo 2016.svg | logo_size = 235 | image            = audi ingolstadt.jpg | image_size = 265 }}

i used before wikixmlj simple dumb parser. shall parse perfectly:

// dumppath should c:\your/path/articles.xml.bz2" wikixmlparser wxsp = wikixmlparserfactory.getsaxparser(dumppath); wxsp.setpagecallback(new pagecallbackhandler() {     @override     public void process(wikipage page) {         //system.out.println("info box:" + page.getinfobox());         string regex = "\\{{infobox company(.|\\n)+";        pattern pattern = pattern.compile(regex);        matcher matcher = pattern.matcher(page.getinfobox());        while (matcher.find()) {        system.out.println(matcher.group(0));}  }     });     wxsp.parse(); }

demo of regex


Comments

Post a Comment

Popular posts from this blog

neo4j - finding mutual friends in a cypher statement starting with three or more persons -

Image
i trying build cypher statement neo4j know 2-n starting nodes name , need find node (if any) can reached of starting nodes. at first tought similar "mutual friend" situation handled (start1)-[*..2]->(main)<-[*..2]-(start2) in case have more 2 starting points around 6 know name. so puzzled how can include third, fourth , on node cypher able find commmon root amongst them. in above example neo4j website need path starting 'dilshad', 'becky' , 'cesar' check if have common friend (anders) excluding 'filipa' , 'emil' not friends of three. so far create statement programmatically looks like match (start1 {name:'person1'}), (start2 {name:'person2'}), (start3 {name: 'person3'}), (main) (start1)-[*..2]->(main) , (start2)-[*..2]->(main) , (start3)-[*..2]->(main) return distinct main but wondering if there more elegant / efficient way in cypher possibly use list of names parameter ...
Read more

php - How to remove letter in front of the word laravel -

Image
based on picture above result of var_dump() , dd() of same variable ,when var_dump() , apostrophe (') symbol in black diamond of question mark , when dd() same variable , apostrophe can seen letter 'b' appear in front of words what want can me result : loreal's sdn bhd and @ same time want remove special character ( excluding these 4 symbols (-) ,(_) ,('),(,) ) simply escape special character resolve issue. mysqli_real_escape_string resolve issue. $name = mysqli_real_escape_string($request()->user); or use addslashes function() $name = addslashes($request()->user);
Read more

minify - Minimizing css files -

i want know how css files can minimized. currently, using external sites css minifier compress css files. however, after removing whitespaces, comments, etc, file sizes reduced 10-15%. how frameworks bootstrap manage make minimized versions kilobytes. as example, css is: body { background-color: #ff0; } the reduction goes 28 bytes 22 bytes other css files have lot more characters still smaller this. actually minifying code means it's remove thing code white space characters new line characters comments block delimiters it reduces amount of code has transferred on web , speed site load fastly.but unreadable format.
Read more