Bulk insert from MongoDB to elasticsearch python -


background

i have data in mongodb collection(not lot ~100 mb).it consists of several documents.pymongo queries collection user input.through query appropriate document need searched , returned.the data won't update regularly.

i have decided use elasticsearch in order search , return relevant document.my workflow is(when queried everytime)-

retrieve documents mongodb-->bulk insert in elasticsearch--> search using elasticsearch-->return top document.

i using pymongo,elasticsearch-py

problem

as bulk insert every query fired,the documents keep on accumulating(the count variable increases everytime number of documents in collection).as understand need bulk insert once.but seems bit hacky.(setting run bulk if count==0)

from elasticsearch import elasticsearch elasticsearch.helpers import bulk pymongo import mongoclient   es = elasticsearch()    index_name = "documents" type = "document" stop_words=[]   client=mongoclient("uri")  db=client["database"]  collection=db["collection"]    def make_documents():     faq in collection.find():         doc = {             '_op_type': 'create',             '_index': index_name,             '_type': type,             '_source': {'text': faq['answer']}         }         yield (doc)      # put documents in index in bulk bulk(es, make_documents())  # count matches count = es.count(index=index_name, doc_type=type, body={"query": {"match_all": {}}})  # can searches. print("ok. i've got index of {0} documents. let's searches...".format(count['count'])) while true:     try:         query = input("enter search: ")         result = es.search(index=index_name, doc_type=type, body={"query": {"match": {"text": query.strip()}}})         if result.get('hits') not none , result['hits'].get('hits') not none:             print(result['hits']['hits'])         else:             print({})     except(keyboardinterrupt):         break 

i need load data in elasticsearch once , query lot of times without updating elasticsearch.i can wildly incorrect in approach new elasticsearch , feel there has better way.


Comments

Popular posts from this blog

neo4j - finding mutual friends in a cypher statement starting with three or more persons -

php - How to remove letter in front of the word laravel -

minify - Minimizing css files -