jeudi 29 janvier 2015

Question on Multiprocessing With Pymongo and Bulk Inserts


I would like to use the multiprocessing library to speed up the time it takes to insert data values into my mongo database.


I found from the following resource: http://ift.tt/1LmaykF


Some nifty results performance wise - but I grew concerned about data integrity and the possibility of bulk inserts stepping on each other if they are comming from different processes around the same time? Is that a valid concern with respect to pyMongo?


I am essentially reaching out to a device, parsing data from them, and writing to a single database using a bulk insert. I scale each host I reach out over multiple processes (via the multiprocessing library) at the same time.


I don't have a lot of experiance with this. So I am seeking guideance to see if I need to be concerned.


Thanks!





Aucun commentaire:

Enregistrer un commentaire