mardi 27 janvier 2015

How to manage a large list of distinct numbers


I have a list of around 2 billion distinct numbers stored in memory for some computation. Currently, every time I need to add a new entry I have to search the entire list for a potential duplicate.


I have 18 digit, 2 billion numbers to be stored in memory. Some of the numbers are repeated. So, I want to load new number in the memory and ignore if the number already exists. Right now, I append the number in the list if it does not exist already.


This process of linear scan of the already existing number is taking long time. What can I do to get around this performance problem?





Aucun commentaire:

Enregistrer un commentaire