There are many classical applications that use a relational, transactional database (e.g. postgreSQL, MariaDB, mySQL, Oracle, MS-SQL-Server, or DB2) to store and access their data. Since both computers and databases have reached a very advanced technical standard, it is possible to handle many million data records within a single table, provided you have a carefully modeled and optimized database and good software for retrieval. We all know Wikipedia, which epitomizes this approach and never fails, despite its sheer number of articles and the simultaneous access of millions of users, to immediately deliver the article you were looking for (if it exists).
Having expanded an application to process call data for prepaid wireless service in a previous project, we have the expertise needed to develop applications for the processing of large data volumes within a short time frame.
However, everybody also knows Google, Yahoo, Yandex, and other search engines, where the processed data volumes and simultaneous access are even more gigantic. The above-mentioned databases could hardly manage such a workload; however, ACID transactions are not necessarily required either. Enter Big Data, a technology that opens a world of opportunities, not just for the creation of yet another search engine. It is also a technology that offers advantages for borderline situations that could theoretically still be handled by relational databases, but where a Big Data approach saves you time and money.
It is important to grow a sound understanding of the task at hand and the nature of the data to develop an architecture that allows extensive parallel computing of processes, keeping them as separate as possible. In these cases, NoSQL databases are the data storage facilities of choice.
Having not only used but tailored such a NoSQL database to meet specific business requirements during a previous project, we have the expertise to work with these technologies.
Count on IT Sky Consulting to help you with any software project that requires special algorithms.