No-SQL. When our logical assumptions become illogical

SQL or Structured Query language has been the prevailing mode of database organization for over 40 years. The fundamental concepts that form the basis of SQL were introduced in the early 1970’s.  And fifteen years later, in the mid 1980’s standards were introduced that enforced uniformity for all SQL database solutions from a wide variety of the most respected and pervasive software vendors in the industry:  IBM, Oracle and Microsoft to name the most prominent.  What more could we ask for?

• an overwhelmingly logical database structure,
• accepted by the leaders of our industry,
• with standard that promote uniformity and compliance across commercial software products.

But beginning in 2006 and more recently, we find two forces emerging that are challenging the leadership, acceptance and viability of  SQL.

The first emerging force is “big data”.  We are beginning to see databases of 500 billion or more records.  These databases span disk storage devices and even span computers themselves.  For the past 40 years, it was reasonable in traditional SQL to make logical connections between related data sets.  For example:

• An order and the underlying order items are related.
• The customer and customer payments are related.

But the tables in our databases are beginning to take on sizes that exceed our wildest imagination.  And while the task of joining or connecting these “related” tables seems logically sensible, the practical task of doing so for immense data tables is no longer feasible.

Faced with this dilemma, we begin to question the logical purity of SQL and revisit the most fundamental of our data organization questions and assumptions.  Rather than continually splintering and reassembling the components of a logical data record when needed, why don’t we simply store all related information in one record.

We are finding that the very cornerstone and foundation of database logic is being turned on its head.  As is frequently the case, once we challenge our most fundamental assumptions, absent this foundation, elated ideas also begin to give way.  Things that were previously difficult now become easy.  But the contrary is also true.  Those things which were easy with SQL become more difficult.

In subsequent blogs, we will construct a new logical vision.  We will explore these logical issues, the benefits and costs, as we leave the accepted real of SQL and enter the world of No-SQL.