No-SQL. The topography of a database

In our previous BLOG on No-SQL databases, we focused on Big Data.  We explored the idea that because of the enormous size of the underlying data, our former notions of data efficiency and order no longer apply.  Rather than spread related data across numerous, normalized tables, we strive to keep related data together.  In doing so, we greatly simplify the task of retrieving and storing data.  When we need it, the data is stored in one complex record in one table.  One read and we have it.

But in simplifying the retrieval and storage of data, we create complexity of another kind.  How do we keep track of data that we formerly parsed out with logical precision to individual tables?

• Customers can place many orders.
• The orders can contain many line items.
• The line items can, in turn represent many products.
• There are invoices to be sent out
• Backorders to be dealth with, and
• payments to be received.

How do we propose to store all of this data in one record? In answering this question, we find that our data takes on an unusual shape or topography.  Each”record” is no longer flat like Kansas.  On the contrary, it has contours, shapes, and texture, like Colorado.

We find that each of our data records are lumpy.  They accommodate all the data necessary to describe the underlying business or information problem.

In No-SQL, tThe records and tables are so different, in fact, that when we refer to them we must use different terms.  We refer to collections rather than tables because the structure of the collections are diverse enough to accommodate many different aspects of one data problem.  And we refer to documents rather than records because a record implies structural uniformity rather than the diversity of information that the No-SQL database can accommodate.

But do not confuse No-SQL documents with a word document or other kind of unstructured computer text files.  These are highly-structured data-rich groupings of information designed expressly to accommodate our high performance data storage and retrieval needs.

In attempting to understand the benefits of No-SQL, we can find a helpful analogy in physics where the conceptual transition from Newtonian physics to Einsteinian physics comes to mind.  In Einsteinian physics, space is no longer Euclidean.  It becomes curved.  And time is no longer purely fixed intervals, it behaves differently depending on the relative speed of the objects in question.

Similarly, in No-SQL we no longer think of documents (formerly records) as uniform in length or field count.  Documents can contain a variety of related information that is stored together to describe our business problem or data problem.

We refer to the topography of the documents (the diverse shape of the records)  in a No-SQL database.  Understanding this topography and knowing that it is dynamic and can be changed over time with relative ease is a powerful concept indeed.

 

 

 

No-SQL. When our logical assumptions become illogical

SQL or Structured Query language has been the prevailing mode of database organization for over 40 years. The fundamental concepts that form the basis of SQL were introduced in the early 1970’s.  And fifteen years later, in the mid 1980’s standards were introduced that enforced uniformity for all SQL database solutions from a wide variety of the most respected and pervasive software vendors in the industry:  IBM, Oracle and Microsoft to name the most prominent.  What more could we ask for?

• an overwhelmingly logical database structure,
• accepted by the leaders of our industry,
• with standard that promote uniformity and compliance across commercial software products.

But beginning in 2006 and more recently, we find two forces emerging that are challenging the leadership, acceptance and viability of  SQL.

The first emerging force is “big data”.  We are beginning to see databases of 500 billion or more records.  These databases span disk storage devices and even span computers themselves.  For the past 40 years, it was reasonable in traditional SQL to make logical connections between related data sets.  For example:

• An order and the underlying order items are related.
• The customer and customer payments are related.

But the tables in our databases are beginning to take on sizes that exceed our wildest imagination.  And while the task of joining or connecting these “related” tables seems logically sensible, the practical task of doing so for immense data tables is no longer feasible.

Faced with this dilemma, we begin to question the logical purity of SQL and revisit the most fundamental of our data organization questions and assumptions.  Rather than continually splintering and reassembling the components of a logical data record when needed, why don’t we simply store all related information in one record.

We are finding that the very cornerstone and foundation of database logic is being turned on its head.  As is frequently the case, once we challenge our most fundamental assumptions, absent this foundation, elated ideas also begin to give way.  Things that were previously difficult now become easy.  But the contrary is also true.  Those things which were easy with SQL become more difficult.

In subsequent blogs, we will construct a new logical vision.  We will explore these logical issues, the benefits and costs, as we leave the accepted real of SQL and enter the world of No-SQL.

When the price of oil decreases, what happens to the price of gasoline?

We turn on the morning news and learn that the price of a barrel of oil has reached a 6 month high, or a 6 month low, or has not changed at all.

But what is the impact of these oil price changes on the price of gasoline at our local Exxon station?  One would think that the two prices should move in parallel.  But is that really the case?

The table below illustrates how the price of a barrel oil has changed over the past 8 years.  Alongside the change in oil prices, we see the change in gasoline prices.

Year  (A)	Gal of gas (B)	Barrel of oil (C )	Ratio (D)	Price of  gas should be (E)	we overpaid (underpaid) (F)
2005	2.75	66	24	2.54	0.21
2006	2.76	43	16	1.65	1.11
2007	2.76	89	32	3.42	(0.66)
2008	3.91	146	37	5.62	(1.71)
2009	2.70	66	24	2.54	0.16
2010	2.76	78	28	3.00	(0.24)
2011	3.33	80	24	3.08	0.25
2012	3.34	77	23	2.96	0.38

Column B above indicates the price of a gallon of gas.

Column C indicates the price of a barrel of oil.

Column D shows the ratio between the two.  In 2006, a barrel of oil was 16 times that of a gallon of gasoline.  Two years later, this ratio was 37.

Based on an average ratio of 26, column E shows what a price of a gallon of gasoline should have cost, if the relationship between oil and gasoline were constant.

Finally, in column F, we see how much we overpaid or underpaid at the gas pump.  At the moment, we are being overcharged by 38 cents for each gallon of gas that we buy.

The levers of politics : Fear and Greed

Many of us are shocked by the degree of polarization that prevails in contemporary politics.

We are excoriated from the right about our moral turpitude and our profligate spending.  We are harangued from the left about our insensitivity to the plight of the disadvantaged and the inordinate benefits that accrue to the wealthy.

But while those on the extreme right and extreme left make the most noise, these are not the groups that will ultimately determine our political future.  Rather, our political future will be actualized by those who reside in the middle, the fat part of the bell curve. We are speaking, of course, about the vast undecided majority that occupies the middle of our political spectrum.

And which way will this middle of the spectrum decide?

We have suffered and continue to suffer from a faltering economy.  Business seems to course through the veins of our economy as does blood through the veins of a patient suffering from chronic low blood pressure.  There is economic activity, but with little or no conviction.

Given this environment, the health and strength of the economy will surely influence the election.  And as with all things economic, our political leaders turn to those reliable levers of control — fear and greed.

Those on top of the economic heap strive to convince the vast middle that they aspire to be wealthier.  That we are all alike in this regard.  Stick with us and you will be wealthy too.  This group, in effect, appeals to the greed in each of us.

Then there are those who claim to represent our fellow citizens on the bottom of the economic heap.  These leaders send out a different message altogether.  They caution that the real danger in our economic future lies in the failure of the economy to sustain the old, the sick and the less well-educated.  We are all alike in this regard, they admonish.  Our neighbor has become the victim of economic indifference.  The next victim will be one of us. The appeal of  this group is directed to the intrinsic fear in each of us.

From a distance, it appears that we are engaged in  a gigantic game of tug-of-war, with 350 million participants.  Which of our fundamental impulses will hold sway.

Do we associate   with the upper class, driven by Greed to better our lot.

Or do we associate with the underclass, governed by fear that the wheels will fall off our listless  economy and we all wind up in the underclass.

In politics, like economics,  fear and greed are the master controls.

 

Why government savings are just pennies on the dollar

Our federal budget deficit is $1,000 billion dollars each year, year in and year out, as far as the eye can see.  This is an annuity in reverse, every year, on into the future.  After five years, we will have accumulated $5,000 billion of debt.  After 10 years, $10,000 billion of debt.  You get the picture.

So what is the federal government doing about this?  They are looking for ways to reduce spending.  But here is the curious part.  When they find a suitable candidate for expense reduction, they always multiply the savings by 10 and call it a savings over 10 years.  As in a $250 billion reduction in educational expenses over 10 years.  And when we hear of their accomplishment, we are so excited about the $250 billion number that we fail to pay attention.  We never see or hear the part about over 10 years.

And it sounds like a pretty good effort. After all, we are saving $250 billon compared to a deficit of $1,000 billion.  It feels like we are taking a solid chunk out of our annual deficit.

But are we really?   Closer reading, and listening, reveals a painful truth.  The $250 billion in savings is spread over10 years.  In actuality, we are saving only $25 billion each year. 

In other words we are not reducing our annual deficit by 25%.  No, we are reducing it by a mere 2.5%.

And this, after all, is just pennies on the dollar.

We are in budget star wars when budget talk becomes budget hyper talk

I  have not been the most loyal star wars viewer.   I saw the first movie, then maybe the second.  After that I confess that I lost interest.

But for me,    the most vivid memory of my star wars experience was the moment when the space ship’s captain clicked on the hyper speed switch and the craft instantly lurched ahead at a speed that was larger by a factor of 10 or100.   The millions of stars that populated the surrounding galaxy rushed past in a blur of streaking white shapes.

For today’s economic discussion, the notable issue is not the stars speeding past.  It is the instantaneous increase of speed by a factor of 100.   So it is with the economic discussions of our day.  We talk about dollar amounts that approach $500 billion, $600 billion, $900 billion.  And then, all of a sudden, we are back at 1.  Not 1 billion, but 1 trillion.

To maintain context and continuity of the economic narrative, we should not let our political or economic leaders refer to $1 trillion.  Rather we should insist that they call it like it is $1,000 billion.

After all, which sounds larger, 800 or 1?  If someone were to tell you we were incurring a budget deficit of 1 something, or 800 something, which would sound smaller?  If you were to say the 1, you would not be alone.  And that is the problem.

We should not permit our politicians to hypershift from billions to trillions.  It lulls the vast majority of the public into a sense of comfort and complacency.   If our deficit is larger than $900 billion and is approaching $1,000 billion, let’s call it like it is.  Do not let our political leaders downshift to trillions to make us feel more comfortable with what is fundamentally a very large budget indeed.

When I think back to my childhood, my father would frequently ask which is heavier. a pound of lead or a pound of feathers.

I ask you the budgetary equivalent.  Which is larger?  A budget of $3,800 billion or a budget of $3.8 trillion?

Economics 101 — How did we have a Federal Budget surplus in the year 2000

Who among us can remember the most controversial economic topic of the Presidential Campaign of the year 2000?

As unimaginable as it sounds today, the topic was  “what to do with the federal budget Surplus“.  That’s right, Surplus !!

Without arguing about what individuals or events were responsible for growth of the budget deficit during the intervening years, let’s simply take a look at what happened to the dollar expenditures during the 11 year period from 2000 to 2011.

Total Federal expenditures in $Billions

	Federal budget item	Year 2000	Year 2011	Largest budget items
a	Social Security	$449	$775	3
b	Health care	352	858	2
c	Education	60	113
d	Defense	358	878	1
e	Welfare	176	472	4
f	Protection	28	56
g	Transportation	47	93
h	General govt	15	29
i	Other spending	80	96
j	Interest	223	230	5
k	Total Spending	1,788	3,600
l	Surplus (Deficit)	$236	($1,299)

As we can see, the total expenditures (as shown in row k) have more than doubled in these 11 years from $1,788 billion to $3,600 billion dollars.

If we look further, we can see that the top 5 expense items are

Social security,
Healthcare,
Defense,
Welfare  and
Interest

In fact, these five items accounted for 87% of the total budget in 2000.  By 2011, they accounted for 89% of the budget.  Arguably, these are commitments and entitlements with no flexibility whatsoever.

I am not trying to justify this growth in expenditures, nor am I trying to argue for a reduction in these expenditures.  I am simply attempting to establish a common understanding of where the money is going.

Now let’s look at these same Federal expenditures from a different angle.  What percentage of the total do we spend on defense, or social security, or healthcare?  And how has this percentage changed from 2000 to 2011?

Federal line item expenditures as a percent of the total budget

		Year 2000	Year 2011	Change in Pct of total	Change in Pct of total
a	Social Security	25.11	21.53		(3.58)
b	Health care	19.69	23.83	4.15
c	Education	3.36	3.14		(0.22)
d	Defense	20.02	24.39	4.37
e	Welfare	9.84	13.11	3.27
f	Protection	1.57	1.56		(0.01)
g	Transportation	2.63	2.58		(0.05)
h	General govt	0.84	0.81		(0.03)
i	Other spending	4.47	2.67		(1.81)
j	Interest	12.47	6.39		(6.08)

As you can see, healthcare, defense, and welfare have risen dramatically as a percentage of total expenditures.  Social security and interest expense have fallen as a percentage of the total.  The other budget items have changed negligibly.

Perhaps the most interesting item is social security.  In total dollars, this item has risen 72%.  Even so, it has become a dramatically smaller part of the total budget in 2011 than it was in 2000.

And watch out for that interest expense item on row j.  If rates were to increase the windfall that the Federal Government has been receiving will disappear.

We now have a sound basis for understanding the trend in Federal spending.  This common understanding will be a foundation for future discussions.