Blockchain Poised for Innovation

The buzz around Bitcoin and the broader blockchain ecosystem has been simmering for years, in particular how this technology as an infrastructure will revolutionize / disrupt / and otherwise affect most (all?) transactions conducted today.  However, despite the growing hype (and increasing market value), blockchain has yet to materially affect many areas, save the black market and a few other niche transactional systems.  I believe this is about to change.

Led by Ethereum and others, new code sets are going to production that will (finally) enable real innovation to happen.  Hyplerledger's Fabric goes to production early next year.  R3's Corda is making steady progress.

I view this time as "big data" was in the early 2000s - the promise of the technology was understood by growing numbers, but the effort and expense required to realize any of this promise significantly hindered adoption and further innovation.  Then Hadoop and other infrastructure technologies became available, lessening the effort required to deploy big data applications, while also expanding the pool of engineers capable to get up to speed and deploy functioning applications.

With the rise of Ethereum, the upcoming release of the initial Hyperledger projects, and R3's Corda work, I see that the time is now to begin allocating resources to actually building things using blockchain technology.  The pace of maturation for these and other projects is accelerating, opening up a wealth of infrastructure to enable real application of blockchain technology. It will be fun to watch!

A Different Way to Go to Market in China

When most US persons think of China, they see its vast, authoritarian central apparatus and it's one-party political system.  All roads lead there, and therefore to be successful in China, you must work directly with this apparatus.  This view is wrong in many (most?) cases.

The primary mission of the Chinese central government is stability.  Progress towards economic development, displays of military strength, even the kerfuffle in the South China Sea is boiled down to the continued stability of the system.  (The South China Sea is the primary route for CHina's growing need for energy imports, among other resources integral to their continued growth.)  With such an ardent focus on stability comes a strong disdain for risk, the lifeblood of innovation.

Stating the obvious, the Chinese market is different than most any other.  Cultural norms, economic structures, and paths to execution all make for a steep hill to climb.  Each of these elements (and more) also add risk to even the most proven ideas when brought in to this market from other regions of the world.

Rather than focusing on the central apparatus, instead look to the provincial level to support your go-to-market ambitions, ideally well outside the hubs of Beijing and Shanghai.  There you will find emerging leaders thirsty for risk, to help bolster their credentials for future roles within the national party leadership.

Given the emerging nature of the economy, the leash on what the provincial authorities can and cannot do is long, much longer than the US system.  Leeway can be found to experiment and iterate, especially when the focus is on further developing quality of life for the Chinese people.  Rules can be bent as long as the value is evident.  Time can be given to adapt.  Metrics and case studies can be developed, proving the success of the idea inside the Chinese system.  Once proven, the central apparatus can be approached for a larger rollout.

This is of course not the right path for all.  However, for many US businesses eyeing China, targeting provinces first can offer an optimal path towards accessing this vast and growing market.

American Biases

One of the core issues of working between China and the US is the cultural biases we both impose on our  personal interactions.  As with any bias, these biases are often born from experience, direct or tangential.  Sadly, when it comes to the American media, they are too often willing accomplices in reinforcing such biases, particularly when it comes to China.

In this case, when I refer to the American media, I am referring to the grand idea of great institutions of our day, that have developed a process and a skill to search for the idea of truth.  Objectivity is still the goal, if not always attained.  I know this is idealistic in this political climate, but I have seen first-hand an institution that represents this idea at its core.  Unfortunately, with this same institution, it seems the idea of objectivity is thrown out the window when it comes to China.

Take, for example, this article from today's Washington Post.  The idea that "China" is still conducting government-sanctioned cyber attacks is sexy, and it fits our perception of both an untrustworthy leadership (Xi), and an evil enemy (China).  There are two issues with such a report.  The first is that the decision to halt such attacks was made just a month ago.  To expect the government apparatus of 1.2 billion people to halt activities in such a short time is naive to the challenge of managing any large organization, let alone one such as China's.

The second issue is that the article does not attempt to define "government".  I wonder if Ms. Nakashima even asked her US sources to define the organizations they label as government sources of the attacks?  If she did, she does not report on their answer.  China is a nation of 34 provincial governments, hundreds of state-owned enterprises, among many other government operations.

The idea that the central authority dictates the micro actions of every government-tied organization throughout the land is wrong.  Americans (and the American media it seems) would be surprised to learn that the provincial authorities wield tremendous power, in some instances significantly more than even our state governments.

To assume that a cyber attack happening now is at the behest of the central authority is naive, and to not dissect these questions further as a a journalist is at best lazy, and at worst negligent.

Impact of Finance

I have been following this great multi-part project at The Washington Post on the declining middle class.  The latest piece struck a chord with me.  I lean left - I generally believe people are good, and that many need assistance when times are tough.  Assistance is not only necessary, it is a core component of what makes us human.  Fraud happens, but that should not distract us from helping our fellow neighbor.

I am now in the finance world.  I get more and more exposure each day of the machine that Mr. Tankersley describes in the article above.  So, knowing this, I have a quibble with this article.  It is extremely narrow-minded.  To measure the impact of finance solely by it's contribution to the economy is myopic.  Such an analysis negates all the externalities that occur given the growing investments in finance.

As I marinade on the system that is finance, we are benefiting elsewhere by the development that has occurred over the last few decades.  In particular, I argue that the growth of the internet as a foundational contributor to our economy is directly tied to the learnings, development, and talent cultivated throughout the recent finance evolution as Mr. Tankersley describes.

The metaphor used in the article, that finance is like a plumbing system enabling capital to move from those that have to those that need, is directly akin to how many describe the internet.  The exorbitant growth of the finance system in recent decades has led to new technology discoveries and an emerging talent pool - assets that are rapidly transferring to other industries.

Take data intelligence, for example.  Over the last decade, we have seen hyper growth in the amount of structured data, given the growth of the web and other advancements.  Vast datasets are becoming accessible for analysis in numerous industries, from commerce to education to even government.  Many of the technologies to handle these vast datasets are rooted in finance, as are many of the early people, transferring their knowledge and experience.  And growing the economy...

Structuring the FinTech Disruption: Apple vs. MCX < Apple + MCX

For anyone following the general disruption happening of the electronic payments space, or the Apple Pay / MCX war in particular, this exchange is an interesting data point.  There are two takeaways I see from this.

The first is that "it's on".  The siege by new players / models on the electronic payments ecosystem is starting to impact the conversation, if not the bottom line (yet).  Apple Pay and other technologies are beginning to pierce the fees framework that has long filled the coffers of the existing electronic payments system.  Today the more successful emergents operate as a friend to the existing players (but that will change).

The second takeaway is that the existing players do not yet fully grasp the siege underway.  The war is not Apple vs. MCX; it's Apple + MCX vs the status quo.  In this discussion, Apple Pay is being blocked by retailers because it is "too friendly" to the existing system, perpetuating the existing fee structure rather than attacking it.  MCX and it's parter retailers understand that the existing fees structure is outright robbery given today's technology, and they are taking a stand.

The Apple Pay / MCX battle is one between two armies on the same side, each with different strategies and assets.  Apple doesn't win by taking a portion of the existing fee structure, and neither does MCX.  They and the other disruptors win by decimating the economics of the current system.  Apple is building an army of users before striking.  MCX already enjoys its army of retailers and sees Apple as a threat to this asset.

Battles will be lost, but the war will be won.  This salvo may not strike the death blow, but more attempts are coming.  The existing electronic payments system is bloated with technology that has been displaced with significantly cheaper, more secure options.  Significant value that the current system enjoys will be returned to the consumer (through lower prices), retailers will retain some as will new players within the system (e.g. Apple).  The days of merchants paying 2 - 4% per transaction are numbered.

Blockchain and the Internet of Value

We have seen how the internet of information has transformed the world over the last two decades. More recently, the internet of things has emerged as a new frontier of innovation. Following close behind that is what some call the internet of value. Like its siblings, the internet of value is decentralized, open, and a blue ocean for innovation.

The potential for an internet of value goes beyond just making more efficient the existing trillions of world commerce. Wences Cesares, CEO of Bitcoin startup Xapo, provides themetaphor of phones:

There were never more than 1.2 billion landlines. Then the cellphone came and we’re at 6.3 billion. Why? It’s not because only those people wanted to communicate. The landlines were all post­pay. You need to have credit to get one. The cell phones were pre­paid. Suddenly you could get one with cash. It had nothing to do with technology. It was an economic restriction. Now there are 1.5 billion bank accounts, same threshold as land lines. I think [an internet of value] will allow us to see 6.3 billion people banking on their cell phones. That’s what’s so exciting to me. That’s a much better world than we have today.

Many like Cesares believe an internet of value will lead to a similar improvement in enabling access to more commerce.

The primary technology that will lead to an internet of value is blockchain. For the purpose of this blog post, we use blockchain to refer to any transaction processing engine with no central authority required that allows for contracts and transactions between parties. “It’s a technology that allows data to be stored in a variety of different places while tracking the relationship between different parties to that data." (source)  Blockchain is considered the biggest disrupting force in financial services, according to Oliver Bussmann, CIO of UBS.4 Bitcoin is the most widely known example. Another of note is Ripple.

As noted in the definition, the opportunity of blockchain is not limited to currency­based transactions. The technology can be applied to any transaction or contract where security and trust between participants is needed.

There are two areas of near-­term interest as we work to understand the blockchain impact; blockchain as it relates to digital currency (i.e. Bitcoin), and blockchain as an infrastructure (i.e. Ripple). Most focus on blockchain’s impact on fiat currencies, given the emergence of Bitcoin, and the companies and infrastructure that are required to deliver on such a vision.

Blockchain Currency: Bitcoin

Bitcoin is the most prominent blockchain­-based currency in circulation. Others, often referred to collectively as “altcoins”, have not yet developed significant markets as compared to Bitcoin, taking only 8% of all value stored among blockchain-­based currencies.

Bitcoin is trading at 30% of its high as of late. Despite the recent price decline, many observers suggest that we look at the entire ecosystem emerging around the currency, rather than its market price, to understand its growth trajectory:

As the above metrics indicate, the Bitcoin ecosystem is growing. In addition, an entire startup ecosystem is emerging surrounding the development of Bitcoin as a currency:

Wallets store Bitcoins for future transactions. Payment processors manage the transactions themselves. Exchanges bring together the parties of the transaction ­ the buyers and the sellers. Financial service providers are building services on top of the Bitcoin ecosystem. And Miners operate the underlying technology that maintains the trust within the system.

Bitcoin growth has been particularly significant in China in recent months: 

(slide images via CoinDesk)

As the Bitcoin ecosystem grows and matures, the opportunity evolves beyond its role as a currency. Beyond its role as a currency is what else can and will be done with the system. A concept is known as pegged sidechains, “which enables bitcoins and other ledger assets to be transferred between multiple blockchains (source)” is expected to enable non­currency transactions with the same flexibility and security as the current currency transactions enjoy. Some of the altcoins launched have been developed for other transactions beyond just currency. However, none have managed to develop enough of a market to have impact. Enabling this sort of technology within Bitcoin itself eliminates much of the current impediments to innovation in this area.

Blockchain Infrastructure: Ripple

The currency side of Blockchain is interesting and potentially transformative, but it is still very immature. The more relevant area, however, is blockchain’s potential at revolutionizing any transaction infrastructure, significantly reducing costs and improving system security. Bitcoin and most altcoins are trying to revolutionize the current system, purposely intending to supplant the existing players. A nascent few, however, are working within the system to deliver the economic benefits without the intent of displacing their existence.

Ripple is an example of how blockchain can improve on existing banking infrastructure. First focusing on international money transfers, the Ripple protocol allows banks to transfer value anywhere in the world for the minimal cost of implementing its technology. Transfers themselves are near free, and are transacted in the home currency of both the sending and receiving bank. Exchange rates are made part of the same transaction in real­time, reducing exchange rate risk.

The Ripple protocol is open sourced ­ there are no license or usage fees associated with its implementation. Ripple makes its money off the underlying currency that supports the protocol. As more banks employ the protocol to transact, the value of the underlying currency increases.

So far Ripple is the only startup we have seen that is working within the existing system to implement blockchain technology. They already have representation within China. However, significant opportunity exists to support their efforts, should their intent prove valuable to the banking community.

Phases of Data Intelligence: An Update

A couple of weeks ago, I wrote about the Phases of Data Intelligence.  I just came across Benedict Evans' presentation at GE's Mind + Machines event.  He had a more simple yet similar version of the same idea.

His version is:

• Build the data stream
• Ingest the data within the enterprise
• Do something useful

He has a plethora of "3-bullet" nuggets on the Industrial Internet. Worth a watch if you are in to that sort of thing:

Network Effects, Compounding Interest and Inequality

I have seen a rash of "the rich are getting richer" stories in recent months, because, well, they are. But most articles I have consumed focus on tax policy and other mechanisms that have for the most part assisted in this shift of wealth over the last few decades.

This one took a different tact, and it got me thinking about the impact of the concepts of network effects and compounding interest on inequality.  This article focuses on the environmental effects of having wealth on a child's development.  Interesting, but there is a larger ecosystem at play that further expands the gap between rich and poor: the network.

I am fortunate in that I grew up in the US in an upper middle-class family.  I did not have a ready-made network for me to access, but I had other assets that have allowed me to develop my own network much more rapidly and effectively than many.  Principally, my network began because I had more flexibility as to how and when to earn an income.  Paying for college was not a worry, and my living expenses were subsidized (thanks Mom & Dad!). affording me to take an unpaid internship at The White House while in college.  This opportunity not only began my career, but also began the development of my network that continues to compound and pay dividends.  I have reconnected with several folks I met during those early years over the last year, as I develop my latest business.

Providing opportunity is one component of a much larger ecosystem.  Those in need also must have the means and flexibility to fully exploit opportunities.  Policy, philanthropy and other initiatives that attempt to address inequality have to look at the entire ecosystem.  Failing to do so may lead to exponentially less potent effects.

Industrial Internet

I have been following the "internet of things" idea as it has emerged over the past few years.  I own a Nest.  I want a Sonos.  Etc, etc., etc.  But, I've always struggled at understanding the underlying value of such systems.  I haven't seen real problems solved by these devices yet.  I have no doubt that it will come, but the space is just too immature at this point.  Enter the "industrial internet".

I had the fortune of attending the Colorado Innovation Summit in Denver back in August.  There, GE's Jeff Immelt spoke of their significant investment of what they call the industrial internet.  Essentially, this is the enterprise version of internet of things, or industrial IoT.

My "ah ha" moment was the realization that opportunities in industrial IoT can win simply with a clear ROI.  And win they are - Mr. Immelt spoke of several examples already in play among their businesses where they are applying industrial IoT to deliver double-digit returns.

The same is true on the consumer side, but the definition of return is so much more difficult given human behavior.  And how that return is delivered on the consumer side can take so many forms.

However, among key industries, incremental improvements in efficiency have significant impact on profitability, given the minimal cost of implementation of many industrial IoT concepts.  These investments will have a huge impact on every industry and the global economy in the coming years.

Source: GE Estimates

Phases of Data Intelligence

"Big Data" has been a thing for a few years now.  As with any new idea, the hype and promise of what it is can overshadow the effort required to actually deliver.  Big data is no exception.

Many focus on the promise of "predictive" intelligence without understanding the effort of basic data collection.  Others focus on dashboards and other tools without building the infrastructure needed to deliver those pretty pictures.

Through conversations with many, I see the following stages to data intelligence:

Stage 1*: Data Collection - many systems, processes, tools already through off a ton of data.  For most industries and applications, this data is often not stored, let alone organized for use. 

Stage 2: Data Infrastructure - for storage and organization, quality infrastructure is required.  This is where much of the foundational innovation has happened in the last 10 years or so, that has spurred idea of big data.  The idea of it being too costly or too difficult to store and organize vast amounts of data is no longer true.

Stage 3: Data Visualization and Interpretation - this is an area that some skip by either hubris or eagerness.  Hubris is when those not in the trenches believe they know the right path to extract intelligence from data, and build accordingly.  Eagerness manifests by going after predictive intelligence before knowing what data and information is available. 

Stage 4: Data Intelligence - this is the stage where real value is delivered.  The steps above are the plumbing to get you to this point of actually learning from the information gleaned from data. This is the stage where action is taken, given what is learned.

Predictive intelligence is an extension of data intelligence, whereby historical data is used to preemptively make decisions in the future.  As "cool" as it is to do, there is so much that can be learned and value uncovered by effectively developing intelligence from historical data.

*Given that there is so much data thrown by existing systems and processes, I take the data stream as a given.  This may not be true for some markets / industries, but it is fast becoming the norm that the data is there for the taking / analysis / employment...

LinkedIn and the New Economy

The following is either A) an insightful, educated observation of the true reality of the disruption underway of the middle class in the US, or B) a misguided diatribe that is a result of the availability heuristic and a "general" education.

For the purposes of this discussion, we will define the middle class as those that earn +/- 50% of the median household income within the US - currently ~$50K and change.  (This number has apparently been going down over the past decade or two.)

Over coffee with a friend yesterday morning, our conversation diverged to how our lives would be different had we had the tools and knowledge we have today even 10 years ago.  The world today is different, and the skills, tools and other resources necessary to thrive are different too.  My anecdote is that, had I had LinkedIn (and the attached awareness of the value of a nurtured network) while I was traveling the world on behalf of President Clinton, I would be in a different economic rung than I am now.   At the least, "change" would be "easier."

Don't get me wrong - my wife and I live well.  We are in the top 20% in terms of income, likely the top 10%.  I was born in at best the second 20%, and I grew up as our economic situation continued to

improve.  By the time I entered college, my family was well ensconced int he top 20%.  However, my take on the catalyst that enabled this rise within the socio-economic strata is that my Dad decided to make a change in 1980.  He switched careers, from being a school psychologist to pharmaceuticals.  And, with this change cam great opportunity, which he seized.

I remain in contact with many of my White House colleagues, with many reconnections made through my LinkedIn and Facebook accounts.  My point is that I cannot remember let alone contact people even half the people I worked with back in those days, people that likely could be helpful to me now and in the future.  Change is easy for me now, but it could be a lot easier with a wider, more diverse network.

My hypothesis is that the supposed "shrinking middle class" is less a result of a perilous economic attack, and more a result of the disruption of how to succeed - not unlike what has happened to music, is happening to journalism and television, and will soon happen to higher education, among other industries.  Lost in this disruption is a middle class that has a job for life, is promoted every 4 - 5 years, does roughly the same job for most of his / her career.  In it's place is an agile, undulating timeline of new roles, new responsibilities, new companies, and new colleagues.  There is no straight line in one's career trajectory, and those that understand that and equip themselves for that fact, prosper.  Those who do not stagnate.  Those that have built a network for change thrive.  Those that do not, wither.

What we are seeing in the census data is this shift.  The top 20% consists of those that understand and have adapted to this disruption, and are reaping the reward, hence the continued growth of their share of income.  As more understand this new world, more will prosper.  As our system adapts to the new reality, so too will the distribution of income.

Or not.  We shall see...

China

I had the fortunate opportunity to travel to China for the first time last week - what an amazing place.  I got on the plane alone, with no colleagues, and only an itinerary of my flight there and my flight home.  I was invited as a guest of a Chinese entrepreneur.  I cannot imagine a better way to see China than as a guest of a business leader.  And, being alone enabled me to consume the experience at all times - no opportunities to fall in to catty "American-centric" conversations.

I come away from the trip seeing more similarities than differences - among the people, within business, and even in how this Communist government operates.  Conversations were frank, direct, and untethered by political doctrine or even perceived social norms of Chinese culture (unlike my experience in Singapore).  The mode of business is one of service and immediate opportunity.  And, the government is trying to have the best interests of its people at heart, despite its often authoritarian ways.

I have a newfound sense of scale - there are 10 cities there larger than New York.  As I explored Dali City, a city roughly the size of Austin, my stomach churned at the site of what appeared as overbuilding to me at the time.  The bulk of the city appears to have been built int he 80s, if not prior.  It consists of Russian-style architecture, or Bali style native to the region.  Yet, there were several complexes recently completed, and by my count well over 50 projects underway - everything from a 5-star hotel to 5 - 10 building complexes.  It appeared that they are expecting the population to double overnight.

However, after discussing it with my Chinese friends, I realized how little is needed to fill these new apartment buildings, given the scale of growth underway and the vast population still living in the rural countryside.  I am still trying to understand how this scale impacts my view...

Hoffman Misses Key Asset of Current Diploma System

Reid Hoffman pens an interesting piece on how the diploma needs an upgrade.  Though I agree wholeheartedly with his premise, he misses one critical point that enables the current system to thrive - the value of fuzziness.

The root of any economic system is information - who has it and how good is it.  The value of a diploma from a 4-year institution is no different.  Hoffman describes well the pains some people have in clearly articulating the value of their education beyond the blunt instrument that is the diploma (the "sell" side of diploma value), and the tactics employers use to reduce their pools to the most likely candidates (the "buy" side).

However, he neglects the fact that this system provides access that otherwise would not be available for many, given the fuzzy reality of a given person's credentials due to this bluntness.  Think of the kid that skated by without going to class, or the one who only took the "easy" courses.  Would they have had the same opportunities if the system of measurement had been more fine-grained?  Will they engage in a system that penalizes them?

There is another similar issue with fuzziness that will stifle the embrace of a more modular system by the employer  - the fact that most employers do not have the information they need to hire well for a given role.  There is so much bias and prejudice that clouds the current hiring process in most firms that more information could hinder an already clunky process, rather than help. More information generated by a fine-grained system will take more effort from the hiring manager to sift and understand as well.  Roles will have to be broken down in more detail, and more time may be required to assess potential fits.

A critical key to any system that attempts to improve an imperfect but functioning system is to ensure these fuzziness issues are addressed on both the buy and sell sides.  Failure to address them adequately will stop any potential replacement of an archaic yet functioning economic system.

Data in Context

Garance Frank-Ruta clarifies the context surrounding the data "discovered" that show former IRS Commissioner Douglas Shulman visiting The White House 157 times during his tenure.  The short story is that the data used to make that claim is imperfect.  A large majority of the supposed visits were in fact unfulfilled invites.  Another claim made from the same dataset is that he visited more often than any cabinet member - another falsehood given that the system referenced is used primarily to allow access to those walking in to the complex - cabinet members, given their seniority, are able to drive on to the White House complex.

This is yet another example of how important it is to leverage data thoughtfully.  Intelligence requires deliberate thought, not quick assertions and grandiose conclusions.  Minimal effort would have reveled the imperfections of the data referenced.  (The system used was built to track appointments within the White House complex, but only for meetings and "typical events.  Access lists for larger events often forgo the use of this system, as do appointments involving more senior government officials cleared to drive in to the complex.)

To ensure one does not fall in to this trap, there are three questions you must first answer, before acting on the information gleaned from a particular dataset:

1. How was the data collected?
2. What specific data is included in the dataset?
3. And, most importantly, what specific data is NOT included in the dataset?

Only with such context can you begin to understand the information available...

Global GDP

Let this one marinade for a minute:

Information Efficiency vs. the Boogyman

I hate when writers use the boogyman to scare people.  Michael Carney does just that with his article on personal data, "You Are Your Data: the Scary Future of the Quantified Self Movement".

I don't negate the fact that a small minority will "do evil" with the growing exposure of personal data.  My point is that someone of Michael's stature and position should not focus on what will undoubtedly be a small faction, at the expense of the larger, more bountiful majority.  The quantified self (and an exponentially increasing other sets of data) are and will continue to deliver value, much of which we are only beginning to see.

From Michael,

For those of us who don’t measure up compared to the rest of the population, the outcome won’t be pretty.

But what about those that are unnecessarily penalized, given today's information inefficiencies?  The truth is that the industries he cites become more efficient with more (personal) data.  Insurance is at it's heart based on information - the more information available, the more effectively and efficiently risk can be priced.  The more risky clients pay more.  Market dynamics at work.

Health insurance, even home mortgages, are quantified bets given the information made available.   Yes, people will have to pay more, but others will have to pay less.

He finishes with an acknowledgement that he is not focused on the value.  Rather, he bases his argument on the need for user awareness.  I agree that privacy policies and terms of service documents need more transparency and less legalese. Using the boogyman to make the point is wrong.

Calling Bullshit on Big Data

This article has a decent list of ways to call bullshit on data-driven analyses.  Click the link for context, but here are the top points:

1. Focus on how robust a finding is, meaning that different ways of looking at the evidence point to the same conclusion. 
2. Data mavens often make a big deal of their results being statistically significant, which is a statement that it’s unlikely their findings simply reflect chance. Don’t confuse this with something actually mattering. 
3. Be wary of scholars using high-powered statistical techniques as a bludgeon to silence critics who are not specialists. 
4. Don’t fall into the trap of thinking about an empirical finding as “right” or “wrong.” 
5. Don’t mistake correlation for causation. 
6. Always ask “so what?” 

As often occurs with an emerging technology theme, the glitz and glam of the shiny new thing that is big data often overshadows the real value.  The above list is a great start in being sure that the data product or opportunity being pitched truly can add value to your mission.  

#3 is an interesting one - I see a trend in the emerging big data space that vendors and others seeking to exploit big data too often move to high end, overly complex mathematics, when more basic, easier to understand models would suffice.  This is especially true when building out new applications on top of large datasets.  You will often get to the productive answer faster by building simple prototypes before investing more expensive resources.  Data modeling is no different.

#5 above is a particularly important point.  My sense is that it is difficult for most to logically separate the concepts of correlation and causation.  I find myself jumping too far too often, by inferring to much import on a basic correlation that lacks any evidence of causation.  

At the end of the day, high end mathematics do not negate basic economic theory.  Be smart - don't forget your whits when digging in to big data...

Munging Moore's Law and Gay Rights

Moore's law states that computer processing power will double every 18 months or so.  There has been all sorts of extrapolations as to what this may mean to us as a society, the Singularity being one.  I've got another: Gay Rights.

I did a paper back in college (late 90s) on gay marriage - I still remember the feeling of astonishment that, by that time, no state had yet allowed same-sex couples to marry.  If I recall correctly, only a few allowed civil unions.  As of this writing, 9 states now allow same-sex marriages, and several others are well on their way.  That is a major cultural pivot in just 15 years.

My take is that the speed of the pivot has a lot to do with Moore's Law, or rather, the infrastructure it has enabled.  As computer processing has grown exponentially, so too has the speed of communication.  We have moved from The Pony Express to the Daily Paper to the 24-hour News Cycle to now near instant delivery, with each leap coming faster than the last.  In a similar vein, social networking has expedited the sharing of opinions and thoughts among friends.  What used to happen periodically on the front porch is now a constant stream.  Communication is exponentially faster, and so too are its persuasive properties.

As opinions change, the impact of that change radiates with rapid speed.  As one friend openly seeks to understand marriage equality, all connected friends are exposed to this shift.  Even as a lone NBA player comes out as being gay, the rapid dissemination (and exploration) of this story takes over like never before.  As with Moore's Law, change is happening exponentially faster.

David Brooks, Your Premise is Off!

I've already blogged about some of David Brooks' writing on big data.  Though it is admirable that he is taking the time to delve in to the emerging world of data, he needs to apply some differential thinking to the information he is collecting.  In this piece, his premise is again off:

The theory of big data is to have no theory, at least about human nature. You just gather huge amounts of information, observe the patterns and estimate probabilities about how people will act in the future.        

This is not the theory of big data - this is a small sliver of what is and can be done with the explosion of structured data that is popping around us.  To diminish the power of big data to just what can be gleaned through "estimated probabilities" is to focus on the tree and not the forest.

The power of data is in the information it contains, not the method by which it is extracted.  And the limit is our imagination.

The Philosophy of Data

In this article titled, "The Philosophy of Data", auther David Brooks asks:

What kinds of events are predictable using statistical analysis and what sorts of events are not?  

Now, I know an editor likely created the title, but his article limits the value of data to insights derived from statistical analysis  - as if that is the only means to extract information from (big) data.

I think this is the wrong question to ask.  This may be a bit optimistic, but my belief is that data analyses can answer most any question.  The problem (and opportunity) lies in ensuring the data contains the necessary information to answer the question - a problem we have only begun to explore.

In the same article:

...we tend to get carried away in our desire to reduce everything to the quantifiable.

Data is not just about quantification; it's about information.  We are only at the beginning of collecting, structuring, and even analyzing data.  My belief is that we will see great advances in this processing, which will in turn unlock new possibilities for data-driven insights.  Such innovation will enable analyses and insights never before possible.  Data will inform questions we don't even yet know to ask.