In the first post of this series, I mentioned that we are at the advent of a new era of glorious distributed computing of many kinds, and therefore infrastructure has to be reimagined to meet the evolving needs and challenges.  In this article, we will explore how de-centralization via distributing trust across participants using blockchain technology in business transactions is driving some new infrastructure and business model innovations.

The term blockchain burst into prominence when the anonymous creator of Bitcoin cryptocurrency described the underlying distributed ledger which is essentially a public chain of encrypted blocks.  The immutability and public sharing of that blockchain enabled distributed trust in the crypto-currency.

Blockchain refers to a particular kind of data store, defined by the Oxford English Dictionary as “a digital ledger in which transactions … are recorded chronologically and publicly”.  The characteristics and mechanisms of this chain (which we will elaborate upon later in this article) enable the participants to record information and conduct transactions, that cannot be modified nor refuted once made.  Modifications to the chain are only done by adding new strongly encrypted blocks and since the chain is public, there is no centralized ownership.  All participants see and retain the same modifications (such as currency transfers) and hence they cannot be refuted later.

However, in this article, we are not going to discuss the many different blockchain variants that are the basis of the many different cryptocurrencies and their offshoot siblings, tokens, that have bloomed all across the globe.  Instead, we are going to discuss the benefits and challenges of enterprise blockchains that are private, permissioned and have some element of centralization as opposed to the fully transparent, decentralized, public blockchain implementations. Enterprise blockchains are variants of the original crypto-currency blockchain but leverage the same underlying distributed trust and immutability of that original data structure.

Enterprise blockchain has been hailed as a revolutionary new technology, and as a once-in-a-generation opportunity by various pundits of the industry.   Gartner Group has estimated that the business value add of enterprise blockchain will be more than $176 billion by 2025 and it will exceed $3.1 trillion by 2030.  Even if the eventual result is a fraction of that and it takes a bit longer, we can see that the potential for enterprise value is just enormous and that explains the excitement and initiatives that are underway in various enterprise segments. Let us now explore some examples of the kind of value blockchains can add to enterprises.

Examples of enterprise blockchain value

Attribution: some content in this section is abstracted from an article on ETCIO.com published here.

There are many ‘Moments of Value’ that blockchain can deliver enterprises in today’s business context as companies increasingly engage with an ecosystem of external organizations to create and deliver value to their customers. While these ecosystems are often globally distributed, information flow through them is heavily centralized through systems and processes tightly controlled by these companies, often for reasons of securityprivacy, etc. However, they face numerous challenges such as IP theft, contract disputes, reconciliation data across multiple systems, etc. Billions of dollars are wastefully spent in increased costs, lost revenues and regulatory compliance.

While current systems do not solve these problems, blockchain technology has emerged as an effective antidote to address several business challenges such as secure data exchange, single source of truth, optimizing processes, mitigating contract disputes, and governance/compliance.

These characteristics of blockchain technology finds immediate and relevant application in use cases across multiple industries: Know Your Customer (KYC) and Anti Money Laundering (AML) compliance in financial services; issuing insurance policies as smart contracts to fight claims fraud; finding counterfeit drugs in pharmaceuticals; identifying and tracking source of defective products in a supply chain; accurately maintaining legal or real estate records; minimizing contract disputes in complex multi-vendor software projects etc. The list of applications spawned by blockchains is literally endless, amplifying the significant potential that this technology extends to businesses.

One of the biggest benefits of blockchain technology for enterprise use cases is the removal of the stacked cost and delays of intermediaries.  Because the participants can trust the data in the shared blockchain and hence can directly trust each other’s actions there is no need for an intermediate certifier or clearing house that is needed in today’s application and even business models.

An example of disintermediation is the real-estate industry. There are numerous parties involved in a real estate transaction such as a sale of a home: the buyer, the seller, the broker, the mortgage lender, the appraiser, the inspector, etc.  Since they all cannot directly rely upon each other for all the data and payments, a central party such as the title company is employed to keep track of the actions of each party and the amounts are held in escrow by that company.  This process can be both accelerated and made much more efficient if the parties involved can make use of a shared blockchain where the trust is implicitly distributed across the participants and the reliance on a centralized agency can be drastically reduced if not eliminated.

A similar issue affects international money transfers.  A central clearing house holds and assures the transfer which involves multiple steps. Each step takes a long time and adds costs becoming quite expensive overall.  As a result, an international money transfer can take several days and cost tens of dollars.  But blockchain based alternatives are emerging which can make the transfer almost instantaneous and cut the costs to a small fraction of current charges.

One other much cited example of enterprise use cases are tracking provenance in supply chain and retail. There are many scenarios under emerging regulations as well as branding single source products such as coffee/tea and other such products, whereas, the eventual buyer wants to be assured of the authenticity of origin and blockchain has emerged as the ideal solution.

We can keep on describing many more such examples of the enterprise value that can be accrued due to the distributed trust business models engendered by the adoption of blockchain, in various enterprise segments such as healthcare, finance, retail, manufacturing, government, and more general use cases such as digital rights. Such use cases are described further in the detailed enterprise blockchain report fond hereBut let’s stop here with the examples and go on to explore the current status of blockchain initiatives and the challenges.

Current status of Enterprise Blockchain

A lot of enterprises are excited by the potential of blockchain technology and are experimenting with it.
There are many, many prototypes, proofs-of-concept, and trials that have been implemented. However, there seem to be remarkably few full-scale complex production applications!  There are some international governments such as Dubai, Singapore and a few others who have rolled out simple some e-government apps using blockchain technology.  But as for enterprises we tend to read about a few examples of limited rollouts, and even some rollbacks.   Gartner group with its hype-cycle notes that widespread production deployments are perhaps about a year to two years away, and large scale deployments may take a year to two years after that.  There will, of course, be many currently piloted projects graduating to production gradually over that period.

Why is there such slow progress in production deployments? Yes, there is a lot of promise in the technology, but it must be noted that there are also a lot of challenges like any new pathbreaking technology.  There are issues with interoperability, data integration, performance, security, and ease of application development.  We will now elaborate on each of them a bit more.  I will just touch upon each of them and leave more details elaboration to others who are already toiling hard at solving these issues.  Some references to such efforts are provided at the end of this article.

Interoperability

One of the issues that are limiting speedy production deployment of enterprise blockchain is that there are many variants of them, some of them quite radically different from others.  Ethereum, Hyperledger, and R3’s Corda are two notable blockchain technologies but there are several more, in fact too numerous to name!

Such availability of different blockchain technologies has led to the fragmentation of initiatives, which cannot interoperate with each other.  For instance, one major enterprise uses Ethereum as the basis for its supply chain initiative, and another major enterprise uses Hyperledger as the basis, and perhaps both use Ripple for cross-border transactions. In such a scenario, suppliers to both enterprises now have to support interoperability between all 3 protocols with enormous difficulty and their own applications that support that two initiatives cannot interoperate internally!  Not to mention, it becomes hard to impossible for those two major enterprises to interoperate to share data or conduct transactions (for instance, think what happens if a major pharmacy chain uses Ethereum whereas a major health insurer uses Hyperledger!)

The consortium formed around Hyperledger Fabric has made a good start to bring some standardization around Hyperledger itself, but a fair number of other enterprise initiatives are utilizing Ethereum or other such blockchains so fragmentation still is the order of the day.

The problem, of course, is not limited to interoperability just amongst various blockchains!  Think about the vast ocean of enterprise legacy apps that already exist.  There are already several offerings that attempt with varying degrees of success, to integrate across an enterprise’s own internal custom apps and the SaaS offerings that they are increasingly adopting.  Now throw multiple different blockchains in the mix and we have the makings of major chaos!    To avoid that, blockchain initiatives also need to work to smoothly plug into and interoperate with major enterprise application integration initiatives.

Such interoperability is now ad-hoc and lacking in cohesion and standards.  This is one of the reasons for the slow progress.  Perhaps the availability of a framework that can make it easy to operate across blockchains and also let blockchain apps interoperate easily with legacy apps and SaaS will make the going easier and faster.

Performance

The original blockchain tech is very slow, and in the case of bitcoin, intentionally so to apply brakes on the number of new coins as the total approaches the pre-established limit.

However, enterprise blockchains are also slow, compared to relational and no-SQL transaction rates. Each transaction can take seconds invariants of the base blockchain tech whereas applications are used to hundreds or even thousands of transactions per second rates!  Hyperledger has improved performance quite a bit over the base blockchain level and can be used as the basis of practical enterprise apps.   However, there is room for major improvements, particularly when interoperability across chains and also integration with legacy data is required.

Some acceleration capabilities need to be brought forth, both intrinsic to the data format and processing steps (such as the one provided by Hyperledger to improve upon the base version), and also via transparent acceleration of the kind that has been put in place for stronger larger bit-width SSL key exchanges.

Data Integration

Enterprise Data analysis today benefits from a variety of tools that can process data stored in diff data stores.   Relational data systems as well as no-SQL sources such as many different key-value stores.  In this context, blockchain can be considered to be a new type of data store, one that offers immutability and is naturally distributed.  As we have seen, certain applications of distributed trust require this new data store.

Data stored in block-chains will need to be collated with data stored in the earlier data stores for enterprise data mining, analytics, machine learning, and AI-driven processes. However, there are no easy tools or frameworks to extract data from blockchains and make it available for various systems such as data marts, data lakes, etc., which form the rock bed on which further data processing such as analytics and machine learning take place.

Security

Blockchain applications need to be also integrated with enterprise Identity and Access control systems, and audit trails at adequate levels (similar to legacy applications already in use) need to be generated for regulatory compliance.  Configurations of blockchain systems need to be auditable for security assurance.

Furthermore, every data store also has its own unique vulnerabilities to data theft – blockchain is no exception. Also, enterprises need to protect against malware infiltrating automatic smart contracts.  Because these are automatically triggered when pre-set conditions are met, enterprises need safeguards in place that constrain the data and applications the smart contract procedures can access.  Also, the execution environments for such procedures may need to be isolated from existing data center installations.  There are proposals of “smart enclaves” and other such mechanisms to address this requirement.

Also, how do we ensure the privacy of some data such as health records while preserving the public access and immutability of the blocks themselves?  Enterprise blockchains are bringing up such new challenges that need to be solved.

Ease of Development

Building blockchain applications requires a legion of extremely skilled experts in the relevant APIs and technologies.  This can be likened to the early days of programming when it was an arcane art which eventually gave way to click and assemble capabilities of today.

We need to provide higher-level abstractions, especially when it comes to providing commonly applied business services codified into APIs and micro-services.

There are some “blockchain as service” capabilities available on public clouds.  However, they are best thought of as starter kits, providing the basic APIs that can still only be exploited by expert programmers.  What is needed is the availability of higher-level abstractions and common business services, much as “foundation class” libraries eased the development of object-oriented applications a couple of decades back!

Wrap-up

We can see that the distributed trust model of enterprise blockchain shows enormous promise, much like AI promises to revolutionize automation of business processes.  However, we are in very early days and there are quite a few infrastructure challenges across data, integration, performance, and security.  Furthermore, application development needs to be simplified with tools and higher-level business services.

Thankfully, some efforts are already underway to address these issues.  Hyperledger fabric has kicked off some aspects but needs to be complemented with additional efforts, especially as cross-chain initiatives ramp up. Copperwire has posted a blog published here, which discusses some of these efforts.  We are looking forward to more such great solutions!

In the first post of this series, I mentioned that we are at the advent of a new era of glorious distributed computing of many kinds, and therefore infrastructure has to be reimagined to meet the evolving needs and challenges. In this article, we will explore how computing infrastructure has become over-centralized with SaaS and IaaS cloud services, and a new need and opportunity is emerging to broadly distribute computing to solve some needs and cater to new opportunities.

In the early days of computing, except for a few government institutions, no enterprise had its own data center. Educational/research institutions and businesses used computers hosted by large service providers such as IBM. IBM founder Thomas J. Watson is supposed to have remarked that the entire worldwide market for computers was limited to just 5 (yes you read that right, just five!), large installations! We all know how that turned out! It is a different matter that today’s IaaS clouds are forming into about five large installations – it is not the same thing — each cloud has millions of servers far more powerful than supercomputers of that era!

As software solutions started to emerge to provide specialized application capabilities, and computers started to come down in price due to advances in electronics, every enterprise started to be able to afford data centers of their own. The advent of mini-computers and personal computers/servers then proliferated the widespread deployment of computers and application installations everywhere.

Slowly the cost capital expense (Capex) started to be outstripped by operational expenses (Opex).

And also, the cost of professional services to customize applications software far outstripped the license costs to the tune of 5 times license expense for services! That’s when SaaS burst into the scene, exploiting the internet and web, to offer quick and easy, pay-as-you-go consumption of application capabilities with simple click-and-done customizations.

Although before the dot.com bubble burst some services offered by ASP’s (application service providers), the services were still single instance, heavily customized software with just the license and data-center being owned by the provider, largely following the same old heavy-weight installation model. It was as if the total cost of ownership was just being amortized over some longer period rather than being a real multi-tenant pay-as-you-go easy-to-use SaaS.

I think Salesforce.com can be said to be the true pioneer of SaaS as we know it today. “Software is dead” was their motto, meaning don’t buy, rent; don’t install, use. Kudos to them! SaaS has become so mainstream that these days everyone looks askance if an enterprise buys and installs the software. This trend centralized application services, but not all applications, of course.

Enterprises still have lots of custom applications, particularly when the use of the software is intrinsic (such as eBiz) or crucial to business advantage (e.g., large financial services firms).

About a decade or so ago, Amazon.com brought forth a new revolution: Infrastructure as a service (IaaS) in the form of Amazon web services (AWS). In part 2 of this series, we explored the dramatic impact it had on application development & deployment and consequently speed of innovation.

As enterprises, particularly e-Businesses, saw the advantages IaaS provided, they started to adopt IaaS with increasing comfort as AWS began to provide more and better services. This led to the centralization of deployment of even custom applications. Interestingly, some enterprises have even started to dismantle data centers entirely and have even gone to the extent of re-deploying their packaged vendor software on IaaS clouds. It has been an ongoing transition which has been accelerating recently.

However, application delivery owners were confronted with three major challenges: end-user experience, data gravity, and data sovereignty.

These challenges are driving SaaS providers, e-Businesses and enterprises to start to reconsider whether the centralized deployment of applications in their data centers (or co-location centers) or on IaaS public clouds should be complemented by using distributed edge locations. The advent of 5G telecom services is adding weight such considerations. Let us now elaborate on each of these issues.

End-user experience

Many SaaS providers use a few co-located data centers, since SaaS pre-dated IaaS, and only lately adopted IaaS to complement or replace (perhaps in distant future) their data centers.

Even though public clouds have global deployments, they are built for scale and hence tend to have concentrated deployments in major locations. One major issue that has been observed is that the resulting end-user experience for such centralized apps is poor, especially in remote branches or for mobile/home-office workers.

Although many solutions have been tried, end-user experience from cloud locations (albeit spread around the globe) and from SaaS/E-Biz data centers has been uneven at best and unacceptable at worst, sometimes even triggering service-level penalties or outright cancelations (also known in SaaS world by the dreaded term, churn!)

Not only that, it has been proven beyond doubt that end-user response times directly result in increase or decrease of revenues for businesses that rely upon revenue generation from online activities in whole or in most parts (such as e-commerce, social networks, etc.) Lost shopping carts or lost advertising revenue haunts the nightmares of application delivery executives of such businesses.

While there are numerous factors that can improve or degrade end-user response times, two factors dominate the time it takes a request or a response to traversing the network. One is quite simply the speed of light. As some smart-aleck once quipped, 300,000 km per second is not just a good idea; it is the law, the ultimate universal speed limit!

Because of this limit, the farther a user is from the application location, the slower the response time, and conversely the closer an application are to the end-user, the faster the interaction can be.

The upcoming deployment of 5G mobile tech is supposed to reduce the latency between a connected device and the first possible packet-processing unit to just two milliseconds.

This is revolutionary! Already applications are being devised to exploit this ultra-low latency, such as multi-player gaming and Augmented/Virtual Reality (AR/VR) apps. But to exploit this low latency, applications need to be close to the location that the 5G “last-mile” connection enters into the telecom provider’s network.

Otherwise, the vagaries of the “mid-mile” links will overwhelm the advantages of brought by 5G.

The second factor is network connectivity and path lengths. This is how a request or response is routed through the network, the number of router hops and the path length between each hop.

 

 

Due to vagaries of service provider interconnections (aka peering), routing protocols and preferences, sometimes network packets can end up taking quite roundabout paths, with more hops than strictly needed and/or much greater path lengths. This can result in momentary or lasting poor application performance. This issue also can potentially be addressed by network optimization services and application acceleration services deployed closer to the end-user, in various edge locations.

Such locations could include 5G service delivery clusters provided by telecom service providers, which would drastically cut latencies, making it possible to deliver blindingly fast, high bandwidth services such as multi-player gaming, AR/VR experiences, etc.

A variant of end-user experience is end “thing” experience. This occurs in the internet of things (IoT) installations where a decision needs to be made nearly instantaneously (such as when a monitored aluminum smelter is over-heating) or when disconnected from the cloud (such as at a really remote oil rig). In both such cases, it would be imperative for the application component (micro-service?!) to be located as close as possible to the “things” if possible, in the same local monitoring network.

Data Gravity

We all know about gravity! The more massive an object is, the more gravity weighs things down, and it takes a lot more effort to lift them.

Data gravity” is, however, not a term in common parlance, and probably needs a bit of explanation and context. It can be defined as somewhat simplistically as “the amount of data that is needed by a particular application task at any given time”. Going by this definition, if an application task requires just one byte of data, it has very low data gravity. If, however, an application task has to churn through a huge amount of data, it can be said to have high data gravity.

 

As opposed to real gravity which gets weaker as distance increases (per Newton’s universal law of gravitation), the strange and annoying thing about data gravity is that its effect gets worse as the “time distance” between the application and location of the data increases! Effects of data-gravity can also be exacerbated by the amount of bandwidth to transfer data between the storage location and the compute location (including high-speed data-center links, high-speed system hardware backplanes, and even the links between processors and hardware main memory). Modern data-crunching analytics and machine learning apps have very high data gravity.

The natural conclusion one can reach is that if an application task’s data gravity is high the closer it is executed to the data it processes, and the larger the bandwidth of data-access, the better it will perform. There are several situations where executing an application task on the cloud makes it suffer from data gravity.

One such scenario is when a large amount of data is pre-existing which needs to be processed along with new data (for instance for machine learning), and it is not practical or secure to copy the data to the cloud. (Some financial services companies refuse to upload sensitive data to the cloud and hence are firmly wedded to data-center only or at best, a hybrid-cloud deployment model for this reason).

Another scenario is when a huge volume of data is continuously produced (such as in many internet of things installations with thousands of devices being instrumented and a large volume of data collected). In such cases, many times, it does not make sense to transport all data to the cloud, but only transmit it when it makes sense. It may make sense to transmit only when some monitored item changes.

An example is when someone is monitoring temperature: it may be collected every second, but if it has not changed, why transmit it?! To accomplish such change detection or some other aggregation functions, we will need to locate processing close to where data is generated. As a discerning reader, you may have noticed that this is similar to the IoT case we discussed for end “thing” experience. Kudos to you!

Therefore, we can surmise that high data gravity “attracts” application tasks to be placed as close as possible to the data.

Data Sovereignty

Recently you may have heard the term “data sovereignty” a lot. For those of you who may not be cognizant with what it means, I would like to explain it briefly as the context for how it matters for distributed compute infrastructure. Sovereignty in this context means that data needs to stay in the location where it is generated. In other words, the location of data generation has sovereignty over that piece of data, just like a nation has sovereignty over its citizens who reside there.

Several countries (e.g., China), groups of countries (e.g., European Union) and even smaller domains (e.g., the state of California in the USA) have enacted data sovereignty laws.

India recently started enforcing strict data sovereignty laws, which forced international companies to scramble to implement stop-gap solutions to be compliant and avoid hefty fines or even possible business downtime. EU’s general data protection regulations (GDPR) has caused enterprises to think and work ever harder to devise ways to remain compliant. Many SaaS companies also who have customers in domains with such sovereignty laws have to devise ways to comply.

All of these leads to one plausible answer: to paraphrase the well-known aphorism, if the (data) mountain cannot go (be moved) to the miner (app), then the miner has to go to the mountain (much like the data gravity situation). That is, if the data which legally cannot be exported, move the application that processes data to the location of the data.

Moving an entire application is easier said than done of course, but some part of the application that can process the data and extract and return relevant compliant results (such as AI models) that are needed elsewhere, or take any other action on the data locally (including actions such as “forget me” required by data privacy laws).

Wrap-up

As we can see from the description above, there are compelling reasons not to limit applications to be deployed only at central cloud locations or data centers, but to distribute them (at least parts of them) as close as possible to users, things and data. This calls for a distributed edge compute infrastructure for applications and/or acceleration services.

In the next article, we will explore enterprise blockchains, a different, application-level trend that is distributing trust across the participants.

Kmesh, The Fabric co-created company, supports the transition from centralized data lakes to distributed data ponds and provides software that distributes your data, across onprem, clouds, countries, and edges, as a single global namespace.

Kmesh‘s CEO Jeff Kim on using cloud data management tools to avoid single cloud vendor lock-in

5 mistakes to avoid:

1. Developing a fragmented cloud management strategy

2. Not using common data management tools across multiple clouds

3. Failing to analyze and understand application performance requirements

4. Deploying inadequate security

5. Falling victim to vendor lock-in

Read the whole article here