Cloud Storm Chasers : Service Measurement Index

CSMIC Initiative Update - A Mission to Build Cloud Service Measurement Metrics

Jeffrey Abbott — Wed, 13 Apr 2011 08:00:00 GMT

You may have seen recent news here on the blog about the expanding group of members in the Cloud Service Measurement Initiative Consortium (CSMIC). For those of you less familiar with the organization, its objective is to develop a standardized method for measuring and comparing IT business services based on metrics that matter to the consumers of those services. There are some tools (such as http://www.cloudsleuth.net/) in the market today that can show real-time service performance monitoring of uptime and latency. But research from CSMIC shows that people demand a more comprehensive analysis to understand what service is best for them (or at least better/worse than what they have). The research revealed six top-level metrics that should be considered: security, capability, quality, risk, cost, and agility (and those can be broken down even further).

This is where things start to get really interesting. As a result of this research, CSMIC has developed a formula to compile qualitative and quantitative user-submitted experience data to produce ratings and comparisons for various types of services (such as database or storage on demand). This is called the Service Measurement Index (SMI), which will be accessible to members of the Cloud Commons community (http://www.cloudcommons.com/) soon. For example, a user looking for the best email solution can set the weighting of the business priorities across the six metrics: quality, agility, risk, cost, capability, and security. The service ratings will be calculated and presented accordingly with scores from 0-99 - with different outcomes depending on the amount of "importance" the user places on each metric. From those scores, the user can then dig deeper into reviews or select specific services to perform comparisons.

I recently had the opportunity to get an update from Peter Stapleton, a colleague in the Cloud Computing group here at CA Technologies who is a consortium member and who also attended the recent group meeting. Here is a summary of our conversation:

Who are some of the CSMIC members and what are their interests in the initiative?

In addition to Carnegie Mellon University and CA Technologies, the new members are Accenture, Cask LLC, City University London, Data Security Council of India (DSCI), International Association of Outsourcing Professionals (IAOP), Mycroft, State of Colorado's Office of CIO, Stony Brook University (NY), TM Forum, and the University of Melbourne in Australia. Each organization has different reasons for participating, which is critical to SMI's success as a standard. The reasons range from interests in research, to thought leadership, to influencing standards for IT, to direct use of the tool to help IT make better decisions.

How will people access the Service Measurement Index?

All CSMIC members have agreed to use Cloud Commons as the consortium working space, which is where Cloud Commons members can access SMI. You can see some information there now, but the SMI tool itself will be released mid-2011.

What is the current status of the index?

The SMI characteristics (the long list of metrics by which business services will be measured) and the measurement processes are being rationalized now within the consortium. The development of the public user interface is also on schedule for mid-2011.

Will SMI become an official standard - acknowledged by any industry standards bodies or government entities?

That is also in progress. The CIO of the State of Colorado will sponsor SMI with NASCIO (National association of CIOs), which sets standards for state governments. Also CMU is placing SMI and the consortium on Cloud-Standards.org (this location is referenced by the US Federal Government via NIST (National Institute of Standards and Technology) as a meaningful location for cloud standards. However, there is still more work to be done before SMI becomes an official standard.

How do I submit ratings for services that I've used?

There is a link that says "Rate a service now" on http://www.cloudcommons.com/. Here you can take a short (5-minute survey about your experience with various types of services and providers. Anyone with experience using IT services is encouraged to participate. This data helps contribute to the quality of the ratings and comparisons that SMI can generate.

About Peter Stapleton

Peter Stapleton, a senior product manager for the Cloud Computing business at CA Technologies, brings to his role broad experience in security, governance, and risk management. His focus is on the Service Measurement Index, a CA Technologies developed framework for service comparison, which is now managed by a consortium of industry and academic organizations. He has 15 years in the enterprise software industry, of which 7 are at CA Technologies, primarily in product management. Past roles have also included technical consulting, operations management, and product marketing.

Peter represents CA Technologies on the Cloud Service Measurement Initiative Consortium (CSMIC) and leads the TM Forum Cloud Service Definition and Taxonomy projects.

Change is in the air and forecast calls for more clouds

Jeffrey Abbott — Wed, 16 Mar 2011 20:00:00 GMT

Maybe I'm biased, but it really seems like this cloud thing is real. I'm a skeptic about almost everything but it seems that nearly every software vendor, service provider, and IT department has decided that there is a better model for how IT services are created, delivered, consumed, and paid for. We've agreed to call it cloud. What makes it newsworthy now is that the technology finally exists to execute on it.

But with so much change and so many options to consider, the daunting task of matching one's IT needs to an optimal solution seems impossible. Sure, it's relatively easy to find real-time performance data about a particular instance of a particular public cloud service. But a single performance data point is a small piece of the puzzle when you're managing an IT budget, security requirements, SLAs, and capability requirements in a dynamic IT environment. So how can you identify the best service for your needs?

Fortunately, some smart people saw this coming and we now have the Cloud Service Measurement Initiative Consortium (CSMIC). Unveiled in May 2010 (CA Technologies is a founding member; Jay Fry wrote about it here), a primary objective is to create and promote the adoption of a standardized method for comparing cloud and other IT services. The product of these efforts is the Service Measurement Index (SMI), which is an impressive set of business-relevant KPIs that provide qualitative and quantitative insight into the comparative user experiences of particular service options - internal and external. In other words, it helps IT make the right decisions about their service offerings.

This is a win-win-win for end users, IT directors, and service providers. I'm glad to see that momentum is building around this effort. Carnegie Mellon University, which is driving the initiative, just announced 10 new members spanning research, academia, government and industry leaders - including Accenture, Cask LLC, City University London, Data Security Council of India (DSCI), International Association of Outsourcing Professionals (IAOP), Mycroft, State of Colorado's Office of CIO, Stony Brook University, NY, TM Forum, and the University of Melbourne in Australia. Here is the announcement for additional info. The consortium's efforts will be hosted on Cloud Commons - please feel free to register and share your thoughts in this user-focused community focused on cloud topics and trends.

Comparing the Relative Quality of Cloud Services – Is a single metric enough?

Jeffrey Abbott — Wed, 18 Aug 2010 20:59:00 GMT

I've been seeing some buzz lately around the need to identify a method for rating and comparing the performance of IT services - especially since the understanding and availability of cloud services has risen significantly. Specifically, one recent ReadWriteCloud blog post stuck out to me where several resources were named but the definition of performance was based on only one or two parameters, such as cost, response time, or availability.

I'll use the auto industry as an example. Performance is a term that is loosely thrown with respect to cars, but as we know, it involves a lot of quantitative and qualitative information for someone to determine if a Lincoln is really better than a Cadillac. Reviewing one or two variables is simply not enough.

Hence, the need (back to IT for a minute) for the Service Measurement Index (SMI). This provides information about the quality of IT services (including cloud) - relative to other services. After all, the decision is often not whether or not to buy, but instead, "which one." Instead of being completely subjective or completely objective, SMI enables business services to be expressed and compared in business terms. It analyzes IT services across six metrics, including: quality, agility, risk, cost, capability, and security.

Carnegie Mellon University (CMU) and CA Technologies teamed up to start the initiative, and CMU is leading the charge going forward with a growing consortium to support it. The result is that the industry now has a standard way to perform IT service comparisons and make more informed purchasing decisions. The results are compiled from CMU's initial and ongoing analysis, plus end-user-submitted surveys about their own experiences with existing IT services. The data is processed to generate a relative score, compared to other services of the same category based on the six key business metrics.

This provides a similar value to a component from one of my favorite magazines: Car and Driver. As a car fanatic who seldom gets the chance to test drive cars myself, I've always enjoyed the index in the back that compares many noteworthy cars across common metrics such as: price, power, acceleration, handling, top speed, lateral acceleration, braking, etc. This lets me quickly and easily single out the best (and worst) performers in their classes.

So that's all well and good. The only missing component now is that more subjective content that lets me dig in and learn more about particular cars, customer experiences, and those little details that make some people buy a Corvette, and other people a Porsche 911 - even though they match up pretty closely on the track (yes, they do).

Enter Cloud Commons. Consider this to be the rest of the magazine, and much more. In addition to being where you find SMI, Cloud Commons is an independent community of IT professionals, analysts, technology providers, and industry experts (CA Technologies is also a sponsor of Cloud Commons). It is a place to find, and contribute, user experiences, best practices, cloud-related news, and discussions. Cloud Commons, with the Service Measurement Index, is a living, breathing community and resource which is growing as more and more people weigh in along their journey toward cloud. And while it's still in its early stages, it's available today. SMI has over 30 organizations expressing interest in joining the consortium and membership for Cloud Commons has been growing steadily since its launch in May of 2010.

*Image used under Creative Commons License, courtesy of Luciano Meirelles.

Thinking about IT as a supply chain creates new management challenges

Jay Fry — Fri, 28 May 2010 04:12:00 GMT

CA World has wrapped up, finally giving me time to post a few comments about the major product news of the week from the CA Cloud Products & Solutions Business Line: the announcement of the CA Cloud-Connected Management Suite. (You can catch up on the other supporting announcements, the launch of Cloud Commons and the formation of a consortium to create and drive the Service Measurement Index, in my previous post.)

I heard a lot of very positive feedback about CA Technologies’ plans, many from customers, partners, and industry watchers & experts that we’ve been talking to quite a lot for advice and feedback.

However, I think it’s always a good idea to start from the beginning, especially for those who didn’t make it to Vegas to hear the CA execs (or comedian Jake Johannsen, Maroon 5, or James Cameron, for that matter) in person. I think the premise behind this forthcoming product suite warrants a bit of color and background, especially since the implications for IT are pretty significant, if all goes as described.

Cloud computing means IT loses its monopoly

The premise is this: cloud computing enables you to run IT differently than before. IT has had a type of monopoly. It has, for the most part, been a monolithic provider of IT service to support an organization.

Enter cloud computing. As cloud computing matures, it breaks IT’s monopoly. Business users now have a choice: they can certainly go to their IT teams for support with new initiatives as they have been doing. But, they can also investigate SaaS or many different types of cloud services directly.

As in, without IT’s involvement.

I’ve talked about business users “going rogue” in a couple previous posts. Business users are investigating cloud services on their own more and more frequently given the increasing ubiquity of cloud capabilities, their pay-as-you-go models, and the speed you can get something up and running.

As Chris O’Malley, CA’s EVP for Cloud Products & Solutions, said in his CA World cloud track opening session last week (video replay here), “IT faces a rather stark choice: either add value to the acquisition of cloud services by business users, or risk becoming increasingly less relevant.”

Ouch.

Instead, IT can be the manager of a supply chain of services

So what things would IT need to be able to do in order to help business users make the best IT sourcing choices, regardless of what the final answer is? They’d need to do less of what they’ve typically done – manually making sure the low-level components are working the way that are supposed to – and become more of a trusted adviser to the business.

The big shift here is how the business is getting its services: those services are now potentially coming from many disparate sources. And the source of a particular IT service can (and should) change over time.

The result? IT needs to manage – and constantly improve – a changing IT service supply chain. Gartner analysts Ben Pring and Allie Young published a paper last May about these "service value chains," saying they "will be at the heart of cloud services." Their research updated Gartner's model for "understanding the ways in which multiple components sourced from multiple suppliers will come together in the new service delivery models loosely defined as 'cloud computing.'"

"Cloud computing isn't your future," said Forrester analyst James Staten in his more recent post, "it's a new part of your overall IT portfolio." IT will use internal capabilities, external ones, or a smart mix of both. And that is something it can’t manage with existing tools.

To be clear, things like security, service assurance, virtualization management, automation, and other similar capabilities that are extended to take into account cloud-connected environments are still critical (CA Technologies provides a portfolio of solutions for this, as do other big & small vendors). However, they don’t help you understand, control, and constantly improve this new IT supply chain.

Delivering on the new IT supply chain requirements

We decided to do something about this. We started by figuring out what pieces would be needed to make this possible. In other words, what are the new requirements if IT is to take on this role?

O’Malley rattled them off in his CA World presentation. We think the list includes these things:

· Gain insight to compare. First, you need insight into your IT services – both internal and external. You need to discover who is using what, and you need a standard way to describe and evaluate those services, based on important business-related metrics – something that could be used for apples-to-apples comparisons.

· Connect to expert knowledge. Second, you need access to relevant data about cloud services, feedback about those services, and best practices from a community of both peers and experts. This helps you decide if, when, and how you should use cloud computing for a particular application or capability. Customers told us that they need and want a place to share experiences with likeminded people on many, many cloud topics to overcome the huge information gap -- a gap that comes from the rapid changes driven by innovation itself.

· Cloud-enable applications. Third, you need an environment where you can make changes and act on this information. You need to enable both existing and new applications and IT infrastructure to be more flexible by abstracting them from current physical dependencies. This will make it easy to move workloads to an internal or external cloud environment and back again, if you decide that is the right thing to do.

· Challenge your choices. You also need to optimize the choices you’ve made while selecting and implementing available options. You need to constantly challenge and reassess those choices to take into account both new information and changing business goals.

· Deploy, manage, & secure those decisions. Finally, you need to be able to deploy, manage and secure services that leverage the choices you’ve made to meet enterprise-grade, industrial-strength requirements.

Once you have these core capabilities, you need to use them over and over again, making sure you are constantly improving over time. Because in this scenario, the right answer is going to change as time goes on.

As you might guess, we took this list as a good starting point for the capabilities of our new CA Cloud-Connected Management Suite. The suite will have 4 components: CA Cloud Insight, CA Cloud Compose, CA Cloud Optimize, and CA Cloud Orchestrate. For a summary of what each of these will be, you can look at this short paper or the press release. But you’ll notice they follow this list pretty closely.

How do you turn ideas for cloud management into reality?

CA Technologies is now in the midst of making the concept I just described real. We’ve been focused on organic development efforts, and have also found a number of innovative companies in the market with some core technologies to help deliver on this.

For those of you who have been following the string of acquisitions that CA has made in the cloud computing space in the past 11 months, you can probably see how this suite weaves together technology from the acquired companies. You’ll recognize core capabilities of Oblicore Guarantee as part of CA Cloud Insight, 3Tera AppLogic at the heart of CA Cloud Compose, and Cassatt in CA Cloud Optimize.

But, these technologies alone don’t fully flesh out the requirements I listed. Nothing in the industry does yet. That’s why, even after the acquisitions, there’s a bunch of development that’s on-going to deliver what customers need.

Understanding the IT supply chain: using Cloud Commons and the Service Measurement Index

To help get a handle on the components and management of this IT supply chain idea, we believe that IT needs not only a new set of product capabilities, but also a place to do research on the many cloud services and capabilities – and a way to measure them. Thinking it through, in order for customers to get what they need out of those functions, it seemed much smarter to have them exist separate from CA.

So, we set up Cloud Commons as an independent community and website. And we collaborated with Carnegie Mellon University to get the Service Measurement Index (SMI) started, and then placed SMI in the stewardship of their consortium, so it wouldn’t be tied to CA alone. These moves hopefully get the ball rolling, helping make both useful to the industry as a whole, regardless of whether someone is a CA Technologies customer or not.

But, put all these pieces together – a community of expert information, a standard way to evaluate cloud services, and this new suite of products – and you have the beginnings of a new set of tools and capabilities to help IT manage this new supply chain. And, in fact, a new, expanded role for IT.

Where are we now? Both potential and pitfalls

CA World was our first stake in the ground. The industry has been wondering how we were piecing all of our acquisitions and other work together. We intended the announcements and various sessions at CA World to at least provide the strategic framework for what we’re doing.

Now comes the hard part: delivery. We’re back in our offices, hard at work on that, only a little sad not to still be in Vegas (a week there is quite enough, though, I have to tell you).

Meanwhile, customers can get started with things like the Oblicore and 3Tera products now, giving them a taste of what’s to come, with a convenient upgrade path to the new suite later.

The first deliverables (CA Cloud Insight & CA Cloud Compose) are due by the final quarter of the calendar year. You can be sure we (and I) will keep you posted on our progress.

Until then, we want, expect, and welcome lots of feedback on every piece of this. It’s probably not what people were expecting from CA, but that’s just fine with me. Please feel free to chime in.

This piece is cross-posted at my Data Center Dialog blog.

A new way to compare your cloud computing options

Jay Fry — Tue, 18 May 2010 14:49:00 GMT

If cloud computing does what everyone is saying it's going to do, organizations are going to end up with many, many ways to get the IT service they need to support their business. And choice is good.

But having thousands of choices with no clear way to decide (or even prioritize) is not a recipe for success. Three of the announcements I'm helping with at CA World in Las Vegas this week (#caworld on Twitter) are aimed at addressing that problem: finding smart ways for companies to make those choices - and ways to constantly challenge those choices. Read on for a bit of detail and commentary on the first two; I'll post another blog after Chris O'Malley's Cloud & SaaS Focus Area Opening to cover the third.

Answering the bigger questions

These announcements are the result of quite a bit of soul searching at CA (now called CA Technologies, a hint that significant change is afoot) about how the company could and should help fill in some key missing pieces for really getting utility out of cloud computing.
What we saw and heard from customer after customer was that there are a number of very important issues around the cloud, issues that everyone's been talking about: security, assuring performance and availability, managing and automating underlying technologies like physical and virtual servers. And, to be sure, we (and others) have answers to a lot of those that are getting better and better all the time (including some announcements from us coming this week).

However, there was a bigger problem not getting answered.

Business users are realizing that the IT department is no longer the single source for delivering the IT service they need. They can go around IT. Or at least use cloud services, SaaS offerings, and the like as pretty strong bargaining chips in the negotiations with IT.
And IT has had a pretty difficult time putting what they can deliver side by side with what can be sourced from the cloud and doing a fact-based comparison so the business can make the right decision.

We also realized that there are things that CA Technologies could and should deliver to help answer some of these questions, and things that are better driven by others with relevant expertise and experience. So we are doing a mix of both.

First, a standard way to measure IT service in business terms

Sunday afternoon, Jeff Perdue, a senior researcher at Carnegie Mellon University, and CA Technologies' David Hodgson announced an effort to provide the industry a standard way to do these kinds of IT service comparisons. The result is the beginnings of the Service Measurement Index and a consortium to support it.

The Service Measurement Index (SMI) is the first of its kind - a relative index focused on describing IT services in business terms. Instead of looking just at cost, SMI provides a way to balance between six important characteristics: quality, agility, risk, cost, capability, and security.

CMU and CA Technologies got this started, but CMU will take the lead from here on out, pulling in educational institutions, global end user organizations (both commercial and public sector), and technology vendors. There's no way this is something a single vendor could pull off, and CMU has the chops to help make this consortium -- and its work -- real.

And, even if you're not a big user of cloud computing services now, SMI can be used to rate your existing internal services. Given that ways to do apples-to-apples comparisons are hard to come by, that should be good news.

There is a panel this week at CA World with lots more information about SMI and the consortium; I'll forward (and tweet) more details as the week progresses.

Second, a community and website for qualitative & quantitative feedback on cloud services

Hodgson also announced Cloud Commons on Sunday (I need to have a word with whoever's doing this wacky announcement scheduling). Cloud Commons (http://www.cloudcommons.com/) is an independent community that CA Technologies is launching and supporting, again with the help of a lot of partners, that is now open and available for use. It's intended as a place for cloud fans and skeptics alike. It's for end users, technology providers, industry experts, and others to share experiences, best practices, and qualitative & quantitative information about the many types of cloud services available.

Cloud Commons will use the Service Measurement Index as one way for people to describe their experiences with cloud services and compare them to others. There will also be ways to add commentary and to interact with others in the community. Think of it as a Consumer Reports for cloud services, except you and your peers are the experts providing input.

Since this is going to be a living, breathing community, much of the content that will make Cloud Commons useful will come directly from the specific input of many, many IT folks. Knowing that we have to start somewhere, we got the ball rolling with a broad range of current and historical data gathered for major cloud sites and starting-point SMI data from an extensive research project with a leading analyst firm on the characteristics of particular cloud services (including e-mail, e-commerce, and a few other common examples).

But frankly, what you'll see up there today are the bare bones. I'm expecting that the most interesting content started trickling in tonight when customers on the CA World show floor started registering and filling in survey data about what they are doing with cloud services - and their opinions on those services, good or bad.

In the press conference today, Hodgson was asked how Cloud Commons could possibly provide all the right information. "This isn't CA doing this, this is the community," said Hodgson. The site isn't going to be perfect right out of the gate, but it will morph and grow with feedback, he believed. And that's actually a strength. "Some of the answers will come from the community - what do people want to see? Just the fact that it's going to exist as a concept will start the conversation that the industry needs."

Next up: new CA Technologies cloud management solutions

The third cloud-related announcement is about a set of products that we have up our sleeves that Chris O'Malley will announce from stage on Monday. I'll post on that after it happens.

Until then, take a look at Cloud Commons, dig into SMI, and give feedback both on cloud services and on the concepts I've been talking about here in general. That's what will make all of this useful, after all.

Also of note: if you are interested in being a regular, more official contributor to Cloud Commons or would like to make sure your cloud services appear in the Cloud Commons marketplace, drop me a line. And to reiterate, we're very serious about everyone being welcome: partners, customers, competitors, industry analysts, pundits of all types. Definitely take us up on the offer.

This piece is cross-posted at my Data Center Dialog Blog.