crystaliz.com

Scale, Speed, Flexibility, and Accuracy — the way to organize analytics

February 7, 2009 on 8:42 pm | In analysis management |

Scale, Speed, Flexibility, and Accuracy — some thoughts on the way to organize analytics groups

Introduction

If you are the manager of an analysis group with a number of analysts who work on periodic and ad-hoc analysis tasks ranging from data compilation to statistical analysis to data mining, your organization is probably using you as a service organization. While you worry and are probably managed on the output of the group, with more and more companies are managing you on other metrics as well. What are the dimensions along which those implicit and explicit measurements lie? How can you be a star by addressing un-articulated needs of your business. If you want to know more, read on…

While software development organizations have used techniques that were developed over many years of research and development in the field of software engineering to define metrics, processes, methods, and cultures to address business goals, analysis groups don’t have the same thorough background to rely on. So, you are probably wondering about ways to organize your organization such that you can meet and exceed your organization’s expectations of your group. In this note, I try to provide my view of an organizing framework and mention how choices in people, processes, systems and tools you make may help you accomplish or not accomplish your objectives.

An organizing framework

At steady state, the type of analysis intake can be classified into two broad categories:

1. Periodic analysis: Analysis performed on a periodic basis whose outcome is more or less well specified and is repeated from time to time to either update your models to improve the accuracy of your results or to produce results that reflect more current data.
2. Adhoc analysis: This type of work can range from firefights to answer questions or work that was not planned but has to be performed to deal with the companies requirements.

As your organization becomes more of a data and analysis centric organization, your group has to scale to support more and more analysis work of both types, be flexible to be able to manage the mix of periodic and ad-hoc work, provide results with utmost speed, and publish accurate results that could be used in down-stream business processes or automated systems.

How do we achieve an effective analysis organization along these dimensions? To answer the question, first let us expound on why these objectives are important to consider.

• Scale: Scale in an analysis organization implies that for every incremental add in resources, the analysis group can take incremental work that is more than a 1-to-1 ratio. If you add one resource and you can only add one corresponding amount of analysis work, then the incremental resource is not leveraged and at some point you will not be able to take on as much of the analysis work that your organization wants your group to perform. Another perspective on scaling is the complexity of the analysis project itself and how the complexity scales in relationship to the incremental resource being added.
• Flexibility: Flexibility in analysis implies not only flexibility in taking on periodic and ad-hoc work, but also the type of work ranging from creating analysis ready data sets for other organizations to complex predictive models without much disruption for the analysts as well as the customers and partners of the analysis group. Flexibility to a significant degree is affected by the tools and systems used, collaboration process between customers of the analysis results and the team work on the analysis, and finally the talent of the analysis and customer team members.
• Speed: Speed implies that the analysis work is done with utmost efficiency given the goals of the project. Analysis speed is often significantly affected by the availability of analysis ready data. In analysis projects where new analysis algorithms are not being invented assembly of data, correlating multiple fields, extracting information from unstructured sources, etc., etc. take the longest amount of time. If this work is repeated on a project basis, not only does the work not happen in time, but scaling is also affected.
• Accuracy: Finally, managing the accuracy of the output of the analysis tasks and projects is a crucial issue for analysis group and is affected the currency and validity of the data used, analysis techniques, tools, and expertise resident in the group, and tests performed with the results that test the accuracy in a simulated or test environment before live deployment. All other things being the same (tools, systems, processes, etc.) the talent of the analysis team and customers who receive the analysis results makes a huge difference in delivering accurate results that address (some times not well specified) business requirements.

Taking stock of where you are

Before embarking on improving scalability, flexibility, speed, and accuracy let us consider what your current organization looks like. In my experience, I have seen the following types of analysis expertise contained in analysis service groups:

• Database and domain specialists: These groups are often staffed by domain experts who also have the access and ability to create reports from databases for various business purposes.
• Statistical analysis specialists:  The analytical staff in these groups are mostly trained in statistical techniques (with potential a smattering of data mining specialists) and work with database experts who create data for them to analyze. The database experts may work as part of the analysis groups or may work outside the group.
• Algorithm driven specialists: The staff in these groups is a mix of computer scientists, database experts, and statistically trained experts (in some cases the same individuals having multi-disciplinary background). Their work often consists of not just developing models and delivering results, but delivering on-line capabilities that can affect business outcomes (including on-line products)

The management of the intake of work into these groups could be well organized with customers expected to specify their needs, filing out forms, etc. or could be more ad-hoc driven by personal meeting between key decision makers. Management of the projects may also be done using classical water-fall methodology of project plans, periodic reviews, etc. or more informally based on weekly/bi-weekly meetings. Very few analysis groups today use practices learned in agile software development methodologies.

One of the bed-rock differentiating factors in these groups is the choice of systems and tools that they use to perform their tasks and their orientation to developing news ones to help achieve their business goals. Typically, the groups made up of domain specialists or statistical analysis experts are bound by the tools and systems they use whereas the algorithm driven specialists are willing to create the tools and systems they need. Furthermore, another key differentiating factor is the use of large, shared systems and tools (such as SAS running on a shared multi-processor computers or large data warehouse machines) for their day-to-day tasks vs. using a network set of distributed computers to perform their tasks.

The choice of database management systems, BI tools, Statistical analysis systems and tools also affect achievement of the objectives presented above. For instance, users of shared SAS analysis systems are often saddled with large files that they repeatedly copy and modify along with waiting for creation of analysis ready SAS files from production systems. On the other hand, analysts who use Gnu R are often stymied by the fact that R keeps all its data in main memory. While R can be run on distributed, shared nothing workstations to parallelize the analysis task, in order to take advantage of such a capability, it requires algorithm driven specialists.

Finally, you also need to consider the funding model of your group. If the funding model is based on a project by project support with no funding support for experimentation, then despite best intentions you most likely can only use existing tools and systems to perform the project. If the funding model supports experimentation and innovation you can think about how the experimentation and innovation helps you improve your performance along the objectives presented above.
In summary, you need to take stock of the following aspects of your group:

• Talent base of your analysts
• Engagement, intake and project management processes used by your group
• Orientation and funding support for tools/system development and experimentation
• Tools and systems you use today in your group, and
• Funding model for activities in your group

Thinking about where you want to go

This is one of the hardest tasks for managers of analysis groups to do correctly. They are constantly buffeted with requests and it is often easy to think about where you want to go in the context of requests you are getting. For example, say that you received a lot of ad-hoc, fire-fighting type of analysis requests last year, it is natural for you to request funding to deal with such contingencies. But, this is incremental thinking that does not result in a sustained competitive advantage you can provide for your company.

I propose that you should think about this problem just as an entrepreneur of a company would think of the problem. First, sit down and think about the opportunity for analysis for your company. Opportunity is often created by trends in the market place as well as changes in technology, systems, and processes. Sit back and think about these trends. For example, it is clear that more and more analysis organizations are adopting agile methods to deal with analysis problems. Can the ad-hoc requests be better dealt with an agile process with business owners essentially running the analysis in collaboration with an analyst with very little documentation? Is your company changing its strategy of creating and selling its offerings from one business model (e.g., low volume, high priced) to some thing different (e.g., high volume, low priced in a very competitive market). Think about how that change will affect the analysis demands from your organization. A company moving into a competitive, high volume business would need more real-time and rapid analysis. So, you have to make drastic changes along the speed and flexibility dimensions.

Next think about what the competition is doing. If your competition is beating you by using analysis methods and processes that are significantly different from yours, perhaps you can think about how your organization can also change to not copy what they have done, but innovate such that you are better than them.

Finally, think about the team talent mix, their objectives, orientation, culture, and processes that will be needed to out innovate your competition taking advantage of the underlying trends that affect your analysis group.

Now write a simple plan for achieving your objectives along the dimensions presented above having considered all of the above. That becomes your strategy and execution plan for improving your group.

Good luck!