Clint Eastwood And Data Scientists. Can Cowboy Processes Drown Your Business?

Learn Data Science
Teradata Employee

Cowboys have a reputation of getting stuff done, often regardless of the consequences. One of my favorite tough-guy western actors is Clint Eastwood. You never got in his way, or crossed him. 

“Young fella, if you’re looking for trouble I’ll accommodate ya”

Clint Eastwood in True Grit (1969)

Cowboys are stereotypically resourceful, determined, and ultimately productive, but may lack a longer term vision resulting in undesirable consequences. If not recognized, embraced and managed, you end up with the Wild West. Sometimes, today’s data scientists are not much different than yesterday’s cowboys. They also can be resourceful, determined, and ultimately productive, but also may lack a longer term vision. If their talents and personalities are not recognized, embraced and managed, your analytic organization could also turn into the wild-wild west, drowning in cowboy processes.

Let me explain. Many years ago, when website personalization was being used by a handful of ecommerce companies, my analytic team developed a substantial improvement to the way products were displayed on a website. Instead of immediately implementing this function in a production environment (which would have required massive IT investment), we decided to prototype it, test it and estimate the actual upside first. The results were astounding, many millions of dollars straight to the top and bottom-line. In a meeting with senior executives, there was palpable excitement, until... 

... IT said, “We estimate this will take about 6 months to operationalize. Which of these critical initiatives do you want taken off the roadmap to accommodate that timeline?”

The company president responded “We can’t delay any of these.” Looking straight at me he asked, “Scott, can you continue to execute this until we find space on the roadmap?”

With millions of dollars at stake, what else could I say other than: “Of course!” 

Fast-forward a few years, and my team was still manually running the analytic process daily. Everything was working fine. We had implemented checks, versioning and documentation of the process to ensure it would continue to be executed daily. We were doing what IT is great at, but definitely weren’t doing it as well. And there was no plan to properly operationalize this process.

Business teams are often frustrated that the prototypes built by their analytic teams cannot be operationalized quickly. To a non-IT resource the long development time does not make sense. After all, the data exists, they have working code... all they need is someone to implement this logic in a production environment with proper monitoring and governance. How hard can that be? And, the frustration is magnified when adjustments (even minor) have to be made to the logic, IT still requires many weeks or even months to make those changes.

So, what causes these types of bottlenecks? Well, there are a few specific causes, including:

  • Antiquated Development Processes:  IT implementing every analytic process with traditional IT standards, which often is a detailed, methodical, and time-consuming approach. Analytic processes require much more agility.

  • Misguided SLAs:  Business requiring strict SLA’s around downtime or performance, which requires the process be built with ironclad reliability, eliminating the flexibility and agility truly required by the business.

  • Isolated Analytic Teams:  The analytic technologies are so specialized, that there are no IT resources available that can understand and implement the analytical processes effectively.

  • Poor Requirements:  To an analytic professional, writing requirements can be a tedious task.   And, often these are done without a long-term view on how the process will evolve.

  • No Productionalized Analytic Technology:  Analytic technology does not exist in the production environment, requiring analytic process prototype code to be completely rewritten.

This lack of agility forces the business to look for alternatives. Analytic teams have many of these skills, and can work in a very agile way. So why not have the analytic team create and execute this business process? 

I call these “cowboy processes,” which are regular, business critical, well-defined analytical tasks or jobs (often based on a Proof of Concept (POC) prototype) running on an analytical (i.e. “non-production”) system that ether is...

  • ... waiting for IT resources and prioritization

  • …not on the IT queue as the business expects it would take too long

Analytic teams, specifically data scientists, are very efficient at creating these cowboy processes. While not ideal, they are allow the business additional agility, but if not intentionally monitored they could become a large organizational risk.

What are the benefits of Cowboy Processes?

Analytic teams are uniquely trained to plow through data quickly and efficiently. Their tools are designed to extract data from many sources, shuffle it together and spit out an answer. And since they have first-hand knowledge of the complex math used in the algorithms and formulas, it is relatively simple for them to execute these in a no production way.

This results in two substantial benefits of cowboy processes:

  1. Immediate implementation -  Once the prototype has been completed, impact has been tested and the executive team desires a rollout, the analytic team simply needs to execute the prototype at scale. Unless the volumes are beyond the capabilities of the analytic tool, this is usually simple to do.

  2. Agility: abilty to adjust logic quickly- since this code is not “production,” it is usually very simple for the analytic team to adjust and “test” and implement the process.  

  3. Implementation of smaller analytic initiatives – since the investment required to develop cowboy processes is so much shorter than traditional development, it allows for analytics to be applied to much more targeted and specialized applications.

Are Cowboy Processes a Problem?

When I talk with business executives, they ask the obvious question: “Are cowboy processes a problem?”

They can be if you can’t adequately answer the following questions:

  • Do you know how many/what business critical processes are dependent on the analytic team for execution?  If not, you need to take an inventory. You could have massive exposure here.

  • What percentage of your analytic team’s time is being spent on these processes? Are there other things they could be doing that would provide more value to the organization? What would happen if you lost 10 percent, 20 percent or 50 percent of your team – would the cowboy processes dominate their time, keeping them from innovating?

  • What would happen if these processes were not executed? Would anyone notice? Who? Would there be a business impact? The processes that create the most exposure should be prioritized high to operationalize.

  • Do you know if these processes can only be executed by a single individual?  If so, there needs to be a deliberate attempt to cross-train and document the process to reduce the risk of that person leaving. 

  • Is the organization aware (business, IT, executives, where the jobs are executed?

Maybe the organization is willing to accept the above risks with cowboy processes, but there needs to be awareness of those risks.

Ideally Cowboy Processes shouldn’t exist.  But this is not practical without an infrastructure that allows analytic solutions to be very rapidly implemented (hours or days, not weeks or months) in production with IT providing oversight and the control of the logic and implementation being driven and owned by the business and analytic teams

How Can I Prevent Cowboy Processes?

Once a cowboy process is running, it is very difficult to justify having IT operationalize it – after all, there is no measurable financial “upside” in passing it over to IT once these processes are running, only a difficult-to-quantify reduction in risk.

You need to recognize that not every cowboy process should be operationalized. If it is not critical to the business (e.g. not much revenue exposure if it periodically fails) or it is simple to execute (and would require a substantial IT investment to operationalize), then it might not justify the resources to needed.

At the core of it, operationalizing conversations with IT and senior executives need to happen before the prototype is developed. There needs to be an agreement on the lift that will result in which potential actions. Three factors should determine the action:

  1. Financial Upside Thresholds

Based on “what if” results from the prototype testing

  1. Cost to Operationalize

Back of the envelope IT development estimates based on analytic team input of high-level requirements. (Note these requirements must include the ability to adjust the rules in a matter of hours or days, not weeks or months.) At the same time, IT has to communicate what would and would not be costly to implement, resulting in a deprioritization of very difficult/costly process.

  1. Effort and risk to manually execute

How much of analytic team resources will be required to maintain and execute this process, and how critical is it to ensure this is executed reliably?

The efficiency of your analytic team is critical to your business success.  Their cowboy skills should be embraced and managed; otherwise you could find your organization living out of control in the Wild West.

Originally Published on Jan 7, 2105 by TeradataVoice: