Data Scientist - When to Pivot and when to Generalize ?

The best minds from Teradata, our partners, and customers blog about relevant topics and features.
Teradata Employee

In this blog post, I discuss two common ideas and why getting this right is very important for data scientists (and sponsors) in order to be successful in projects.


Most of the folks in the business community are generally aware of these terms when they present ideas to a key sponsor or audience. These concepts are like bread and butter for them. But for data scientists, the terms and the usefulness of this may not be that familiar and hence my intention to write this blog post.


What does Pivot & Generalize mean in a data science project mean?


Pivot means turning your work and aligning it with something that is expected of you or pointing towards a specific goal. You found 4 animals in the data, and somebody is looking for animals like a cat or a dog. You do your best effort to map your animals to "cattish" or "doggish" definitions in the hope that your results are accepted.

© Can Stock Photo / Gajus


Generalize is something that you create in an abstract form from raw data with your creativity for ease of use or for communicating your analysis. We are basically saying there are five animals in your data as we found it, and not even care about what's expected out of it or the usefulness. Generalization can also happen with models data scientists build. We want to make sure the work is used across multiple domains and build the algorithmic stack with best of breed techniques.


Most business folks will tend to Pivot findings to a business objective. Most data scientists will Generalize for the sake of science. Therein lies the basic difference.


Why is Pivoting tough for a data scientist?


Give your customers what they want right? Ask the business person - Change your product, strategy to align towards those goals etc.,. For a data scientist, this could be harrowing and in fact, the whole concept could be anti-thesis of what they stand for. My algorithm says X and takes these inputs and outputs. There is no output I can give you EXACTLY, for the fear of misinterpretation! I used Random Forest and here's the explainability of the trees that made the ensemble. How do I put this one slide as a solution?


And why consuming Generalized results is difficult for a business?


Business folks want results targeted towards specific goals. If you give them something abstract, they will not be able to make use of it. It has to solve a business problem! It's like building technology first looking for a problem, however, cool the technology is. Startups often try to do this and some are successful. It's common knowledge that most startups fail if business problems are an afterthought.


Crossing the Chasm doesn't have to be difficult!

© Can Stock Photo / alphaspirit


Most of the time it requires the right mix of art and science that bridges the science to the requirement or business outcome. A blend of experienced people like business consultants, data scientists, and a mature platform can bring data science to the business all the way from prototyping to production. Ask Teradata/Thinkbig if you want to know more. This is exactly what we have been helping customers with!


Wait - I thought all along, data scientists are supposed to be capable of communicating to the business aka Pivot!


In reality, most data scientists try very hard to map the science to business problems. However, only a very few are successful on their own 100%. They would indeed be the unicorns!

Most fall short of the goal and the experience can be very frustrating and be challenging while focussing on both pivoting stuff to business and generalizing the data science methods. Translating science to business or vice versa is absolutely a team effort the requires deep appreciation/respect to both technical and business domains from everyone in order for projects to be successful.

1 Comment
Teradata Employee

An excellent perspective on how to get better business results from deep analytics.  Thanks for sharing your wisdom Karthik.