This blog concentrates on the expected unexpected external factors that can have a (negative) impact on your organizations’ Integrated Data Warehouse (IDW). The current discussions around what NSA can and cannot capture and store for data analysis got me thinking about the biggest elephant in the room: the government.
In the WSJ’s coverage of a house bill overhauling the NSA phone program, “Instead of collecting millions of Americans' phone records en masse, the NSA would ask phone companies to query their databases for connections to suspicious phone numbers.” In other words, all processing power that the NSA has been dedicating to the analytics of the phone metadata would be pushed back into the private phone companies’ IT departments. Do they have this in their analytics processing capacity plan budget? And how do they prioritize the security letter information requests vs. their own processing on a shared platform?
What about the major social media sites that have acting as the repository for the world’s online activities? Up to now, based on the press reports and Edward J. Snowden’s revelations, the NSA has been vacuuming-up this data in the background for internal pattern analysis. Can the social media sites expect a major uptick in security letters asking for analytics on their IDWs if this is put to a halt? If the data collection activities are further restricted, I can imagine a large broadening of the impact across all industries: queries from hardware chains on material purchases; queries from large retailers looking for specific market baskets; etc.
Here’s a link to the 2013 security letter activities. There were nearly 100,000 “targets” affected. It will be interesting to see what this looks like in 2015 if the current legislation restricting NSA collection and storage activities gets through the Senate.
But it isn’t just the NSA that has an impact on private company IDW infrastructures. Sarbanes – Oxly changed the storage requirements on practically every company activity. Transportation companies have to track and report in detail on passenger and shipping manifests. The list goes on and on.
So, when building your organization’s IDW capacity plan, please do keep an eye on the Washington (or your home country’s) legislature. Something big may be coming your way.