Big Data Challenges Holding Organizations Back

Posted on Posted in General, News

Big Data Challenges Holding Organizations Back

Big Data Challenges
Articles from ADTMag  (Using Garter, Inc surveys)  site some very interesting things coming out about the adoption of Big Data.   In a nutshell, dealing with big data is not easy.  Simply obtaining Hadoop, Netezza, Teradata or the technology du jour doesn’t solve the problem. There’s finding the data you want,  parsing it if necessary, loading it and then transforming it into appropriate data models for end user digestion.  If you want tools like Tableau and Microstrategy to make meaningful reports, you’re going to need to transform and cleanse all that loaded data into something these tools can use. A few sentences can sum up this process, but the in the real world this requires orchestration. Big Data is, well…BIG and like most big things, it takes work to reel it in. Below are some interesting highlights from the articles (links to references are at below).

“Challenges are holding organizations back from making the improvements that they need,” the survey report said. “The majority of those surveyed report that their organization finds analyzing large amounts of data (73 percent) and acting on data in real-time (65 percent) as very real obstacles.”

  1. Around three-quarters (73 percent) agree that analyzing a lot of data is a challenge for their organization.
  2. Acting on data in real-time is seen as a challenge for 65 percent.
  3. Over eight in 10 (83 percent) state that their operational processes take too long.
  4. The majority of respondents’ organizations cannot act on (57 percent), mine (61 percent) or use (68 percent) real-time data.
  5. 56 percent of respondents’ organizations are using manual processes to collect and analyze data.
  6. Technology executives overwhelmingly initiate Hadoop adoption, and 68 percent of adoption is initiated within the C-suite.
  7. The main reasons for not investing in Hadoop are: no need; use of another system; no business case or management support; and no prioritization for the technology.
  8. Hadoop lags substantially when compared with general Big Data adoption.
  9. Focus on who will actually use the software — include application plans or tool enablement to drive consumption in planning — or risk expensive systems used by very few constituents.

Number 5 is the one that’s really surprising.  Having been involved implementing systems that acquire and process billions of records a day,  even if there is automated collection and ingestion of the data,  there still needs to be appropriate alerting and recovery when somethings goes wrong. When there’s that much data, a small issue with a single feed or stream can turn into a big problem very quickly.   Once the problem is fixed you need to process the backlog.  This would seem next to impossible if attempting to process manually, if not impossible, incredibly frustrating.

Big data challenges don’t have to hold you back, we can help make it easier. Our tools and solutions really do make the hard things easier,  our founding members have a ton of experience in big data and 365x24x7 mission critical systems. Many of us were working on big data before it was #BigData. Check us out at for more information on our tools like Conductor and our Data Warehouse In the Cloud solution : CloudCella.

References  :
Big Data Paralysis
Study Reveals what Drives Hadoop Adoption: Not Much, Actually
Garter, Inc