Big data and getting bigger all the time

Posted by on in Blogs
When you read or hear the term "Big Data" what do you think about?  Do you visualize an extremely large database or data warehouse?  Do you think about architectures and algorithms for storing and processing big data?  When haven't we had big data in relation to the systems and services that we are asked to program?  Is yesterday's big data small data today?  With the Internet and the number of humans creating and consuming data, I guess we do have really big data today and it is only getting bigger.  When I see a temperature rise in the number of articles, blog posts, conferences and books that appear on a topic, I love to dig into the topic.

Here are a number of links to articles that I've been reading about big data:

To paraphrase some lines from the Beatles song "It's Getting Better All The Time" (apologies to John, Paul, George and Ringo):

I used to get mad at my data

The databases I used weren't cool

You data has grown, relations complex

We might partially blame George Boole.

I've got to admit data's getting bigger

A little bigger all the time

I have to admit data's getting bigger

It's getting bigger since you've been online.

How big is your data?  Is it getting even bigger?  What architectures and tools are you using to mine it and keep it under control?

Gold User, Rank: 1, Points: 2466
David Intersimone (known to many as David I.) is a passionate and innovative software industry veteran-often referred to as a developer icon-who extols and educates the world on Embarcadero developer tools. He shares his visions as an active member of the industry speaking circuit and is tapped as an expert source by the media. He is a long-standing champion of architects, developers and database professionals and works to ensure that their needs are folded into Embarcadero's strategic product plans. David holds a bachelor's degree in computer science from California Polytechnic State University at San Luis Obispo, California.


  • Guest
    Alexandre Machado Friday, 3 June 2011

    Given the fact that Delphi's compiled programs are behind C# and even Java (!) in terms of execution speed (at least in string/floating point operations), what are compiler guys doing today to make Delphi be as fast as C again?

  • Guest
    David I Friday, 3 June 2011

    Alexandre - the compiler teams continue to work on next generation compilers for Delphi and C++Builder. These next generation compilers will start appearing next year and in the years after.

    Do you have big data?

  • Guest
    Bear Saturday, 4 June 2011

    yes, we have big data, millions of articles, but how to use them, mine them is a problem.
    there are some open source software for this: data mining, text mining. etc.

  • Guest
    Stephane Wierzbicki Saturday, 4 June 2011

    Hello David I,

    I'm currently looking for an data mining solution (it's getting more and more complex to work with simple SQL Statements / Huge SQL Scripts). I'll certainly go with Pentaho (open source software).

  • Guest
    David M Sunday, 5 June 2011

    Hi David,

    We have fairly large datasets, measured in multiple gigabytes. One instrument some of our users use records data at about 2GB per minute, although that's unusual. Our software processes all the data and obviously has to handle this sort of quantity, which isn't easy in a 32-bit process. We have indexed and caching systems, so we can calculate slices or portions of the end data at a time, and read in from disk only the bits we need. A 64-bit C++ compiler (with good FP code, too!) would be exceptionally handy, mostly because it would allow us to keep more data in memory at any one time.



  • Guest
    Captain Monday, 20 June 2011

    When can I expect Delphi to support GPGPU developments for processing my Big Data?

  • Guest
    David I Monday, 20 June 2011

    >Captain said - When can I expect Delphi to support GPGPU developments for processing my Big Data?

    We have research projects to do two things with the GPU - 1) graphics/imaging/UI processing, 2) CUDA support. No time frame yet when the research will result with the technologies being included in projects.

    Is there specific capabilities you are looking for? CUDA support?

  • Please login first in order for you to submit comments
  • Page :
  • 1

Check out more tips and tricks in this development video: