View all articles

A brief history of databases

By Stefan Gerber

Graphs are a part of everything we do. The people, places and things in our lives and the relationships that connect them. They are the social network of social network companies, the product recommendation engines at large online stores, and the fraud detection system of banks and financial firms, to name a few.

They are a fundamental part of many of the services we use every day.

But these services haven’t been around for long. And with the ever-increasingly connected nature of the world we live in, engineers have had to innovate to find ways of managing our highly connected data.

Where it started (1960s) #

Technological advancements and the invention of magnetic disc storage (hard drives) in the 1960s allowed programs to perform random, direct data accesses instead of the previously used sequential, magnetic tape storage. Igniting the start of a revolutionary era in computing.

During this era, data was organised in hierarchical, tree-like structures, with records being retrieved by direct access or, more often, by scanning and navigating through the tree’s links.

A visual depiction of a hierarchical data structure containing a car node connected to car parts.

An interesting thing to note is the inherent graph-like structure of these hierarchical or navigational databases. Not too surprising, though, as most people intuitively think and reason about collections of information as a graph.

Unfortunately, slow hardware at the time, combined with the difficulty of adding and removing records from these databases, sparked a dire need for a more performant database solution able to adapt quickly to changes within the corporate environment.

The relational database boom (1980s) #

It was in the late 70s when Edgar Codd published his revolutionary piece on the relational data model introducing the most well-known and often used database type, to this day, because of how well-documented, understood and tested it is.

Relational databases introduced a way of structuring data in a tabular form and grouping together similar entities, and then linking different entities together by means of keys and linking tables.

For better or worse, this era introduced and ingrained the mentality that all data should map to a table.

However, during the dot com boom and the surging popularity of the internet in the late 90s, previously unplanned and unimaginable shapes and volumes of data were making their way into corporate data stores. Unfortunately, there was no way to analyse and gain insight from this data. A new way of structuring masses of data was needed.

The social era (2000s) #

With the rise in popularity of web-based applications in the early 2000s, new ways of transferring data over the web started to emerge. People started socialising, listening to music, uploading images, and playing games. Behaviours which generate data that is inherently not tabular, paving the way for new database types, including key-value, document and graph.

These new technologies allowed innovative companies to introduce new ways of interacting with customers, including real-time recommendations, personalised user experiences, fraud detection and orders of magnitude improvements to their intelligence and analytics pipelines compared to relational databases, to name a few.

The next large leap in storage innovation came from the need to scale. Companies were dealing with gigantic amounts of data, too much for one machine. On top of that, global audiences made it necessary to have servers in multiple global regions, leading to the boom of distributed systems and microservices.

For the past twenty-odd years, NoSQL technologies have enabled developers to pick their data storage solution based on the data’s shape, velocity and scalability requirements. Making it possible for small teams to rapidly scale, manage and analyse the data generated by millions of customers.

However, the industry has moved on. Managing and delivering data to customers at scale is a solved problem, and customers expect services to work. If your offering doesn’t, they will move on to other platforms.

Instead, companies have shifted their focus from speed and cost, to value.

The value of relationships (2020s) #

Graphs are relevant now because the tech industry’s focus has shifted over the last few decades. The realisation today is that data is inherently more valuable when it is connected, and that most data is inherently connected. We just haven’t been able to effectively store and analyse it until recently with the emergence of native, distributed graph database technologies. The industry has gone full circle.

At Rockup, we believe in the power of relationships. If you think your company can benefit from a deeper understanding of the data that drives your success, contact us to schedule a free consultation call.

Subscribe to Rockup’s Insights

We are committed to keeping your data safe.

 

Too much email?

RSS