ai-course/docs/section1/How LLMs work/Difference between structured and unstructured data.md

Structured vs. Unstructured Data: A Core Comparison

Structured data and unstructured data are two fundamental categories used to classify data based on their format, organisation, and the methods required for their processing and analysis. They differ significantly across various aspects, including their structure, nature, storage, processing, and the expertise needed to handle them.

Structured Data

Structured data is characterised by its highly organised, predefined, and tabular format, typically arranged in rows and columns with a fixed schema. It is often described as quantitative data, which means it commonly contains numbers or textual elements that can be easily counted, measured, and searched. This inherent organisation makes it straightforward to group, sort, and analyse based on specific attributes or characteristics.

Examples of structured data include:

• Spreadsheets (such as Google Sheets and Microsoft Office Excel files).
• Relational databases (like those used in CRM systems and financial records).
• Log files.
• Specific details such as employee names, contacts, ZIP codes, addresses, and credit card numbers.
• Information within airline reservation systems, including passenger names and flight numbers.
• IoT sensor readings (e.g., temperature or humidity) when recorded in a tabular format.

Structured data is typically stored in relational databases (often called SQL databases) or data warehouses. These storage systems have rigid schemas, where the configuration of columns and data types is strictly defined. Data is easily queried and manipulated using Structured Query Language (SQL), which was developed by IBM in the 1970s.

Handling structured data is generally accessible to business users with standard tools, as it does not require in-depth data science knowledge. There is an abundance of mature analytics tools for structured data, including Business Intelligence (BI) tools (like Tableau and Power BI) and Online Analytical Processing (OLAP) systems (such as Oracle Essbase and Apache Kylin). The structure for structured data is defined before storage, a concept known as "schema-on-write".

Advantages of structured data include its ease of use and access, its suitability for machine learning applications, and the wide array of available tools. However, its predefined nature limits its flexibility and usability, and changes to its schema can be time and resource-intensive.

Unstructured Data

In contrast, unstructured data has no predefined structure or schema and is stored in its native or raw format. It is often classified as qualitative data, containing subjective information that is harder to quantify. This type of data is rapidly growing, comprising 80-90% of all enterprise-generated data.

Examples of unstructured data include:

• Emails and text documents.
• Social media posts (including pictures and text within posts and comments).
• Multimedia files (such as image, video, and audio files).
• Call transcripts and message text files.
• Sensor data from Internet of Things (IoT) devices.

Unstructured data is best managed in non-relational (NoSQL) databases or data lakes, which are designed to handle massive amounts of raw data in any format. Storing unstructured data typically requires more space due to its volume and varied formats.

Analysing unstructured data is more complex and requires advanced techniques and specialised tools. These include:

• Machine learning (ML) libraries (e.g., TensorFlow, PyTorch, Hugging Face).
• Natural Language Processing (NLP) tools (e.g., Azure Cognitive Services, spaCy) for text-based data.
• Data search and indexing tools (e.g., Elasticsearch).
• Big data processing tools (e.g., Apache Spark, Apache Hadoop).
• Visual data analysis tools for image recognition (e.g., Google Cloud Vision AI platform).

Extracting insights from unstructured data often involves preprocessing steps, which can include data stacking or mining to detect patterns and interactions. Meaningful analysis typically requires data science expertise. The structure for unstructured data is applied when it is accessed or analysed, known as "schema-on-read".

Benefits of unstructured data include its flexibility, fast accumulation rates, and relatively easy and cheap storage. However, its challenges include the need for specialised expertise and tools, as well as potential data cleanliness issues due to its volume and non-uniform structure.

Fundamental Differences

The fundamental differences between structured and unstructured data can be summarised as follows:

• Format and Schema:

  • Structured Data: Adheres to a strict, predefined data model or schema, presented in a tabular format. This is "schema-on-write".
  • Unstructured Data: Has no predefined format or schema and is stored in its native or raw form. Its structure is applied "on-read".

• Nature:

  • Structured Data: Generally quantitative, easily measurable and searchable.
  • Unstructured Data: Predominantly qualitative, often subjective and harder to quantify.

• Storage Systems:

  • Structured Data: Resides in relational databases and data warehouses with rigid schemas.
  • Unstructured Data: Stored in file systems, content management systems, data lakes, or non-relational (NoSQL) databases.

• Processing and Analysis:

  • Structured Data: Easily queried and analysed using SQL and traditional business intelligence tools.
  • Unstructured Data: Requires advanced techniques such as machine learning (ML) and natural language processing (NLP) to extract insights.

• Accessibility and Expertise Required:

  • Structured Data: Accessible and easy to use by general business users.
  • Unstructured Data: Requires specialised skills and tools, often involving data science expertise for meaningful analysis.

It is worth noting that the lines between these two categories have become somewhat blurred. Many datasets today are considered semi-structured, incorporating certain markers or metadata (such as tags in a photograph or headers in an email) that enable some level of organization, while the core content remains unstructured. Both structured and unstructured data are essential in modern analytics and are leveraged across various industries and applications.