Added 2 sections; added logo

docs/section1/How LLMs work/Difference between structured and unstructured data.md

@@ -0,0 +1,75 @@
+# Structured vs. Unstructured Data: A Core Comparison
+
+**Structured data** and **unstructured data** are two fundamental categories used to classify data based on their format, organisation, and the methods required for their processing and analysis. They differ significantly across various aspects, including their structure, nature, storage, processing, and the expertise needed to handle them.
+
+### Structured Data
+
+**Structured data** is characterised by its **highly organised, predefined, and tabular format**, typically arranged in rows and columns with a fixed schema. It is often described as **quantitative data**, which means it commonly contains numbers or textual elements that can be easily counted, measured, and searched. This inherent organisation makes it straightforward to group, sort, and analyse based on specific attributes or characteristics.
+
+Examples of structured data include:
+
+- **Spreadsheets** (such as Google Sheets and Microsoft Office Excel files).
+- **Relational databases** (like those used in CRM systems and financial records).
+- **Log files**.
+- Specific details such as **employee names, contacts, ZIP codes, addresses, and credit card numbers**.
+- Information within **airline reservation systems**, including passenger names and flight numbers.
+- IoT sensor readings (e.g., temperature or humidity) when recorded in a tabular format.
+
+Structured data is typically stored in **relational databases** (often called SQL databases) or **data warehouses**. These storage systems have rigid schemas, where the configuration of columns and data types is strictly defined. Data is easily queried and manipulated using **Structured Query Language (SQL)**, which was developed by IBM in the 1970s.
+
+Handling structured data is generally accessible to business users with standard tools, as it does not require in-depth data science knowledge. There is an abundance of mature analytics tools for structured data, including Business Intelligence (BI) tools (like Tableau and Power BI) and Online Analytical Processing (OLAP) systems (such as Oracle Essbase and Apache Kylin). The structure for structured data is defined **before storage**, a concept known as "schema-on-write".
+
+**Advantages** of structured data include its ease of use and access, its suitability for machine learning applications, and the wide array of available tools. However, its predefined nature limits its flexibility and usability, and changes to its schema can be time and resource-intensive.
+
+### Unstructured Data
+
+In contrast, **unstructured data** has **no predefined structure or schema** and is stored in its native or raw format. It is often classified as **qualitative data**, containing subjective information that is harder to quantify. This type of data is rapidly growing, comprising **80-90% of all enterprise-generated data**.
+
+Examples of unstructured data include:
+
+- **Emails and text documents**.
+- **Social media posts** (including pictures and text within posts and comments).
+- **Multimedia files** (such as image, video, and audio files).
+- **Call transcripts** and message text files.
+- **Sensor data** from Internet of Things (IoT) devices.
+
+Unstructured data is best managed in **non-relational (NoSQL) databases** or **data lakes**, which are designed to handle massive amounts of raw data in any format. Storing unstructured data typically requires more space due to its volume and varied formats.
+
+Analysing unstructured data is more complex and requires **advanced techniques and specialised tools**. These include:
+
+- **Machine learning (ML) libraries** (e.g., TensorFlow, PyTorch, Hugging Face).
+- **Natural Language Processing (NLP) tools** (e.g., Azure Cognitive Services, spaCy) for text-based data.
+- **Data search and indexing tools** (e.g., Elasticsearch).
+- **Big data processing tools** (e.g., Apache Spark, Apache Hadoop).
+- **Visual data analysis tools** for image recognition (e.g., Google Cloud Vision AI platform).
+
+Extracting insights from unstructured data often involves preprocessing steps, which can include data stacking or mining to detect patterns and interactions. Meaningful analysis typically requires **data science expertise**. The structure for unstructured data is applied **when it is accessed or analysed**, known as "schema-on-read".
+
+**Benefits** of unstructured data include its flexibility, fast accumulation rates, and relatively easy and cheap storage. However, its challenges include the need for specialised expertise and tools, as well as potential data cleanliness issues due to its volume and non-uniform structure.
+
+### Fundamental Differences
+
+The fundamental differences between structured and unstructured data can be summarised as follows:
+
+- **Format and Schema**:
+    
+    - **Structured Data**: Adheres to a **strict, predefined data model** or schema, presented in a tabular format. This is "schema-on-write".
+    - **Unstructured Data**: Has **no predefined format** or schema and is stored in its native or raw form. Its structure is applied "on-read".
+- **Nature**:
+    
+    - **Structured Data**: Generally **quantitative**, easily measurable and searchable.
+    - **Unstructured Data**: Predominantly **qualitative**, often subjective and harder to quantify.
+- **Storage Systems**:
+    
+    - **Structured Data**: Resides in **relational databases** and **data warehouses** with rigid schemas.
+    - **Unstructured Data**: Stored in **file systems, content management systems, data lakes**, or **non-relational (NoSQL) databases**.
+- **Processing and Analysis**:
+    
+    - **Structured Data**: Easily queried and analysed using **SQL** and traditional business intelligence tools.
+    - **Unstructured Data**: Requires **advanced techniques** such as **machine learning (ML) and natural language processing (NLP)** to extract insights.
+- **Accessibility and Expertise Required**:
+    
+    - **Structured Data**: **Accessible and easy to use** by general business users.
+    - **Unstructured Data**: Requires **specialised skills and tools**, often involving data science expertise for meaningful analysis.
+
+It is worth noting that the lines between these two categories have become somewhat **blurred**. Many datasets today are considered **semi-structured**, incorporating certain markers or metadata (such as tags in a photograph or headers in an email) that enable some level of organization, while the core content remains unstructured. Both structured and unstructured data are essential in modern analytics and are leveraged across various industries and applications.

docs/section1/How LLMs work/Tokenization.md

@@ -0,0 +1,65 @@
+# Tokenization: Building Blocks for Language Models
+
+**Tokenization** in language models is the process of **converting a sequence of text into smaller, discrete components known as tokens**. These tokens serve as the **basic building blocks that a language model processes and understands**. The primary reason for tokenization is to help machines understand human language by breaking it down into manageable units that are easier to analyze.
+
+### Why Tokenization is Important
+
+Tokenization is a **crucial first step in training Large Language Models (LLMs)** and is foundational for artificial intelligence (AI). It matters for several reasons:
+
+- **Makes text machine-readable** Machine learning algorithms, including LLMs and deep learning models, are generally incapable of processing plain text in its raw form; they require numbers as input. Tokenization converts raw text into a numerical format, allowing models to process it mathematically.
+- **Facilitates pattern recognition** By breaking text into tokens, algorithms can more easily identify patterns, relationships, and context within language. This pattern recognition is essential for machines to understand and respond to human input.
+- **Defines the model's vocabulary** The set of unique tokens an LLM is trained on forms its vocabulary. This vocabulary determines what kinds of things a model is trained to recognise and produce.
+
+### How Tokenization Works in Practice
+
+The process typically involves several steps:
+
+1. **Conversion to Numerical Format** Text is first transformed into a numerical format that the machine learning model can understand.
+2. **Vocabulary Creation and ID Assignment** The LLM is trained on a **vocabulary of unique tokens** derived from large text corpora. As training continues, any new tokens encountered are added to this vocabulary and assigned a **unique numerical ID**. For example, "I heard a dog bark loudly at a cat" could be tokenized and assigned IDs like `I (1), heard (2), a (3), dog (4), bark (5), loudly (6), at (7), cat (8)`, resulting in a sequence like ``.
+3. **Creation of Embeddings** After tokenization, **multi-valued numeric vectors, known as embeddings, are used to represent these semantic relationships**. An embedding is assigned to each token based on how commonly it's used together with, or in similar contexts to, other tokens. Words with similar meanings or contexts are expected to have similar vector representations, with distance and direction between vectors encoding their similarity. For LLMs, the **context of every word becomes an embedding**, in addition to the word itself, allowing the meanings of entire sentences, paragraphs, and articles to be searched and analysed. Embeddings are foundational for AI as they enable machine learning models to find similar objects.
+4. **Output Generation** During output generation, the model predicts a vector value for the next token in the sequence. It then selects the most probable token from its vocabulary based on this vector value. This is an **iterative process**: the predicted token is appended to the sequence so far and used as input for the next iteration, building the final output one token at a time.
+
+It is important to note that the **correlation between words and tokens is complex**. Sometimes a single word is broken into multiple tokens, or the same word might be tokenized differently depending on context or surrounding words. Spaces, newline characters, and punctuation may or may not be included in tokens.
+
+### Common Tokenization Methods
+
+Tokenization methods vary based on the granularity of the text breakdown and the specific requirements of the task. Common methods include:
+
+- **Word tokenization**: Splits text into individual words, typically based on delimiters like spaces. This is most effective for languages with clear word boundaries like English.
+- **Character tokenization**: Splits text into individual characters. This method is useful for languages lacking clear word boundaries or for tasks requiring granular analysis, such as spelling correction.
+- **Subword tokenization**: Breaks text into units larger than a single character but smaller than a full word. This approach balances the need for a manageable vocabulary size with the ability to handle unknown words or complex syntax. Examples include **Byte-Pair Encoding (BPE)**, which merges frequent pairs of characters or bytes to form tokens until a set vocabulary size is reached, and **WordPiece**. GPT models developed by OpenAI use Byte-Pair Encoding.
+
+The choice of token size affects model efficiency and language coverage:
+
+- **Smaller tokens** (character or subword tokenization) enable the model to handle a wider range of inputs, including unknown words, typos, or complex syntax. This might also reduce vocabulary size, requiring fewer memory resources. However, a given text is broken into more tokens, requiring additional computational resources and reducing the maximum input/output size for a fixed token limit.
+- **Larger tokens** (word tokenization) result in fewer tokens per text, requiring fewer computational resources. However, this might increase vocabulary size and limit the model's ability to handle unknown words or typos.
+
+### Tokenization Challenges
+
+The nuances of human language present unique challenges for tokenization:
+
+- **Ambiguity**: Language is inherently ambiguous, and tokenization can lead to different interpretations of a sentence.
+- **Languages without clear boundaries**: Some languages, like Chinese, Japanese, or Thai, lack clear spaces between words, making word tokenization more complex. Multilingual tokenization models like XLM-R and mBERT use subword tokenization to handle these languages.
+- **Special characters**: Handling email addresses, URLs, or special symbols can be tricky, requiring advanced models to incorporate rules for consistent processing.
+- **Morphology**: While subword tokenization aims to capture frequent substrings, it does not guarantee decomposition into linguistic morphemes (smallest meaningful units). Research is mixed on whether the lack of morphology in subword tokenizers impairs model performance, particularly for languages with complex morphology like German.
+- **Understanding character-level information**: It is not trivial for a model to "look inside" a token to know its constituent characters (e.g., that "dog" contains "d", "o", "g"). Though state-of-the-art LLMs can perform well on some character-level tasks, they may struggle with others like counting character occurrences. Some variability in how a root word is tokenized can actually improve predictions about contained characters.
+- **Token limits and context window**: LLMs have limitations on the maximum number of input and output tokens, often combined into a "context window". Larger context windows significantly increase computational complexity quadratically, making it costly and inefficient. This also affects the maximum length of text that can be provided as input or generated as output.
+- **Computational Cost**: Tokenization and subsequent processing require significant computational resources. Training large language models involves processing petabytes of data.
+
+### Use Cases for Tokenization
+
+Tokenization forms the backbone for a wide array of AI applications by enabling machines to process and understand vast amounts of text data:
+
+- **Search engines**: Tokenization dissects user queries to sift through documents and retrieve relevant results.
+- **Machine translation**: It segments sentences in a source language for translation and reconstruction in the target language, retaining original context.
+- **Speech recognition**: Spoken words are converted into text and then tokenized, allowing systems like Siri or Alexa to process and act upon commands.
+- **Sentiment analysis**: Tokenization extracts insights from user content like reviews to determine sentiments (positive, neutral, negative).
+- **Chatbots and virtual assistants**: Tokenization enables chatbots to understand user intent and respond appropriately.
+- **Named Entity Recognition (NER)**: It identifies and classifies entities like names, places, or organisations in text.
+- **Text classification**: LLMs can classify text based on similar meanings or sentiments for tasks like spam detection or moderating user-generated content.
+
+### Tools and Future Directions
+
+Prominent tools for tokenization include NLTK (Natural Language Toolkit), spaCy, and Hugging Face Transformers. Hugging Face Transformers provides fast tokenizers built using Rust and supports various subword tokenization methods like BPE and WordPiece. BERT tokenizer, for example, excels in context-aware tokenization.
+
+Future research aims to explore more efficient model architectures that can handle longer sequences with lower computational complexity, and design new self-supervised pre-training tasks. There's also ongoing interest in understanding how morphological structure impacts language model performance and whether incorporating such information can lead to better generalisation, particularly for languages with complex morphology.

docs/section1/How LLMs work/Vector Databases.md

@@ -0,0 +1,70 @@
+# Vector Databases: AI's Semantic Memory Powerhouse
+
+**Vector databases** are specialized systems designed to **store, manage, and index high-dimensional vector data**, which are often used as embeddings representing various forms of data like text, images, or audio. Unlike traditional relational databases that use rows and columns, vector databases represent data points as arrays of numbers, enabling them to handle unstructured datasets more effectively. They are increasingly popular because they provide the speed and performance necessary for generative Artificial Intelligence (AI) applications.
+
+### How Vector Databases Work
+
+The fundamental principle behind vector databases involves **converting data into numerical representations called vectors or embeddings** and then enabling **similarity searches** based on their proximity in a multi-dimensional space.
+
+1. **Vectors and Embeddings**:
+    
+    - A **vector** is an array of numbers that defines a point in a dimensional space. Each number in the array indicates the object's position along a specified dimension. These can represent complex objects like words, images, videos, or audio. For example, the word "cat" might be represented as `[0.2, -0.4, 0.7]` in a 3-dimensional space. In practice, vectors can have hundreds or thousands of dimensions to capture intricate relationships.
+    - **Embeddings** are the output of a deep learning process that creates these vectors. They are numerical representations of data points that convert various data types into arrays of numbers that machine learning (ML) models can process. The creation of embeddings is often a hidden layer within a neural network, automatically transforming inputs into these multi-dimensional representations. This allows models to find similar objects because words with similar meanings or contexts are positioned close to each other in the vector space.
+2. **Embedding Generation**:
+    
+    - Machine learning models, particularly deep learning models and neural networks, are trained on large datasets to convert raw, unstructured data (like text or images) into these numerical vectors. For example, a word embedding model like Word2Vec uses a neural network to create word embeddings, where the context of a word helps infer its meaning and position in the vector space. In Large Language Models (LLMs), the context of every word, and even entire sentences, paragraphs, or articles, can become an embedding.
+3. **Storage and Indexing**:
+    
+    - Vector databases store these generated embeddings, along with their metadata (e.g., title, description).
+    - To accelerate searches within high-dimensional data spaces, vector databases create **indexes** on these embeddings. Common indexing algorithms include Hierarchical Navigable Small World (HNSW), Locality-Sensitive Hashing (LSH), and Product Quantization (PQ), which map vectors to new data structures for faster similarity searches.
+4. **Similarity Search**:
+    
+    - When a user queries or prompts an AI model, the query itself is converted into a vector embedding.
+    - The vector database then calculates the **distance** between this query vector and the stored vectors using various similarity metrics, such as **cosine similarity** or **Euclidean distance**. The closer the embeddings are, the more similar the original data points are considered.
+    - The database returns the most similar vectors or "nearest neighbours" to the query vector. This process is known as an **Approximate Nearest Neighbour (ANN) search**, which balances speed with accuracy.
+
+### Why Vector Databases are Used (Advantages)
+
+Vector databases offer several benefits that make them crucial for modern AI applications:
+
+- **Speed and Performance**: They enable rapid searches through large datasets (millions or billions of data points) by using indexing techniques and efficient distance-calculating algorithms, returning results within milliseconds.
+- **Scalability**: They can store and manage massive amounts of unstructured data by scaling horizontally with additional nodes, maintaining performance even as data volumes increase.
+- **Lower Cost of Ownership**: By enabling faster data retrieval, vector databases can speed up the training of foundation models and reduce the need to re-process entire datasets for every query, leading to cost savings.
+- **Data Management**: They often include built-in features for updating and inserting new unstructured data.
+- **Flexibility**: Vector databases are designed to handle the complexity of multi-dimensional data, including images, videos, and various text types. They can be customised to meet specific business and AI requirements.
+- **Semantic Understanding**: Crucially, they allow machine learning models to understand the relationships between words and other objects, not just exact matches, which is foundational for AI.
+- **Long-Term Memory**: For machine learning models, especially LLMs, vector databases provide a form of "long-term memory," allowing the model to remember previous inputs and context without retraining or re-running the entire dataset through the model each time.
+
+### Use Cases for Vector Databases
+
+Vector databases form the backbone for a wide range of AI applications by enabling machines to process and understand vast amounts of text and other data.
+
+- **Retrieval-Augmented Generation (RAG)**: This is a key use case, where LLMs use vector databases to retrieve relevant, up-to-date, and domain-specific information from external data sources (e.g., product manuals, FAQs, databases, web pages) and incorporate it into their responses. RAG helps LLMs overcome limitations like outdated knowledge, hallucinations (generating false information), and generic responses.
+- **Conversational AI and Chatbots**: They enable chatbots and virtual assistants to understand user intent and respond appropriately by efficiently accessing and parsing relevant knowledge bases, providing real-time, contextual answers.
+- **Recommendation Engines**: By representing user preferences and item attributes as vectors, these systems can suggest similar products, movies, or articles based on vector similarity, enhancing user experience and increasing engagement.
+- **Vector Search (Semantic Search)**: This technique allows searches based on the meaning or context of a query, rather than just keywords, leading to more precise and relevant results. It includes finding similar images, audio, video, or text data.
+- **Anomaly Detection**: By identifying data points that are significantly different from existing clusters, vector databases can be used to detect unusual behaviour, such as fraudulent transactions or network anomalies.
+- **Text Classification and Sentiment Analysis**: LLMs can classify text based on similar meanings or sentiments, aiding in spam detection or analysing user-generated content.
+- **Drug Discovery**: Graph neural networks and generative neural networks, powered by underlying vector representations, have been used to predict properties of molecules and produce novel biomolecules.
+
+### Challenges and Considerations
+
+While powerful, implementing vector databases and RAG systems can pose challenges:
+
+- **Integration Complexity**: Integrating retrieval systems with LLMs can be difficult, especially with multiple external data sources in varying formats. Ensuring data consistency and uniform embeddings across sources is crucial.
+- **Data Quality**: The effectiveness of a RAG system heavily relies on the quality of the input data; poor source content will lead to inaccurate responses. Human oversight and expert curation are often necessary.
+- **Computational Intensity**: Tasks like generating embeddings, comparing text meanings, and real-time data retrieval are computationally intensive, which can slow down the system as data size increases. This necessitates distributed computing and robust hardware infrastructure.
+- **Approximate Results**: Vector databases often provide approximate results, meaning applications requiring higher accuracy might need different database types at the cost of slower processing.
+- **Query Limitations**: Vector indexes are well-suited for fact-based queries, but may not be optimal for tasks like summarization, where an LLM needs to process all possible contexts on a topic rather than retrieving specific facts.
+
+### Tools and Integration
+
+Prominent tools and services for working with vector databases include:
+
+- **Cloudflare Workers AI and Vectorize**: Cloudflare offers Vectorize, a globally distributed vector database, combined with Workers AI, to make it faster and more affordable to generate and query embeddings for AI applications without extensive backend infrastructure.
+- **LlamaIndex**: This data framework helps developers manage the flow of information from external data sources to LLMs for RAG-powered applications.
+- **LangChain**: An open-source orchestration framework that provides integrations for various embedding methods and vector stores.
+- **Open-source solutions**: Examples include Weaviate or Milvus, which offer RESTful APIs and programming language support.
+- **Database extensions**: Some traditional databases, like PostgreSQL, offer extensions (e.g., pgvector) for vector similarity search.
+
+The integration of vector databases with LLMs is further supported by protocols like the **Model Context Protocol (MCP)**, which standardises how AI applications connect and interact with external tools and data sources, similar to how USB-C standardises hardware connections. MCP allows AI agents to be context-aware and use tools and resources by providing a standardised way to retrieve external context, such as documents, knowledge base entries, and database records.

docs/section1/How LLMs work/What is an LLM.md

@@ -0,0 +1,58 @@
+# Large Language Models: Core Concepts and Applications
+
+**Large Language Models (LLMs)** are a type of artificial intelligence (AI) system designed to **understand, process, and generate human language at scale**. They are built using **deep learning techniques**, specifically a neural network architecture known as a **transformer model**. LLMs are trained on **vast amounts of text data**, which is why they are called "large". This training enables them to respond to human queries with text, audio, or visuals that naturally resemble human creations.
+
+### How Large Language Models Work
+
+LLMs operate through a complex interplay of data, architecture, and training processes:
+
+- **Underlying Technology**: At their core, LLMs are built on **machine learning**, a subset of AI that allows systems to learn from data without explicit programming. They leverage **deep learning**, which uses multi-layered artificial neural networks (often referred to as "deep neural networks") to simulate the complex decision-making power of the human brain. These networks comprise an input layer, an output layer, and multiple hidden layers in between, sometimes hundreds or thousands.
+- **Data Collection and Pre-processing**:
+    - **Massive Datasets**: LLMs are trained on enormous datasets, sometimes tens of gigabytes or even **petabytes of text data**, including books, articles, conversations, and web content. For instance, GPT-3 was pre-trained on approximately 45 terabytes of data. The quality of this data significantly impacts the LLM's learning.
+    - **Tokenization**: The first step in processing text is **tokenization**, where raw text is broken down into smaller units called **tokens**. These tokens can be words, parts of words (subwords), characters, or punctuation marks. For example, GPT models use a type of subword tokenization called Byte-Pair Encoding (BPE). Tokens allow the model to identify patterns, relationships, and context within language. Roughly, a token is about three-quarters of a word.
+    - **Embeddings**: After tokenization, LLMs turn these tokens into **embedding vectors**. These are multi-dimensional numerical representations of words or phrases that capture their semantic meaning and relationships. In this high-dimensional space, semantically similar words or concepts are mapped to vectors that are closer together.
+- **Training and Learning**:
+    - **Transformer Architecture**: LLMs primarily use the **transformer architecture**, which enables them to process entire sequences of text in parallel. This architecture, particularly through its **self-attention mechanisms**, allows the model to understand the context of each word by considering its relation to every other word in a sentence. This makes them highly effective at comprehending vague or newly contextualized human language.
+    - **Prediction and Self-learning**: During training, the model learns to predict the next token in a sequence. It achieves this through **self-supervised learning**, where it infers "pseudolabels" from unlabeled data, such as predicting masked words in text samples. The model iteratively adjusts its internal **parameters** (billions or even trillions of adjustable weights and biases) to minimize prediction errors.
+    - **Computational Cost**: This training process is **compute-intensive, time-consuming, and expensive**, often requiring thousands of clustered graphics processing units (GPUs) and weeks of processing, costing millions of dollars.
+- **Fine-tuning**: After initial pre-training on vast datasets to gain general language capabilities, LLMs are often **fine-tuned** on smaller, more specific datasets or through other methods like **Reinforcement Learning with Human Feedback (RLHF)**. Fine-tuning refines the model's understanding to perform specific tasks more accurately.
+
+### Applications of Large Language Models
+
+LLMs are incredibly versatile and are used in a wide range of applications across industries:
+
+- **Generative AI**: Creating new and realistic content such as long-form text (essays, poems, articles), high-quality images, realistic video or audio, and code in response to user prompts.
+- **Natural Language Processing (NLP)**: Tasks include text summarisation, language translation, question answering, sentiment analysis, named entity recognition, and text classification.
+- **Chatbots and Virtual Assistants**: Powering intelligent conversational agents for customer service and general information.
+- **Software Development**: Generating and reviewing code, writing functions, and assisting with programming tasks.
+- **Recommendation Systems**: Providing personalised suggestions based on semantic understanding of text data.
+- **Business Intelligence**: Analysing large volumes of business reports and market research documents to derive meaningful insights.
+
+### Advantages and Limitations of LLMs
+
+**Advantages**:
+
+- **Flexibility and Adaptability**: LLMs can perform a wide variety of tasks and respond to unpredictable queries in natural human language, unlike traditional programs that require specific syntax.
+- **Accessibility**: Pre-trained LLMs, especially open-source foundation models, make advanced AI capabilities more accessible to developers and organisations without extensive computational resources or labelled data.
+- **Reduced Development Time**: Using pre-trained models as a starting point can significantly reduce the time and resources required to develop AI applications from scratch.
+
+**Limitations**:
+
+- **Hallucinations**: LLMs can confidently generate information that is plausible-sounding but factually incorrect or nonsensical, a phenomenon known as "hallucination".
+- **Limited/Outdated Knowledge**: Their knowledge is restricted to their training data, which can become outdated or lack domain-specific information, leading to generic or inaccurate responses for specific queries.
+- **Bias**: LLMs are susceptible to inheriting and amplifying biases present in their training data, potentially leading to skewed representations or unfair treatment based on factors like race, gender, or political viewpoint.
+- **Computational Intensity**: Training and fine-tuning LLMs require immense processing power and energy, raising concerns about economic and environmental sustainability.
+- **Black-Box Nature**: The decision-making processes of complex LLMs can be difficult to interpret, posing interpretability challenges.
+- **Context Window Limits**: LLMs have a maximum limit on the number of tokens they can process in their input and generate as output, which can restrict their ability to handle very long documents or conversations.
+
+### Current Advancements and Future Directions
+
+The field of LLMs is rapidly evolving, with several key advancements:
+
+- **Retrieval-Augmented Generation (RAG)**: This technique enhances LLMs by integrating them with external, up-to-date knowledge bases, typically stored in **vector databases**. RAG allows LLMs to retrieve relevant information and combine it with their generative capabilities, significantly reducing hallucinations and providing more accurate, grounded, and contextually relevant responses.
+- **Multimodality**: LLMs are increasingly being developed to handle and generate multiple types of data beyond text, including **images, audio, and video**. For example, GPT-4o can process and generate text, audio, and images.
+- **Reasoning Abilities**: Newer models are being trained to perform complex reasoning tasks by generating step-by-step solutions before providing final answers, akin to human problem-solving. Techniques like "Chain-of-Thought Prompting" encourage this methodical approach.
+- **Efficiency and Smaller Models**: There's a growing trend towards developing smaller, more efficient LLMs (e.g., "mini" versions of GPT models) that can perform complex tasks with fewer resources, making them suitable for deployment on everyday consumer hardware like smartphones. **Knowledge distillation** is one method used to transfer knowledge from large models to smaller ones.
+- **Agentic AI**: LLMs are being integrated into **AI agents**, providing them with the ability to interact autonomously with dynamic environments, recall past behaviours, plan future actions, and use external tools (like APIs and databases). The Model Context Protocol (MCP) is an emerging open standard that simplifies how AI agents connect and interact with these external services, providing context and enabling tool usage.
+- **Increased Context Windows and External Memory**: Developments aim to increase the amount of information an LLM can process in a single prompt (context window), with some models now supporting up to 1 million tokens. Additionally, LLMs can be given **long-term memory** as an external tool, often via vector databases, to retrieve and use information beyond their immediate conversation history.
+- **Open Source Ecosystem**: The open-source LLM landscape is thriving, offering increased transparency into models' architecture and training data, and fostering community contributions that allow for fine-tuning and diverse applications.

docs/section1/How LLMs work/_category_.json

@@ -0,0 +1,7 @@
+{
+  "position": 3,
+  "link": {
+    "type": "generated-index",
+    "description": "5 minutes to learn the most important Docusaurus concepts."
+  }
+}