Added 2 sections; added logo

docs/intro.md

@@ -45,3 +45,14 @@ The `cd` command changes the directory you're working with. In order to work wit
 The `npm run start` command builds your website locally and serves it through a development server, ready for you to view at http://localhost:3000/.
 
 Open `docs/intro.md` (this page) and edit some lines: the site **reloads automatically** and displays your changes.
+
+<iframe
+  width="100%"
+  height="450"
+  src="https://www.youtube.com/embed/osKyvYJ3PRM"
+  title="YouTube video player"
+  frameBorder="0"
+  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
+  allowFullScreen
+  style={{ maxWidth: 800, margin: "0 auto", display: "block" }}
+/>

docs/section1/AI vs AGI/AI vs AGI - notebookLM.md

@@ -0,0 +1,64 @@
+# Artificial General Intelligence: Scope, Research, and Challenges
+
+The core difference between Artificial Intelligence (AI) and Artificial General Intelligence (AGI) lies in their scope and capabilities. **AI, in its current form, refers to systems designed to perform specific tasks** by mimicking aspects of human intelligence, such as pattern recognition, decision-making, and language processing. These systems are often called "narrow AI" or "weak AI" because they are highly specialized and excel only in the defined areas they were trained for, lacking broader cognitive abilities. For example, an AI model trained for image recognition cannot build websites, nor can an AI algorithm used for object classification perform natural language processing. Current AI systems typically require substantial training before they can handle related tasks within the same domain.
+
+In contrast, **Artificial General Intelligence (AGI) is a theoretical form of AI that would possess human-like intelligence across a wide range of tasks**, with the ability to understand, learn, and apply knowledge similarly to a human being. An AGI system could perform tasks it was not explicitly trained for, showcasing autonomous self-control, a reasonable degree of self-understanding, and the ability to learn new skills. It represents the fundamental goal of AI development: the artificial replication of human intelligence in a machine or software.
+
+Here's a detailed elaboration on AGI:
+
+### What is Artificial General Intelligence (AGI)?
+
+AGI is a hypothetical stage in the development of machine learning where an AI system can match or exceed the cognitive abilities of human beings across _any_ task. Unlike narrow AI, which is confined to specific domains, **AGI would have the capacity for abstract thinking, reasoning, and adaptability**. It could solve complex problems in settings and contexts that were not taught to it at the time of its creation. The pursuit of AGI involves developing both a framework to understand intelligence in machines and the models capable of satisfying that framework.
+
+**Key Traits of AGI:**
+
+- **Generalization Ability**: AGI could transfer knowledge and skills learned in one domain to another, adapting effectively to new and unforeseen situations. Humans can apply knowledge from one domain to another, for instance, applying educational theories to game design; current AI models are limited in this aspect.
+- **Self-Teaching**: An AGI system would be able to self-teach and solve problems it was never trained for, without manual intervention.
+- **Common Sense Knowledge**: AGI would possess a vast repository of knowledge about the world, including facts, relationships, and social norms, enabling it to reason and make decisions based on this common understanding.
+- **Autonomous Self-Control**: AGI systems would exhibit a significant degree of autonomous self-control and self-understanding.
+- **Human-like Performance**: AGI aims for human-level performance on cognitive tasks, which implies a flexible and general resourcefulness and reliability.
+
+The challenge of achieving AGI is both philosophical and technological. Philosophically, it requires formal definitions of "intelligence" and how it manifests in AI. Technologically, it demands AI models with unprecedented sophistication and versatility, along with the immense computing power to sustain them.
+
+### Theoretical Approaches to AGI Research
+
+Achieving AGI requires a broader spectrum of technologies, data, and interconnectivity than what powers current AI models. AI experts have proposed several methods to drive AGI research:
+
+- **Symbolic Approach**: This assumes AGI can be developed by representing human thoughts with expanding logic networks, symbolising physical objects with if-else logic. However, this approach struggles to replicate subtle, lower-level cognitive abilities like perception.
+- **Close Emulation of the Human Brain**: Given that the human brain is the only known system capable of general intelligence, one approach is to near-perfectly emulate it in software. While artificial neural networks superficially mimic basic brain mechanisms, the actual workings of the brain are far more complex than current deep learning models, and our understanding of the brain's machinations is still limited.
+- **Novel Model Architecture**: This approach posits that AGI requires a new type of artificial intelligence, distinct from both brain emulation and current narrow AI architectures. An example is the proposal for "Objective-Driven AI Systems" with "world models" that learn more like animals and children.
+- **Integrative Approach**: This is the focus of most current real-world initiatives, attempting to combine the progress made in various narrow AI tools, such as Large Language Models (LLMs), image models, and reinforcement learning agents. Current multimodal models are seen as intermediate steps in this direction, often using a central "agent" model (like an LLM) to navigate decision-making and delegate sub-tasks to specialist models.
+
+### Technologies Driving AGI Research
+
+While AGI remains a distant goal, its research is encouraged by emerging technological developments:
+
+- **Deep Learning**: A subset of machine learning that trains neural networks with multiple hidden layers to extract complex relationships from raw data. Deep learning forms the foundation for most modern AI applications and helps systems understand text, audio, images, and video.
+- **Generative AI**: A subset of deep learning where AI systems produce unique and realistic content (text, audio, visuals, code) from learned knowledge. Models like Large Language Models (LLMs) are examples of generative AI, trained on massive datasets to respond to human queries in a human-like manner.
+- **Natural Language Processing (NLP)**: A branch of AI that enables computer systems to understand, interpret, and generate human language. NLP systems use computational linguistics and machine learning to process language data and understand contextual relationships.
+- **Computer Vision**: This technology allows systems to extract, analyze, and comprehend spatial information from visual data. Deep learning enables computer vision systems to automate large-scale object recognition, classification, and other image-processing tasks, crucial for applications like self-driving cars.
+- **Robotics**: Robotics is an engineering discipline focused on building mechanical systems that perform physical maneuvers. In AGI research, robotics is pivotal for introducing sensory perception and physical manipulation capabilities that AGI systems would require, such as a robotic arm learning to peel an orange like a human.
+
+### Challenges in AGI Research
+
+Computer scientists face several significant challenges in developing AGI:
+
+- **Making Connections Across Domains**: Current AI models are limited to their specific domains and cannot easily transfer knowledge or experience between different fields like humans can.
+- **Emotional Intelligence**: Deep learning models hint at possibilities but have yet to demonstrate authentic creativity or emotional thinking, which are crucial aspects of human intelligence. Humans respond to conversations based on emotional sensing, whereas NLP models rely solely on linguistic datasets and patterns.
+- **Sensory Perception**: AGI would require AI systems to physically interact with the external environment and perceive the world as humans do (differentiating shapes, colors, taste, smell, and sound), which existing technologies currently struggle to do accurately.
+- **Defining Intelligence**: There is no universal consensus within the academic community on what precisely qualifies as AGI or how to best achieve it, making the goal ambiguous. Proposed definitions include passing the Turing Test, possessing consciousness ("Strong AI"), analogies to the human brain, human-level performance on cognitive tasks, the ability to learn new tasks, performing economically valuable work, and demonstrating flexible and general capabilities.
+- **Computational Resources and Data**: Achieving AGI would require an unprecedented level of computational power and vast amounts of data. Training current complex deep learning models is already compute-intensive, time-consuming, and expensive.
+
+### Potential Use Cases of AGI
+
+If developed, AGI could bring numerous benefits to society:
+
+- **Solving Complex Problems**: AGI could tackle problems currently beyond human capabilities, revolutionizing fields like healthcare and climate change mitigation.
+- **Enhanced Productivity and Efficiency**: AGI could significantly enhance productivity and efficiency across various industries through advanced automation and optimization, freeing humans for more creative tasks.
+- **Revolutionizing Healthcare**: AGI holds potential for improved diagnosis, treatment planning, and drug discovery.
+- **Personalized Learning**: AGI systems could tailor educational experiences, making learning more accessible and effective.
+- **Enhanced Safety**: AGI-controlled systems, such as self-driving vehicles, could improve safety and reduce accidents.
+- **Increased Convenience**: AGI-powered virtual assistants and chatbots could provide round-the-clock support and assistance.
+- **Innovation and Creativity**: AGI could foster unprecedented levels of innovation and creativity, leading to significant technological and societal progress.
+
+Currently, true AGI does not exist and remains largely a theoretical concept and a long-term research goal. While some researchers have argued that advanced LLMs might represent an early, incomplete version of AGI, this is still a subject of ongoing debate.

docs/section1/AI vs AGI/_category_.json

@@ -1,5 +1,4 @@
 {
-  "label": "Tutorial - Basics",
   "position": 2,
   "link": {
     "type": "generated-index",

docs/section1/Common Terminology/AI model training.md

@@ -0,0 +1,44 @@
+# AI Model Training: Principles and Paradigms
+
+In artificial intelligence (AI), **model training** refers to the **process of "teaching" a machine learning (ML) model to optimize its performance** on a training dataset of sample tasks relevant to its intended use cases. This is the most critical step in the lifecycle of AI models.
+
+Here's a breakdown of what model training means in AI:
+
+- **Purpose and Goal**
+    
+    - The primary goal of model training is to enable the model to **learn patterns and correlations** from data so it can make accurate predictions or draw conclusions on new, unseen data.
+    - In mathematical terms, the objective is to **minimise a "loss function"** which quantifies the error between the model's outputs and the desired outcomes on training tasks. For reinforcement learning, the goal is reversed: parameters are optimised to **maximise a "reward function"**.
+    - When the model's error falls below a predefined threshold, it is considered "trained".
+- **How it Works**
+    
+    - AI training fundamentally involves **feeding AI models large datasets**. These datasets can be structured or unstructured, labeled or unlabeled, depending on the type of model and training paradigm.
+    - During this process, the model **adjusts its internal parameters** (e.g., weights and biases in its algorithms or neural network nodes) to produce more accurate outputs. This iterative process is often called "model fitting".
+    - **Backpropagation** is a common technique used in deep learning to calculate how each node in a neural network contributes to the overall loss, allowing for parameter optimisation.
+    - Developers may need to **fine-tune** models after initial training by providing additional data and correcting outputs to improve accuracy or adapt to new use cases. This is also known as "pretraining" followed by fine-tuning.
+- **Models vs. Algorithms**
+    
+    - While often used interchangeably, **algorithms are the procedures** (mathematical logic or pseudocode) used to generate predictions, whereas **models are the outcome** of optimising an algorithm's parameters to improve its performance on a specific dataset.
+    - Two models can use the same underlying algorithm but have different parameter values because they were trained on different data.
+- **Types of Model Training** AI models are typically categorised into three primary machine learning paradigms, each with unique use cases and training processes:
+    
+    - **Supervised Learning:**
+        - Used when the model is trained to predict a "correct" output for a given input.
+        - It relies on **labeled training data** (where inputs are paired with desired outputs, also known as "ground truth" or "supervisory signals").
+        - Example: Training an algorithm to recognise cat pictures by feeding it images explicitly labeled as "cats".
+        - This is the dominant form of training for deep learning models and underpins most state-of-the-art AI today.
+    - **Unsupervised Learning:**
+        - Used when the model is trained to discern intrinsic patterns and correlations in data **without predefined outputs or labels**.
+        - The algorithm categorises data into groups based on attributes it discovers.
+        - Example: A program looking through online sales data to identify different types of clients without being told what those types are.
+    - **Reinforcement Learning:**
+        - A "learn by doing" approach where an "agent" learns to perform a task through **trial and error**.
+        - It involves an agent evaluating its environment and taking actions that maximise a **reward function** (positive feedback for good performance, negative for poor).
+        - Example: Teaching a robotic hand to pick up a ball, or a chess program learning which moves increase the probability of winning.
+    - **Other types** include semi-supervised learning (combining labeled and unlabeled data) and self-supervised learning (where supervisory signals are derived from the structure of unlabeled data itself).
+- **Computational Resources and Data Quality**
+    
+    - Training AI models, especially large ones like Large Language Models (LLMs), can be **very expensive in terms of compute power** and time, often requiring thousands of clustered GPUs and weeks of processing. This is generally considered a one-time expense.
+    - The **quality and quantity of training data** are critical to the model's effectiveness and accuracy; poor or biased data will lead to flawed models.
+    - **Overfitting**, where a model memorises training data but cannot generalise well to new data, is a challenge that cross-validation helps to mitigate.
+
+In summary, model training is the foundational phase where an AI system develops its "knowledge" by repeatedly processing data and refining its internal structure until it can effectively perform tasks by drawing conclusions from new inputs.

docs/section1/Common Terminology/Embeddings.md

@@ -0,0 +1,38 @@
+# The Essence of Embeddings in AI and ML
+
+In artificial intelligence (AI) and machine learning (ML), **embeddings are numerical representations of data, such as text, images, or audio, transformed into dense, continuous vectors within a lower-dimensional space**. They are designed to be consumed by machine learning models and semantic search algorithms.
+
+Here's a detailed breakdown of what embeddings mean and how they function:
+
+- **Purpose and Goal**
+    
+    - The primary goal of embeddings is to **capture semantic relationships and patterns within data**. This means that data points with similar meanings or contexts are positioned closer together in this vector space, while dissimilar items are farther apart.
+    - They **translate various types of objects into a mathematical form** that AI models can easily process, as most machine learning algorithms are incapable of directly processing plain text or raw unstructured data.
+    - Embeddings are foundational for AI because they enable machine learning models to **find similar objects** and understand the relationships between them.
+- **How They Work**
+    
+    - Technically, an embedding is a **vector**, which is an array of numbers that defines a point in a multi-dimensional space. Each number in the vector indicates the object's position along a specified dimension.
+    - The **process of creating these vectors is called "embedding"** and typically uses deep learning, particularly neural networks. Neural networks, inspired by the human brain, have multiple layers, and the creation of embeddings often occurs in a hidden layer before further processing.
+    - During training, the neural network learns to automatically create this embedding layer, determining which dimensions to include and how words or data points relate to each other without explicit human definition.
+    - The **distance and direction between these vectors encode the degree of similarity** between words or objects. Common metrics for determining similarity include cosine similarity and Euclidean distance.
+    - **Word embeddings**, for instance, represent words as fixed-sized dense vectors of real numbers, capturing semantic relationships and contextual meanings based on their usage patterns in a large body of text. Contextual embeddings, used in transformer models, allow the representation of a word to change based on its surrounding context, capturing more nuanced meanings.
+- **Why They Are Used (Benefits)**
+    
+    - **Semantic Representation:** Embeddings capture semantic relationships, allowing models to understand meaning and generalize better.
+    - **Dimensionality Reduction:** They transform high-dimensional data (like text or images) into lower-dimensional representations, making them computationally more efficient.
+    - **Improved Generalization:** Models learn meaningful representations, which helps them perform well on new, unseen examples, even with limited labeled data.
+    - **Efficient Training:** Embedding layers in neural networks facilitate backpropagation and optimization, making training more efficient.
+    - **Faster and Cost-Effective Inference:** Once created, embeddings can be stored in vector databases. This significantly speeds up query responses and reduces computational costs, as the model doesn't need to re-process the entire dataset for every new query.
+    - **Handling Unstructured Data:** Embeddings are crucial for converting unstructured data (text, audio, images) into a numerical format suitable for ML models.
+- **Types of Objects That Can Be Embedded** Embeddings are versatile and can be applied to a wide range of data types:
+    
+    - **Words:** Word embeddings capture semantic relationships and contextual meanings. Popular models include Word2Vec, GloVe, and transformer-based embeddings from models like BERT and GPT.
+    - **Text:** Extending word embeddings, text embeddings represent entire sentences, paragraphs, or documents.
+    - **Images:** Image embeddings capture visual features and semantic information, useful for tasks like image classification and object detection.
+    - **Audio:** Audio embeddings capture features and characteristics of audio data for analysis and similarity metrics in applications like speech recognition.
+    - **Graphs:** Graph embeddings are essential for tasks in complex networks like social network analysis and fraud detection.
+- **Role in Large Language Models (LLMs)**
+    
+    - For LLMs, such as those used for ChatGPT, embeddings are taken a step further where **the context of every word, and the meanings of entire sentences, paragraphs, and articles, are converted into embeddings**.
+    - LLMs use embeddings to understand how different words relate to each other and the overall context of the language. This allows them to generate more coherent and contextually relevant responses.
+    - **Vector databases** are critical for LLMs, serving as a long-term memory to store and efficiently retrieve these high-dimensional embeddings. This enables capabilities like **Retrieval-Augmented Generation (RAG)**, where LLMs can access external knowledge bases to ground their responses in up-to-date and specific information.

docs/section1/Common Terminology/Inference.md

@@ -0,0 +1,63 @@
+# AI Inference: From Training to Action
+
+In artificial intelligence (AI), **inference** is the process where a **trained machine learning (ML) model** uses its learned knowledge to **draw conclusions or make predictions from new, unseen data**. This is distinct from the training phase, as inference involves the model **applying what it has learned** to make decisions or generate outputs **without requiring examples of the exact desired result**. Essentially, inference is the **AI model actively functioning or "in action"**.
+
+For example, a self-driving car recognizing a stop sign on a road it has never encountered before is performing inference. Similarly, a machine learning model trained on historical sports performance can make predictions about a player's future performance before they are signed to a contract, which is another instance of inference.
+
+### Inference vs. Training
+
+It is crucial to understand that machine learning inference and training are distinct phases within any machine learning project.
+
+- **Training**: This is the **initial phase** where an AI model learns patterns and correlations from large datasets, which can be structured or unstructured, labeled or unlabeled. Training often involves a process of trial and error, or showing the model examples of desired inputs and outputs. The model adjusts its internal parameters to optimize performance and minimise errors between its predictions and actual outcomes. Training is typically a **computationally intensive and expensive process**, but it is generally considered a one-time expense.
+- **Inference**: This process **follows AI training**. Once a model is properly trained and fine-tuned, it applies that training to new data to make additional inferences. The **better trained and fine-tuned a model is, the better its inferences will be**, though they are never guaranteed to be perfect. Inference is an **ongoing process** and can also be very expensive in terms of compute power, as the model constantly applies its training to new data.
+
+Understanding the difference between these two phases is important for allocating computational resources effectively. Using a pre-trained model for inference often requires fewer resources and can offer a competitive advantage in terms of time to market compared to training a model from scratch.
+
+### How Machine Learning Inference Works
+
+To deploy a machine learning inference environment, three main components are needed in addition to the trained ML model, which is typically software code implementing a mathematical algorithm:
+
+1. **Data Sources**: These systems capture **live data** from the mechanism that generates it.
+2. **Host System**: This system accepts data from the data sources and inputs it into the machine learning model.
+3. **Data Destinations**: This is where the host system delivers the output or score from the ML model. For instance, if a model calculates a fraud score, the destination applications might send an "approve" or "decline" message back to the purchase site.
+
+### The Role of Embeddings in Inference
+
+A key factor in how AI models, especially Large Language Models (LLMs), perform inference is their use of **embeddings**.
+
+- **Vector Embeddings** are numerical representations that convert various types of data—such as words, audio, or images—into **arrays of numbers (vectors)** that ML models can process. These vectors often have hundreds or thousands of dimensions, with each dimension representing a learned feature or attribute of the data.
+- **Semantic Proximity**: Embeddings capture the **semantic relationships and patterns** in the data. This means that semantically similar data points (e.g., synonyms or related concepts) are mapped to vectors that are close together in a high-dimensional space, while dissimilar items are farther apart. For example, the words "car" and "vehicle" have similar meanings and their vector representations will capture this semantic similarity.
+- **Querying for Inference**: When a user queries or prompts an AI model, the query itself is converted into a vector embedding. The vector database then calculates the distances between this query vector and the stored vectors to quickly find the most similar results, enabling fast and **context-aware retrieval**. This process, known as **similarity search**, allows searches based on the meaning or context of a query, rather than just keywords, leading to more precise and relevant results.
+
+Embeddings are foundational for AI because they enable machine learning models to understand relationships between words and other objects, not just exact matches. They are the backbone of generative AI applications like ChatGPT.
+
+### Use Cases for AI Inference
+
+Almost any real-world application of AI relies on inference. Some common examples include:
+
+- **Large Language Models (LLMs)**: LLMs can parse and interpret texts they have never seen before, leveraging their training to generate coherent and contextually relevant responses to user prompts. This includes tasks like answering questions, summarizing texts, translating languages, and generating creative writing or code.
+- **Predictive Analytics**: Once trained on past data, models can make predictions based on incoming data, such as forecasting market performance in finance or identifying trends in business intelligence.
+- **Email Security**: ML models can recognise spam emails or business email compromise attacks and make inferences about incoming messages to block malicious ones.
+- **Driverless Cars**: Inference is crucial for autonomous vehicles to recognise objects like stop signs and navigate safely.
+- **Research**: AI inference can be used to draw conclusions from scientific and medical data. For instance, deep learning models are used in medical imaging to support specialists in analysing images and assessing more images in less time.
+- **Conversational AI and Chatbots**: Inference enables chatbots and virtual assistants to understand user intent and respond appropriately by efficiently accessing and parsing relevant knowledge bases.
+- **Recommendation Engines**: These systems suggest products, movies, or articles by representing user preferences and item attributes as vectors and finding similar items based on vector similarity.
+- **Computer Vision**: This technology allows systems to extract, analyse, and comprehend spatial information from visual data to recognise objects, classify images, and make recommendations or take actions based on visual inputs.
+- **Retrieval-Augmented Generation (RAG)**: A key use case where LLMs retrieve relevant, up-to-date, and domain-specific information from external data sources (e.g., knowledge bases, databases) and integrate it into their responses, helping to overcome limitations like outdated knowledge or hallucinations.
+
+### Specialized Inference Techniques
+
+Beyond general inference, machine learning includes more specialised methods:
+
+- **Bayesian Inference**: This technique allows algorithms to make predictions by **updating their prior knowledge based on new evidence** using Bayes' theorem. It's more intuitive as it mimics how humans update their beliefs with new information. Applications include credit card fraud detection, medical diagnosis, image processing (noise removal), and speech processing (emotion recognition).
+- **Causal Inference**: This statistical approach is used in AI and machine learning to understand **cause-and-effect relationships** between attributes, rather than just correlations. It helps identify underlying causal relationships for more accurate predictions and can help mitigate bias by identifying its root causes in data. However, it requires large, high-quality data and strong statistical and domain expertise, and can be challenged by confounding variables.
+
+### Challenges and Considerations
+
+While inference is essential for making ML models useful in real-life applications, there are challenges:
+
+- **Computational Intensity**: Inference is an ongoing process that can be **very expensive in terms of compute power**, especially for large-scale deployments.
+- **Data Quality**: The effectiveness of inference heavily relies on the quality of the input data; poor source content can lead to inaccurate responses.
+- **Approximate Results**: Vector databases, often used for inference, frequently provide **approximate results**, which might not be suitable for applications requiring higher accuracy [internal knowledge, not directly in sources].
+- **Interpretability**: Some deep learning architectures, which underpin many modern AI systems, are considered "black boxes," making it difficult to fully understand why they produce certain results. This "lack of theory" surrounding some methods means confirmations are often empirical rather than theoretical.
+- **Bias and Unintended Outcomes**: AI models are trained on human-generated data, which can naturally contain flaws and biases. If biased information is fed into a machine learning program, the program will learn to replicate and perpetuate discrimination. This highlights the need for careful vetting of training data and human-centred AI practices.

docs/section1/Common Terminology/MCP.md

@@ -0,0 +1,68 @@
+# Model Context Protocol: AI's Universal Connector
+
+The **Model Context Protocol (MCP)** is an **open standard** that standardises how Artificial Intelligence (AI) applications, particularly **Large Language Models (LLMs)**, connect and interact with external tools, data sources, and systems. Its main goal is to solve the complexity and fragmentation that arises from integrating multiple AI models with a wide range of external services by providing a **universal, modular, and context-aware interface**.
+
+MCP is often compared to a **USB-C port for AI**, because it standardises connections between AI applications, LLMs, and external data sources, similar to how USB-C standardises how various peripherals connect to a laptop. This analogy highlights the adaptability MCP provides for connecting AI with different resources.
+
+### Why MCP Matters
+
+Generative AI models, including LLMs, are powerful but often **constrained by their isolation from external data**. LLMs are limited to the knowledge they acquired during their training, which can become outdated or lack specific domain information. Without external tools, an LLM cannot access real-time information. This can lead to generic or even incorrect responses, a phenomenon sometimes called "hallucinations".
+
+MCP addresses these challenges by:
+
+- **Providing access to specific and fresh information**. It enables AI systems, particularly **AI agents**, to retrieve and incorporate current, specific, and real-world data from various external sources such as databases, documentation, and APIs.
+- **Improving relevance, accuracy, and quality** of LLM responses by "grounding" them in up-to-date information.
+- **Reducing integration complexity**. Instead of needing custom integrations for every model-tool combination, MCP allows tool creators to build a single MCP server per tool and app developers to build a single MCP client per app, streamlining the process. It unifies LLMs and external services to communicate efficiently, offering a "plug-and-play" tool usage model.
+- **Supporting persistent, context-aware AI**. It enables stateful, multi-turn, and collaborative interactions, overcoming the limitations of stateless, one-off prompt-response models.
+- **Enabling modularity and reusability**. This makes it easier to integrate, debug, and extend AI capabilities across diverse workflows and systems.
+- **Enhancing security and control**. It facilitates secure, role-based access to data and tools and helps reduce hallucinations by grounding model outputs in real data.
+
+### Architecture and Components
+
+MCP uses a **client-host-server architecture**.
+
+| Component      | Role                                                                                                                     |
+| :------------- | :----------------------------------------------------------------------------------------------------------------------- |
+| **MCP Host**   | The application the user interacts with (e.g., Claude Desktop, IDEs, custom agents).                                     |
+| **MCP Client** | Lives within the host, managing a dedicated connection to one MCP server.                                                |
+| **MCP Server** | Lightweight external program that exposes specific tools, resources, and prompts to the AI model via a standardized API. |
+
+
+- **MCP Host**: This is the application that the user interacts with. It receives user requests and seeks access to context through the MCP. Examples include integrated development environments (IDEs) like Cursor or Claude Desktop apps. The host component contains the orchestration logic and can connect each client to a server.
+- **MCP Client**: The client exists within the host and manages a dedicated connection to one MCP server. It converts user requests into a structured format that the protocol can process. Multiple clients can exist with a single MCP host, but each client has a 1:1 relationship with an MCP server. Clients also handle session management, interruptions, timeouts, reconnections, parse responses, perform error handling, and ensure responses are relevant and appropriate.
+- **MCP Server**: This is a lightweight external program that exposes specific capabilities (tools, resources, and prompts) to the AI model via a standardised API. MCP servers act as middleware between AI models and their operational environment, parsing incoming context, storing relevant memory, and ensuring outputs align with the intended task. They can connect to both internal and external resources and tools. Examples of MCP server integrations include Slack, GitHub, Git, Docker, or web search.
+
+### MCP Server Details and Capabilities
+
+MCP servers expose capabilities through three main primitives:
+
+- **Tools**: These are **discrete actions or functions** that the AI model can call, such as APIs or calculators. For example, a weather service might expose a "get weather" tool, or a calendar service might expose a "create event" tool. When an LLM uses an MCP client to invoke a tool, the MCP server executes the underlying function. Tools can perform side effects or fetch data via an API request.
+- **Resources**: These are **read-only data items or documents** that the server can provide. Clients can retrieve these on demand. Examples include text files, database schemas, or file contents. Resources return data but do not execute actionable computations.
+- **Prompts**: These are **pre-defined templates** providing suggested prompts for optimal tool or resource usage.
+
+MCP servers also maintain a **context memory layer** that stores session, tool, and user-wise interactions, enabling persistent memory across sessions and collaborative workflows. They also define a **context schema**, typically in JSON, for representing contextual data to ensure interoperability and scalability.
+
+### Communication and Transport Layer
+
+Communication between MCP clients and servers occurs over a **JSON-RPC 2.0 session**. This transport layer is responsible for two-way message conversion, converting MCP protocol messages into JSON-RPC format for client-to-server communication, and back again for server-to-client messages.
+
+Two main transport methods are used:
+
+- **Standard Input/Output (stdio)**: Best for integrating **local resources** due to simple input/output transmission, used for lightweight, synchronous messaging.
+- **Server-Sent Events (SSE)**: Best for integrating **remote resources**, handling multiple asynchronous, event-driven server calls. HTTP POST requests are used for client-to-server messages, with SSE for the reverse.
+
+### Relationship with LLMs and AI Agents
+
+LLMs, without external tools, are skilled at subsequent text prediction, basic Q&A within their training data, sentiment analysis, and language translation. MCP provides LLMs with access to real-time, external information, which is crucial for AI agents to perform tasks, solve problems, and adapt to changing circumstances. AI agents use LLMs along with available tools, allowing the model to determine appropriate tool use, adapt to a changing environment, and form synthesised conclusions based on tool output.
+
+While traditional **Application Programming Interfaces (APIs)** also allow systems to access functionality or data, MCP is **purpose-built for LLM integration**. A key difference is that MCP supports **dynamic discovery**. An MCP client can ask an MCP server what functions and data it offers, allowing the LLM application to adapt to available capabilities without requiring a developer to update the client if the server's API changes. In contrast, traditional REST APIs don't typically expose such a runtime discovery mechanism. Furthermore, every MCP server follows the same protocol and patterns, regardless of the service or data it connects to, whereas each traditional API is unique.
+
+Interestingly, **many MCP servers actually use traditional APIs to do their work**, acting as a wrapper that translates between the MCP format and the underlying service's native interface. This means MCP and APIs are not adversaries but rather **layers in an AI stack**, with MCP providing a more AI-friendly interface on top.
+
+### Integration with RAG
+
+MCP can **supplement Retrieval-Augmented Generation (RAG)**. Instead of just providing a retriever to search a vector store or knowledge base, MCP can connect to a vector database through a server action. Searching the database as a tool, rather than passing the retriever in every LLM invocation, allows for a more strategic use of the tool and enables further tool-calling upon data retrieval.
+
+### Development and Community
+
+Anthropic introduced MCP in 2024 and is committed to building it as a **collaborative, open-source project and ecosystem**. Developers can start building and testing MCP connectors today, and pre-built MCP servers for popular enterprise systems like Google Drive, Slack, GitHub, Git, Postgres, and Puppeteer are available to help developers get started.

docs/section1/Common Terminology/RAG.md

@@ -0,0 +1,35 @@
+# Retrieval-Augmented Generation: Enhancing LLMs with External Knowledge
+
+**Retrieval-Augmented Generation (RAG)** is an Artificial Intelligence (AI) approach that **combines information retrieval with language generation** to create more accurate and contextually relevant outputs. It enhances Large Language Models (LLMs) by integrating them with external data sources.
+
+Here's a breakdown of what RAG means:
+
+- **Purpose and Goal**
+    
+    - The primary goal of RAG is to **enhance the accuracy and relevance of LLM responses** by "grounding" their outputs in up-to-date and specific real-world data.
+    - LLMs, by themselves, are limited to the knowledge contained in their training data, which can become outdated or lack domain-specific information. They can also sometimes generate plausible-sounding but incorrect information, a phenomenon known as "hallucinations," or provide vague and generic responses.
+    - RAG addresses these limitations by allowing LLMs to retrieve and incorporate current and specific information from various external sources, such as databases, documentation, and APIs. This process helps minimize contradictions and inconsistencies in the generated text, improving its quality and reliability.
+- **How RAG Works** RAG operates by following a series of steps to enhance generative AI outputs:
+    
+    1. **Data Collection:** First, all necessary external data for the application is gathered. This could include user manuals, product databases, or FAQs.
+    2. **Data Chunking:** The gathered data is then broken down into smaller, more manageable pieces or "chunks".
+    3. **Embedding and Indexing:** Each of these text chunks is converted into **numerical representations called vector embeddings**. Embeddings are dense, continuous vectors in a lower-dimensional space that capture semantic relationships, meaning similar data points are located closer together in this space. These embeddings are then stored and indexed in a **vector database**.
+    4. **Retrieval:** When a user poses a query, it is also converted into a vector embedding. The vector database then uses algorithms to quickly find the most similar stored vectors (i.e., relevant data chunks) to the query vector. This process is known as **vector search** or **semantic search**, as it looks for meaning rather than just keywords.
+    5. **Generation:** The retrieved text chunks, along with the original user query, are then fed into the LLM. The LLM uses this augmented context to generate a coherent, precise, and contextually relevant response. This effectively instructs the generative model to prioritise the retrieved information before giving its response, making it less likely to hallucinate or rely solely on its internal training data.
+- **Benefits of Using RAG**
+    
+    - **Access to Fresh Information:** RAG overcomes the limitation of LLMs being confined to their pre-trained data by providing them with up-to-date information.
+    - **Factual Grounding:** It helps LLMs remain factually accurate, even when dealing with complex or specific domain knowledge.
+    - **Improved Relevance, Accuracy, and Quality:** By grounding the LLM in relevant external knowledge, RAG significantly enhances the quality of generated text, making it more reliable and useful.
+    - **Efficient Inference:** RAG allows for faster and more efficient inference because the LLM doesn't need to re-process an entire massive dataset for every query. Once embeddings are created and stored, retrieval is quick.
+- **Role in LLMs and AI Agents**
+    
+    - RAG is a key technique for extending LLMs and providing them with "external memory" beyond their core training dataset.
+    - It is particularly useful for conversational AI, recommendation engines, and chatbots, allowing them to provide real-time, contextual answers.
+    - RAG also enhances the capabilities of **AI agents** by providing them with dynamic access to external context, which is crucial for agents to perform tasks, solve problems, and adapt to changing circumstances.
+- **Challenges**
+    
+    - Implementing RAG systems can involve **integration complexity**, especially when dealing with multiple external data sources in various formats.
+    - The **quality of the input data is critical**; poor or biased source content will lead to inaccurate responses. Human oversight is crucial for data curation to ensure reliability.
+
+In essence, RAG acts as a bridge, combining the broad language capabilities of LLMs with specific, accurate, and current information from external sources, making AI systems more reliable and effective for real-world applications.

docs/section1/Common Terminology/_category_.json

@@ -0,0 +1,7 @@
+{
+  "position": 4,
+  "link": {
+    "type": "generated-index",
+    "description": "5 minutes to learn the most important Docusaurus concepts."
+  }
+}

docs/section1/How LLMs work/Difference between structured and unstructured data.md

@@ -0,0 +1,75 @@
+# Structured vs. Unstructured Data: A Core Comparison
+
+**Structured data** and **unstructured data** are two fundamental categories used to classify data based on their format, organisation, and the methods required for their processing and analysis. They differ significantly across various aspects, including their structure, nature, storage, processing, and the expertise needed to handle them.
+
+### Structured Data
+
+**Structured data** is characterised by its **highly organised, predefined, and tabular format**, typically arranged in rows and columns with a fixed schema. It is often described as **quantitative data**, which means it commonly contains numbers or textual elements that can be easily counted, measured, and searched. This inherent organisation makes it straightforward to group, sort, and analyse based on specific attributes or characteristics.
+
+Examples of structured data include:
+
+- **Spreadsheets** (such as Google Sheets and Microsoft Office Excel files).
+- **Relational databases** (like those used in CRM systems and financial records).
+- **Log files**.
+- Specific details such as **employee names, contacts, ZIP codes, addresses, and credit card numbers**.
+- Information within **airline reservation systems**, including passenger names and flight numbers.
+- IoT sensor readings (e.g., temperature or humidity) when recorded in a tabular format.
+
+Structured data is typically stored in **relational databases** (often called SQL databases) or **data warehouses**. These storage systems have rigid schemas, where the configuration of columns and data types is strictly defined. Data is easily queried and manipulated using **Structured Query Language (SQL)**, which was developed by IBM in the 1970s.
+
+Handling structured data is generally accessible to business users with standard tools, as it does not require in-depth data science knowledge. There is an abundance of mature analytics tools for structured data, including Business Intelligence (BI) tools (like Tableau and Power BI) and Online Analytical Processing (OLAP) systems (such as Oracle Essbase and Apache Kylin). The structure for structured data is defined **before storage**, a concept known as "schema-on-write".
+
+**Advantages** of structured data include its ease of use and access, its suitability for machine learning applications, and the wide array of available tools. However, its predefined nature limits its flexibility and usability, and changes to its schema can be time and resource-intensive.
+
+### Unstructured Data
+
+In contrast, **unstructured data** has **no predefined structure or schema** and is stored in its native or raw format. It is often classified as **qualitative data**, containing subjective information that is harder to quantify. This type of data is rapidly growing, comprising **80-90% of all enterprise-generated data**.
+
+Examples of unstructured data include:
+
+- **Emails and text documents**.
+- **Social media posts** (including pictures and text within posts and comments).
+- **Multimedia files** (such as image, video, and audio files).
+- **Call transcripts** and message text files.
+- **Sensor data** from Internet of Things (IoT) devices.
+
+Unstructured data is best managed in **non-relational (NoSQL) databases** or **data lakes**, which are designed to handle massive amounts of raw data in any format. Storing unstructured data typically requires more space due to its volume and varied formats.
+
+Analysing unstructured data is more complex and requires **advanced techniques and specialised tools**. These include:
+
+- **Machine learning (ML) libraries** (e.g., TensorFlow, PyTorch, Hugging Face).
+- **Natural Language Processing (NLP) tools** (e.g., Azure Cognitive Services, spaCy) for text-based data.
+- **Data search and indexing tools** (e.g., Elasticsearch).
+- **Big data processing tools** (e.g., Apache Spark, Apache Hadoop).
+- **Visual data analysis tools** for image recognition (e.g., Google Cloud Vision AI platform).
+
+Extracting insights from unstructured data often involves preprocessing steps, which can include data stacking or mining to detect patterns and interactions. Meaningful analysis typically requires **data science expertise**. The structure for unstructured data is applied **when it is accessed or analysed**, known as "schema-on-read".
+
+**Benefits** of unstructured data include its flexibility, fast accumulation rates, and relatively easy and cheap storage. However, its challenges include the need for specialised expertise and tools, as well as potential data cleanliness issues due to its volume and non-uniform structure.
+
+### Fundamental Differences
+
+The fundamental differences between structured and unstructured data can be summarised as follows:
+
+- **Format and Schema**:
+    
+    - **Structured Data**: Adheres to a **strict, predefined data model** or schema, presented in a tabular format. This is "schema-on-write".
+    - **Unstructured Data**: Has **no predefined format** or schema and is stored in its native or raw form. Its structure is applied "on-read".
+- **Nature**:
+    
+    - **Structured Data**: Generally **quantitative**, easily measurable and searchable.
+    - **Unstructured Data**: Predominantly **qualitative**, often subjective and harder to quantify.
+- **Storage Systems**:
+    
+    - **Structured Data**: Resides in **relational databases** and **data warehouses** with rigid schemas.
+    - **Unstructured Data**: Stored in **file systems, content management systems, data lakes**, or **non-relational (NoSQL) databases**.
+- **Processing and Analysis**:
+    
+    - **Structured Data**: Easily queried and analysed using **SQL** and traditional business intelligence tools.
+    - **Unstructured Data**: Requires **advanced techniques** such as **machine learning (ML) and natural language processing (NLP)** to extract insights.
+- **Accessibility and Expertise Required**:
+    
+    - **Structured Data**: **Accessible and easy to use** by general business users.
+    - **Unstructured Data**: Requires **specialised skills and tools**, often involving data science expertise for meaningful analysis.
+
+It is worth noting that the lines between these two categories have become somewhat **blurred**. Many datasets today are considered **semi-structured**, incorporating certain markers or metadata (such as tags in a photograph or headers in an email) that enable some level of organization, while the core content remains unstructured. Both structured and unstructured data are essential in modern analytics and are leveraged across various industries and applications.

docs/section1/How LLMs work/Tokenization.md

@@ -0,0 +1,65 @@
+# Tokenization: Building Blocks for Language Models
+
+**Tokenization** in language models is the process of **converting a sequence of text into smaller, discrete components known as tokens**. These tokens serve as the **basic building blocks that a language model processes and understands**. The primary reason for tokenization is to help machines understand human language by breaking it down into manageable units that are easier to analyze.
+
+### Why Tokenization is Important
+
+Tokenization is a **crucial first step in training Large Language Models (LLMs)** and is foundational for artificial intelligence (AI). It matters for several reasons:
+
+- **Makes text machine-readable** Machine learning algorithms, including LLMs and deep learning models, are generally incapable of processing plain text in its raw form; they require numbers as input. Tokenization converts raw text into a numerical format, allowing models to process it mathematically.
+- **Facilitates pattern recognition** By breaking text into tokens, algorithms can more easily identify patterns, relationships, and context within language. This pattern recognition is essential for machines to understand and respond to human input.
+- **Defines the model's vocabulary** The set of unique tokens an LLM is trained on forms its vocabulary. This vocabulary determines what kinds of things a model is trained to recognise and produce.
+
+### How Tokenization Works in Practice
+
+The process typically involves several steps:
+
+1. **Conversion to Numerical Format** Text is first transformed into a numerical format that the machine learning model can understand.
+2. **Vocabulary Creation and ID Assignment** The LLM is trained on a **vocabulary of unique tokens** derived from large text corpora. As training continues, any new tokens encountered are added to this vocabulary and assigned a **unique numerical ID**. For example, "I heard a dog bark loudly at a cat" could be tokenized and assigned IDs like `I (1), heard (2), a (3), dog (4), bark (5), loudly (6), at (7), cat (8)`, resulting in a sequence like ``.
+3. **Creation of Embeddings** After tokenization, **multi-valued numeric vectors, known as embeddings, are used to represent these semantic relationships**. An embedding is assigned to each token based on how commonly it's used together with, or in similar contexts to, other tokens. Words with similar meanings or contexts are expected to have similar vector representations, with distance and direction between vectors encoding their similarity. For LLMs, the **context of every word becomes an embedding**, in addition to the word itself, allowing the meanings of entire sentences, paragraphs, and articles to be searched and analysed. Embeddings are foundational for AI as they enable machine learning models to find similar objects.
+4. **Output Generation** During output generation, the model predicts a vector value for the next token in the sequence. It then selects the most probable token from its vocabulary based on this vector value. This is an **iterative process**: the predicted token is appended to the sequence so far and used as input for the next iteration, building the final output one token at a time.
+
+It is important to note that the **correlation between words and tokens is complex**. Sometimes a single word is broken into multiple tokens, or the same word might be tokenized differently depending on context or surrounding words. Spaces, newline characters, and punctuation may or may not be included in tokens.
+
+### Common Tokenization Methods
+
+Tokenization methods vary based on the granularity of the text breakdown and the specific requirements of the task. Common methods include:
+
+- **Word tokenization**: Splits text into individual words, typically based on delimiters like spaces. This is most effective for languages with clear word boundaries like English.
+- **Character tokenization**: Splits text into individual characters. This method is useful for languages lacking clear word boundaries or for tasks requiring granular analysis, such as spelling correction.
+- **Subword tokenization**: Breaks text into units larger than a single character but smaller than a full word. This approach balances the need for a manageable vocabulary size with the ability to handle unknown words or complex syntax. Examples include **Byte-Pair Encoding (BPE)**, which merges frequent pairs of characters or bytes to form tokens until a set vocabulary size is reached, and **WordPiece**. GPT models developed by OpenAI use Byte-Pair Encoding.
+
+The choice of token size affects model efficiency and language coverage:
+
+- **Smaller tokens** (character or subword tokenization) enable the model to handle a wider range of inputs, including unknown words, typos, or complex syntax. This might also reduce vocabulary size, requiring fewer memory resources. However, a given text is broken into more tokens, requiring additional computational resources and reducing the maximum input/output size for a fixed token limit.
+- **Larger tokens** (word tokenization) result in fewer tokens per text, requiring fewer computational resources. However, this might increase vocabulary size and limit the model's ability to handle unknown words or typos.
+
+### Tokenization Challenges
+
+The nuances of human language present unique challenges for tokenization:
+
+- **Ambiguity**: Language is inherently ambiguous, and tokenization can lead to different interpretations of a sentence.
+- **Languages without clear boundaries**: Some languages, like Chinese, Japanese, or Thai, lack clear spaces between words, making word tokenization more complex. Multilingual tokenization models like XLM-R and mBERT use subword tokenization to handle these languages.
+- **Special characters**: Handling email addresses, URLs, or special symbols can be tricky, requiring advanced models to incorporate rules for consistent processing.
+- **Morphology**: While subword tokenization aims to capture frequent substrings, it does not guarantee decomposition into linguistic morphemes (smallest meaningful units). Research is mixed on whether the lack of morphology in subword tokenizers impairs model performance, particularly for languages with complex morphology like German.
+- **Understanding character-level information**: It is not trivial for a model to "look inside" a token to know its constituent characters (e.g., that "dog" contains "d", "o", "g"). Though state-of-the-art LLMs can perform well on some character-level tasks, they may struggle with others like counting character occurrences. Some variability in how a root word is tokenized can actually improve predictions about contained characters.
+- **Token limits and context window**: LLMs have limitations on the maximum number of input and output tokens, often combined into a "context window". Larger context windows significantly increase computational complexity quadratically, making it costly and inefficient. This also affects the maximum length of text that can be provided as input or generated as output.
+- **Computational Cost**: Tokenization and subsequent processing require significant computational resources. Training large language models involves processing petabytes of data.
+
+### Use Cases for Tokenization
+
+Tokenization forms the backbone for a wide array of AI applications by enabling machines to process and understand vast amounts of text data:
+
+- **Search engines**: Tokenization dissects user queries to sift through documents and retrieve relevant results.
+- **Machine translation**: It segments sentences in a source language for translation and reconstruction in the target language, retaining original context.
+- **Speech recognition**: Spoken words are converted into text and then tokenized, allowing systems like Siri or Alexa to process and act upon commands.
+- **Sentiment analysis**: Tokenization extracts insights from user content like reviews to determine sentiments (positive, neutral, negative).
+- **Chatbots and virtual assistants**: Tokenization enables chatbots to understand user intent and respond appropriately.
+- **Named Entity Recognition (NER)**: It identifies and classifies entities like names, places, or organisations in text.
+- **Text classification**: LLMs can classify text based on similar meanings or sentiments for tasks like spam detection or moderating user-generated content.
+
+### Tools and Future Directions
+
+Prominent tools for tokenization include NLTK (Natural Language Toolkit), spaCy, and Hugging Face Transformers. Hugging Face Transformers provides fast tokenizers built using Rust and supports various subword tokenization methods like BPE and WordPiece. BERT tokenizer, for example, excels in context-aware tokenization.
+
+Future research aims to explore more efficient model architectures that can handle longer sequences with lower computational complexity, and design new self-supervised pre-training tasks. There's also ongoing interest in understanding how morphological structure impacts language model performance and whether incorporating such information can lead to better generalisation, particularly for languages with complex morphology.

docs/section1/How LLMs work/Vector Databases.md

@@ -0,0 +1,70 @@
+# Vector Databases: AI's Semantic Memory Powerhouse
+
+**Vector databases** are specialized systems designed to **store, manage, and index high-dimensional vector data**, which are often used as embeddings representing various forms of data like text, images, or audio. Unlike traditional relational databases that use rows and columns, vector databases represent data points as arrays of numbers, enabling them to handle unstructured datasets more effectively. They are increasingly popular because they provide the speed and performance necessary for generative Artificial Intelligence (AI) applications.
+
+### How Vector Databases Work
+
+The fundamental principle behind vector databases involves **converting data into numerical representations called vectors or embeddings** and then enabling **similarity searches** based on their proximity in a multi-dimensional space.
+
+1. **Vectors and Embeddings**:
+    
+    - A **vector** is an array of numbers that defines a point in a dimensional space. Each number in the array indicates the object's position along a specified dimension. These can represent complex objects like words, images, videos, or audio. For example, the word "cat" might be represented as `[0.2, -0.4, 0.7]` in a 3-dimensional space. In practice, vectors can have hundreds or thousands of dimensions to capture intricate relationships.
+    - **Embeddings** are the output of a deep learning process that creates these vectors. They are numerical representations of data points that convert various data types into arrays of numbers that machine learning (ML) models can process. The creation of embeddings is often a hidden layer within a neural network, automatically transforming inputs into these multi-dimensional representations. This allows models to find similar objects because words with similar meanings or contexts are positioned close to each other in the vector space.
+2. **Embedding Generation**:
+    
+    - Machine learning models, particularly deep learning models and neural networks, are trained on large datasets to convert raw, unstructured data (like text or images) into these numerical vectors. For example, a word embedding model like Word2Vec uses a neural network to create word embeddings, where the context of a word helps infer its meaning and position in the vector space. In Large Language Models (LLMs), the context of every word, and even entire sentences, paragraphs, or articles, can become an embedding.
+3. **Storage and Indexing**:
+    
+    - Vector databases store these generated embeddings, along with their metadata (e.g., title, description).
+    - To accelerate searches within high-dimensional data spaces, vector databases create **indexes** on these embeddings. Common indexing algorithms include Hierarchical Navigable Small World (HNSW), Locality-Sensitive Hashing (LSH), and Product Quantization (PQ), which map vectors to new data structures for faster similarity searches.
+4. **Similarity Search**:
+    
+    - When a user queries or prompts an AI model, the query itself is converted into a vector embedding.
+    - The vector database then calculates the **distance** between this query vector and the stored vectors using various similarity metrics, such as **cosine similarity** or **Euclidean distance**. The closer the embeddings are, the more similar the original data points are considered.
+    - The database returns the most similar vectors or "nearest neighbours" to the query vector. This process is known as an **Approximate Nearest Neighbour (ANN) search**, which balances speed with accuracy.
+
+### Why Vector Databases are Used (Advantages)
+
+Vector databases offer several benefits that make them crucial for modern AI applications:
+
+- **Speed and Performance**: They enable rapid searches through large datasets (millions or billions of data points) by using indexing techniques and efficient distance-calculating algorithms, returning results within milliseconds.
+- **Scalability**: They can store and manage massive amounts of unstructured data by scaling horizontally with additional nodes, maintaining performance even as data volumes increase.
+- **Lower Cost of Ownership**: By enabling faster data retrieval, vector databases can speed up the training of foundation models and reduce the need to re-process entire datasets for every query, leading to cost savings.
+- **Data Management**: They often include built-in features for updating and inserting new unstructured data.
+- **Flexibility**: Vector databases are designed to handle the complexity of multi-dimensional data, including images, videos, and various text types. They can be customised to meet specific business and AI requirements.
+- **Semantic Understanding**: Crucially, they allow machine learning models to understand the relationships between words and other objects, not just exact matches, which is foundational for AI.
+- **Long-Term Memory**: For machine learning models, especially LLMs, vector databases provide a form of "long-term memory," allowing the model to remember previous inputs and context without retraining or re-running the entire dataset through the model each time.
+
+### Use Cases for Vector Databases
+
+Vector databases form the backbone for a wide range of AI applications by enabling machines to process and understand vast amounts of text and other data.
+
+- **Retrieval-Augmented Generation (RAG)**: This is a key use case, where LLMs use vector databases to retrieve relevant, up-to-date, and domain-specific information from external data sources (e.g., product manuals, FAQs, databases, web pages) and incorporate it into their responses. RAG helps LLMs overcome limitations like outdated knowledge, hallucinations (generating false information), and generic responses.
+- **Conversational AI and Chatbots**: They enable chatbots and virtual assistants to understand user intent and respond appropriately by efficiently accessing and parsing relevant knowledge bases, providing real-time, contextual answers.
+- **Recommendation Engines**: By representing user preferences and item attributes as vectors, these systems can suggest similar products, movies, or articles based on vector similarity, enhancing user experience and increasing engagement.
+- **Vector Search (Semantic Search)**: This technique allows searches based on the meaning or context of a query, rather than just keywords, leading to more precise and relevant results. It includes finding similar images, audio, video, or text data.
+- **Anomaly Detection**: By identifying data points that are significantly different from existing clusters, vector databases can be used to detect unusual behaviour, such as fraudulent transactions or network anomalies.
+- **Text Classification and Sentiment Analysis**: LLMs can classify text based on similar meanings or sentiments, aiding in spam detection or analysing user-generated content.
+- **Drug Discovery**: Graph neural networks and generative neural networks, powered by underlying vector representations, have been used to predict properties of molecules and produce novel biomolecules.
+
+### Challenges and Considerations
+
+While powerful, implementing vector databases and RAG systems can pose challenges:
+
+- **Integration Complexity**: Integrating retrieval systems with LLMs can be difficult, especially with multiple external data sources in varying formats. Ensuring data consistency and uniform embeddings across sources is crucial.
+- **Data Quality**: The effectiveness of a RAG system heavily relies on the quality of the input data; poor source content will lead to inaccurate responses. Human oversight and expert curation are often necessary.
+- **Computational Intensity**: Tasks like generating embeddings, comparing text meanings, and real-time data retrieval are computationally intensive, which can slow down the system as data size increases. This necessitates distributed computing and robust hardware infrastructure.
+- **Approximate Results**: Vector databases often provide approximate results, meaning applications requiring higher accuracy might need different database types at the cost of slower processing.
+- **Query Limitations**: Vector indexes are well-suited for fact-based queries, but may not be optimal for tasks like summarization, where an LLM needs to process all possible contexts on a topic rather than retrieving specific facts.
+
+### Tools and Integration
+
+Prominent tools and services for working with vector databases include:
+
+- **Cloudflare Workers AI and Vectorize**: Cloudflare offers Vectorize, a globally distributed vector database, combined with Workers AI, to make it faster and more affordable to generate and query embeddings for AI applications without extensive backend infrastructure.
+- **LlamaIndex**: This data framework helps developers manage the flow of information from external data sources to LLMs for RAG-powered applications.
+- **LangChain**: An open-source orchestration framework that provides integrations for various embedding methods and vector stores.
+- **Open-source solutions**: Examples include Weaviate or Milvus, which offer RESTful APIs and programming language support.
+- **Database extensions**: Some traditional databases, like PostgreSQL, offer extensions (e.g., pgvector) for vector similarity search.
+
+The integration of vector databases with LLMs is further supported by protocols like the **Model Context Protocol (MCP)**, which standardises how AI applications connect and interact with external tools and data sources, similar to how USB-C standardises hardware connections. MCP allows AI agents to be context-aware and use tools and resources by providing a standardised way to retrieve external context, such as documents, knowledge base entries, and database records.

docs/section1/How LLMs work/What is an LLM.md

@@ -0,0 +1,58 @@
+# Large Language Models: Core Concepts and Applications
+
+**Large Language Models (LLMs)** are a type of artificial intelligence (AI) system designed to **understand, process, and generate human language at scale**. They are built using **deep learning techniques**, specifically a neural network architecture known as a **transformer model**. LLMs are trained on **vast amounts of text data**, which is why they are called "large". This training enables them to respond to human queries with text, audio, or visuals that naturally resemble human creations.
+
+### How Large Language Models Work
+
+LLMs operate through a complex interplay of data, architecture, and training processes:
+
+- **Underlying Technology**: At their core, LLMs are built on **machine learning**, a subset of AI that allows systems to learn from data without explicit programming. They leverage **deep learning**, which uses multi-layered artificial neural networks (often referred to as "deep neural networks") to simulate the complex decision-making power of the human brain. These networks comprise an input layer, an output layer, and multiple hidden layers in between, sometimes hundreds or thousands.
+- **Data Collection and Pre-processing**:
+    - **Massive Datasets**: LLMs are trained on enormous datasets, sometimes tens of gigabytes or even **petabytes of text data**, including books, articles, conversations, and web content. For instance, GPT-3 was pre-trained on approximately 45 terabytes of data. The quality of this data significantly impacts the LLM's learning.
+    - **Tokenization**: The first step in processing text is **tokenization**, where raw text is broken down into smaller units called **tokens**. These tokens can be words, parts of words (subwords), characters, or punctuation marks. For example, GPT models use a type of subword tokenization called Byte-Pair Encoding (BPE). Tokens allow the model to identify patterns, relationships, and context within language. Roughly, a token is about three-quarters of a word.
+    - **Embeddings**: After tokenization, LLMs turn these tokens into **embedding vectors**. These are multi-dimensional numerical representations of words or phrases that capture their semantic meaning and relationships. In this high-dimensional space, semantically similar words or concepts are mapped to vectors that are closer together.
+- **Training and Learning**:
+    - **Transformer Architecture**: LLMs primarily use the **transformer architecture**, which enables them to process entire sequences of text in parallel. This architecture, particularly through its **self-attention mechanisms**, allows the model to understand the context of each word by considering its relation to every other word in a sentence. This makes them highly effective at comprehending vague or newly contextualized human language.
+    - **Prediction and Self-learning**: During training, the model learns to predict the next token in a sequence. It achieves this through **self-supervised learning**, where it infers "pseudolabels" from unlabeled data, such as predicting masked words in text samples. The model iteratively adjusts its internal **parameters** (billions or even trillions of adjustable weights and biases) to minimize prediction errors.
+    - **Computational Cost**: This training process is **compute-intensive, time-consuming, and expensive**, often requiring thousands of clustered graphics processing units (GPUs) and weeks of processing, costing millions of dollars.
+- **Fine-tuning**: After initial pre-training on vast datasets to gain general language capabilities, LLMs are often **fine-tuned** on smaller, more specific datasets or through other methods like **Reinforcement Learning with Human Feedback (RLHF)**. Fine-tuning refines the model's understanding to perform specific tasks more accurately.
+
+### Applications of Large Language Models
+
+LLMs are incredibly versatile and are used in a wide range of applications across industries:
+
+- **Generative AI**: Creating new and realistic content such as long-form text (essays, poems, articles), high-quality images, realistic video or audio, and code in response to user prompts.
+- **Natural Language Processing (NLP)**: Tasks include text summarisation, language translation, question answering, sentiment analysis, named entity recognition, and text classification.
+- **Chatbots and Virtual Assistants**: Powering intelligent conversational agents for customer service and general information.
+- **Software Development**: Generating and reviewing code, writing functions, and assisting with programming tasks.
+- **Recommendation Systems**: Providing personalised suggestions based on semantic understanding of text data.
+- **Business Intelligence**: Analysing large volumes of business reports and market research documents to derive meaningful insights.
+
+### Advantages and Limitations of LLMs
+
+**Advantages**:
+
+- **Flexibility and Adaptability**: LLMs can perform a wide variety of tasks and respond to unpredictable queries in natural human language, unlike traditional programs that require specific syntax.
+- **Accessibility**: Pre-trained LLMs, especially open-source foundation models, make advanced AI capabilities more accessible to developers and organisations without extensive computational resources or labelled data.
+- **Reduced Development Time**: Using pre-trained models as a starting point can significantly reduce the time and resources required to develop AI applications from scratch.
+
+**Limitations**:
+
+- **Hallucinations**: LLMs can confidently generate information that is plausible-sounding but factually incorrect or nonsensical, a phenomenon known as "hallucination".
+- **Limited/Outdated Knowledge**: Their knowledge is restricted to their training data, which can become outdated or lack domain-specific information, leading to generic or inaccurate responses for specific queries.
+- **Bias**: LLMs are susceptible to inheriting and amplifying biases present in their training data, potentially leading to skewed representations or unfair treatment based on factors like race, gender, or political viewpoint.
+- **Computational Intensity**: Training and fine-tuning LLMs require immense processing power and energy, raising concerns about economic and environmental sustainability.
+- **Black-Box Nature**: The decision-making processes of complex LLMs can be difficult to interpret, posing interpretability challenges.
+- **Context Window Limits**: LLMs have a maximum limit on the number of tokens they can process in their input and generate as output, which can restrict their ability to handle very long documents or conversations.
+
+### Current Advancements and Future Directions
+
+The field of LLMs is rapidly evolving, with several key advancements:
+
+- **Retrieval-Augmented Generation (RAG)**: This technique enhances LLMs by integrating them with external, up-to-date knowledge bases, typically stored in **vector databases**. RAG allows LLMs to retrieve relevant information and combine it with their generative capabilities, significantly reducing hallucinations and providing more accurate, grounded, and contextually relevant responses.
+- **Multimodality**: LLMs are increasingly being developed to handle and generate multiple types of data beyond text, including **images, audio, and video**. For example, GPT-4o can process and generate text, audio, and images.
+- **Reasoning Abilities**: Newer models are being trained to perform complex reasoning tasks by generating step-by-step solutions before providing final answers, akin to human problem-solving. Techniques like "Chain-of-Thought Prompting" encourage this methodical approach.
+- **Efficiency and Smaller Models**: There's a growing trend towards developing smaller, more efficient LLMs (e.g., "mini" versions of GPT models) that can perform complex tasks with fewer resources, making them suitable for deployment on everyday consumer hardware like smartphones. **Knowledge distillation** is one method used to transfer knowledge from large models to smaller ones.
+- **Agentic AI**: LLMs are being integrated into **AI agents**, providing them with the ability to interact autonomously with dynamic environments, recall past behaviours, plan future actions, and use external tools (like APIs and databases). The Model Context Protocol (MCP) is an emerging open standard that simplifies how AI agents connect and interact with these external services, providing context and enabling tool usage.
+- **Increased Context Windows and External Memory**: Developments aim to increase the amount of information an LLM can process in a single prompt (context window), with some models now supporting up to 1 million tokens. Additionally, LLMs can be given **long-term memory** as an external tool, often via vector databases, to retrieve and use information beyond their immediate conversation history.
+- **Open Source Ecosystem**: The open-source LLM landscape is thriving, offering increased transparency into models' architecture and training data, and fostering community contributions that allow for fine-tuning and diverse applications.

docs/section1/How LLMs work/_category_.json

@@ -0,0 +1,7 @@
+{
+  "position": 3,
+  "link": {
+    "type": "generated-index",
+    "description": "5 minutes to learn the most important Docusaurus concepts."
+  }
+}

docs/section1/Intoduction/Define and describe deep learning.md

@@ -0,0 +1,76 @@
+# Deep Learning: Principles, Applications, and Challenges
+
+**Deep learning** is a **subset of machine learning (ML)** that utilizes **multilayered artificial neural networks** (also known as deep neural networks) to learn autonomously from large amounts of data, identify complex patterns, and make decisions or predictions with minimal human intervention. It forms the foundation for most artificial intelligence (AI) applications in use today. The term "deep" refers to the presence of multiple layers—typically **three or more hidden layers**, but often hundreds or even thousands—between the input and output layers in its neural network architecture.
+
+### Relationship within AI
+
+Artificial intelligence is the broadest concept, aiming to enable machines to perform tasks that typically require human intelligence, such as reasoning, learning, and problem-solving. **Machine learning is a subset of AI** that allows machines to learn from data and improve performance on specific tasks without explicit programming. Deep learning, in turn, is a **specialised type of machine learning** that employs multi-layered neural networks. This creates a clear hierarchy: Deep Learning ⊂ Machine Learning ⊂ Artificial Intelligence.
+
+### How Deep Learning Works
+
+Deep learning models are "trained" by being exposed to vast amounts of data. This process involves feeding the model input data, allowing it to make predictions, and then **iteratively adjusting its internal parameters** (such as weights and biases) to minimise errors between its predictions and the actual outcomes. This iterative refinement leads to continuous improvement in accuracy over time.
+
+A key difference from traditional machine learning is deep learning's ability to perform **automatic feature extraction**. Instead of requiring humans to manually select, transform, and create relevant input features from raw data, deep learning models learn these features hierarchically through their layers directly from the data itself. This reduces the need for human intervention in feature engineering.
+
+The learning process within a deep neural network often involves:
+
+- **Forward Propagation**: Data moves through the network's layers from the input layer to the output layer, where a prediction or classification is made.
+- **Backpropagation**: Algorithms (like gradient descent) calculate the errors in these predictions and then adjust the weights and biases of the network by moving backwards through the layers to refine the model. This process aims to minimise a "loss function" (or maximise a "reward function" in reinforcement learning), which quantifies the model's error.
+
+### Data and Hardware Requirements
+
+Deep learning models are particularly powerful for analysing **large amounts of unstructured data** like images, audio, and text. However, this power comes with significant requirements:
+
+- **Data Volume**: Deep learning models need **very large datasets** for effective training, often millions of data points. The more data, the better the model's performance.
+- **Computational Power**: Training complex deep neural networks demands **significant computing resources**, typically requiring **high-performance graphical processing units (GPUs)** or specialised processors like Google's Tensor Processing Units (TPUs) due to the massive parallel computations involved.
+
+### Types of Deep Learning Models/Architectures
+
+Deep learning utilises various neural network architectures, each suited to different tasks:
+
+- **Feedforward Neural Networks (FFNNs)**: One of the oldest types, where data flows in one direction through layers of artificial neurons to produce an output. Modern FFNNs are often "deep feedforward" with multiple hidden layers.
+- **Recurrent Neural Networks (RNNs)**: Designed for **sequential data** (like time series or language), RNNs have "memory" of previous layers' outputs, allowing them to process sequences by considering prior inputs.
+- **Long Short-Term Memory (LSTM)**: An advanced form of RNN particularly effective for language modelling and speech recognition, capable of "remembering" information across many layers due to "memory cells".
+- **Convolutional Neural Networks (CNNs)**: Most commonly used in **computer vision and image classification**, CNNs employ distinct layers (convolutional, pooling, fully connected) to filter and identify features within images and videos, starting from simple elements like colours and edges, and progressing to more complex shapes.
+- **Autoencoders and Variational Autoencoders (VAEs)**: Among the first models widely used for generating realistic images and speech, they compress data into dense representations and then reconstruct it, enabling anomaly detection, classification, and generative modelling.
+- **Generative Adversarial Networks (GANs)**: Involve two competing neural networks—a "generator" that creates content and a "discriminator" that evaluates its realism—to improve the quality of generated output.
+- **Transformers**: A powerful non-recurrent architecture that has revolutionised language models. They use a **self-attention mechanism** to process entire sequences in parallel, allowing them to understand the context of each word by considering its relation to every other word. Transformers are foundational for modern Large Language Models (LLMs).
+
+### Benefits of Deep Learning
+
+Deep learning offers several advantages:
+
+- **Automation**: Automates tasks and workflows that previously required human intervention.
+- **Actionable Insights**: Identifies patterns and trends in massive datasets that humans might miss.
+- **Predictive Capabilities**: Enables accurate predictions and forecasting across various domains.
+- **Efficiency and Productivity**: Streamlines processes and improves productivity by handling complex tasks at tremendous scale.
+- **Personalisation**: Provides personalised experiences by learning from individual interactions.
+- **Accessibility**: Pre-trained models and modular platforms (like Hugging Face) make deep learning more accessible, reducing development time and resource requirements.
+
+### Applications of Deep Learning
+
+Deep learning is widely applied across numerous industries and technologies:
+
+- **Speech Recognition**: Powers virtual assistants (e.g., Siri, Alexa) by converting spoken language to text.
+- **Image Recognition and Computer Vision**: Used in facial recognition, self-driving cars, and medical image analysis, allowing systems to identify and categorise aspects of images and videos.
+- **Natural Language Processing (NLP)**: Enables understanding, interpretation, and generation of human language, powering chatbots, summarisers, and translation services.
+- **Generative AI**: Creates original content (text, images, video, code) by learning from patterns in existing content.
+- **Recommendation Systems**: Suggests products, content, or services based on past behaviour and preferences.
+- **Fraud Detection**: Analyses transaction patterns to flag anomalies indicative of fraud.
+- **Healthcare**: Aids in diagnostics and medical image analysis, and accelerates drug development.
+- **Materials Science**: Used to discover new materials and accelerate material innovation.
+- **Finance**: Used for algorithmic trading and assessing business risks.
+- **Robotics**: Enables robots to learn new tasks through observation.
+- **Weather Prediction**: Models trained on historical data can predict weather conditions globally with high precision and speed.
+
+### Challenges and Limitations
+
+Despite its advancements, deep learning faces several challenges:
+
+- **Data Requirements**: It requires **vast amounts of high-quality, accurate, and unbiased training data**. "Garbage in, garbage out" is a significant concern.
+- **Computational Resources**: Training complex models demands **significant computing power** and considerable energy.
+- **Interpretability ("Black Box" Problem)**: Many complex deep learning models are often considered "black boxes," making it **difficult to understand _why_ they make specific decisions** or how their inner workings lead to a particular output.
+- **Bias and Unintended Outcomes**: Models trained on human-generated data can **inherit and perpetuate human biases and existing inequities**, leading to discriminatory outputs.
+- **Accuracy Limitations**: While highly accurate, deep learning models are not always perfect. For critical tasks, very high accuracy is required, and current models are estimated to perform at about 95% of human accuracy.
+- **Security Vulnerabilities**: Artificial neural networks are vulnerable to adversarial attacks, where subtle changes to inputs can trick the model into misclassifying data. Data poisoning can also prevent models from achieving mastery.
+- **Scalability**: While deep learning enables large-scale machine learning, optimising large models with many layers and parameters can add complexity and long training times.

docs/section1/Intoduction/Define and describe machine learning.md

@@ -0,0 +1,82 @@
+# Machine Learning: Concepts, Types, and Applications
+
+**Machine learning (ML)** is a **subset of artificial intelligence (AI)** that focuses on enabling computers and machines to **learn from data and improve their performance on specific tasks without being explicitly programmed for every scenario**. Instead of following fixed, explicit instructions, ML systems use algorithms to **analyse large volumes of data, identify patterns, and then make predictions or decisions based on what they learn**.
+
+### How Machine Learning Works
+
+At its essence, the core principle of AI, including ML, revolves around data. ML models are "trained" by being exposed to vast amounts of data. During this **training phase**, algorithms—which are sets of rules or instructions—guide the AI's analysis and decision-making. The machine "learns" by adjusting its **internal parameters**, such as weights and biases in mathematical functions, to **minimise errors** between its predictions and the actual outcomes. This **iterative process** of evaluation and optimisation allows the model to continuously refine its understanding and become more accurate over time.
+
+Once a model is properly trained, it can then perform **inference**, which is the process of using the trained model to draw conclusions, make predictions, or generate outputs from **new, unseen data**. For example, a machine learning model trained on historical sports performance data could predict a player's future performance, or a self-driving car could recognise a stop sign on an unfamiliar road.
+
+### Relationship to Artificial Intelligence and Deep Learning
+
+ML is a **core discipline within the broader field of AI**. AI is the overarching concept of enabling machines to simulate human intelligence to solve problems. ML is a **specific application of AI** that allows machines to extract knowledge from data and learn autonomously.
+
+**Deep learning (DL)** is, in turn, a **subset of machine learning**. DL uses **multi-layered artificial neural networks** (also known as deep neural networks) to simulate the complex decision-making power of the human brain. The "deep" refers to the presence of three or more hidden layers between the input and output layers, which can number in the hundreds or even thousands. This architecture enables DL models to automatically learn complex patterns and hierarchical representations directly from **large amounts of raw, unstructured data** without human intervention in feature engineering. Most of the AI applications we interact with today are powered by some form of deep learning.
+
+### Types of Machine Learning
+
+Machine learning models primarily fall into three categories based on their learning approach:
+
+- **Supervised Learning**: This is the **most common type** and involves training algorithms on **labelled datasets** (structured data) where the correct output for each input is known. The model learns to map specific inputs to desired outputs. An example is feeding an algorithm pictures labelled as "cats" to train it to recognise cats. Supervised learning underpins most state-of-the-art AI models today and is primary for tasks requiring accuracy like classification or regression.
+- **Unsupervised Learning**: This approach uses **unlabelled data** (unstructured data) to discover hidden patterns, relationships, or groupings within the data without pre-defined outputs. It is particularly effective for tasks like pattern matching, descriptive modelling, and customer segmentation.
+- **Reinforcement Learning**: In this "learn by doing" model, an "agent" learns to perform a defined task through **trial and error**. It receives positive "rewards" for good performance and negative "penalties" for poor performance, continuously adjusting its actions to maximise the reward. IBM Watson's victory on _Jeopardy!_ in 2011 is a good example of reinforcement learning.
+- **Semi-supervised Learning**: This is a mixed approach that uses a **smaller labelled dataset to guide classification and feature extraction from a larger, unlabelled dataset**. It helps when there isn't enough labelled data for pure supervised learning or when labelling is too costly.
+
+### Common Machine Learning Algorithms/Models
+
+ML uses various algorithms and model architectures. Neural networks, modeled after the human brain, are a popular type of ML algorithm. Other common algorithms include:
+
+- **Linear Regression**: Predicts numerical values based on linear relationships.
+- **Logistic Regression**: Predicts for categorical response variables (e.g., "yes/no").
+- **Decision Trees**: Used for both numerical predictions and classification, represented by a branching sequence of decisions.
+- **Random Forests**: Combines results from multiple decision trees for predictions.
+- **Clustering (e.g., K-means)**: Identifies patterns in data to group them.
+- **Support Vector Machines (SVMs)**: Used for classification and regression tasks.
+- **Naive Bayes**: A probabilistic classifier based on Bayes' theorem.
+
+Deep learning further specialises in neural network architectures, including:
+
+- **Feedforward Neural Networks (FFNN)**: Data flows one way through layers.
+- **Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM)**: Designed for sequential data, maintaining "memory" of previous inputs.
+- **Convolutional Neural Networks (CNN)**: Most common for image recognition, using distinct layers to filter image parts.
+- **Generative Adversarial Networks (GAN)**: Involve two competing neural networks (generator and discriminator) to create realistic output.
+- **Transformers**: Powerful for processing sequences, foundational for modern Large Language Models (LLMs).
+
+### Benefits of Machine Learning
+
+Machine learning offers significant benefits to organisations, particularly in handling large and complex datasets:
+
+- **Automation**: Automates tasks and workflows that previously required human intervention.
+- **Actionable Insights**: Identifies patterns and trends in massive volumes of data that humans might miss, enabling data-driven decision-making.
+- **Predictive Capabilities**: Enables accurate predictions and forecasting across various domains.
+- **Efficiency and Productivity**: Streamlines processes and improves productivity.
+- **Personalisation**: Provides personalised experiences for users as models learn from individual interactions.
+- **Accessibility**: Pre-trained models and modular platforms make ML more accessible to a wider range of users and businesses.
+
+### Applications of Machine Learning
+
+ML is widely applied across numerous industries and everyday technologies:
+
+- **Speech Recognition**: Converts spoken language into text, used in virtual assistants like Siri and Alexa.
+- **Image Recognition and Computer Vision**: Identifies and categorises aspects of images and videos, used in facial recognition, self-driving cars, and medical image analysis.
+- **Natural Language Processing (NLP)**: Trains computers to understand, interpret, and generate human language, powering chatbots, summarisers, and translation services.
+- **Generative AI**: Creates original content like text, images, video, and code in response to user prompts, leveraging deep learning models.
+- **Recommendation Systems**: Suggests products, content, or services based on past behaviour and preferences.
+- **Fraud Detection**: Analyses transaction patterns to flag anomalies indicative of fraud.
+- **Predictive Maintenance**: Forecasts equipment failures using sensor data.
+- **Healthcare**: Analyses patient records, aids in diagnostics, and accelerates drug development.
+- **Cybersecurity**: Continuously monitors and analyses network traffic to detect threats.
+- **Financial Services**: Used for algorithmic trading and assessing business risks.
+- **Robotics**: Enables robots to learn new tasks through observation and feedback.
+
+### Challenges and Limitations
+
+Despite its advancements, machine learning faces several challenges:
+
+- **Data Requirements**: ML, especially deep learning, requires **vast amounts of high-quality, accurate, and unbiased training data**. "Garbage in, garbage out" is a key concern.
+- **Computational Resources**: Training complex models, particularly deep neural networks, demands **significant computing power**, often requiring high-performance GPUs and considerable energy.
+- **Interpretability ("Black Box" Problem)**: Many complex ML and deep learning models are often considered "black boxes," making it **difficult to understand _why_ they make specific decisions** or how their inner workings lead to a particular output.
+- **Bias and Unintended Outcomes**: Since models are trained by humans on human-generated data, they can **inherit and perpetuate human biases and existing inequities**, leading to discriminatory or problematic outputs.
+- **Accuracy Limitations**: While highly accurate, ML models are not always perfect, and their level of acceptable accuracy depends on the application. For critical tasks like self-driving cars or medical diagnostics, very high accuracy is required.
+- **Technological Singularity**: While not an immediate concern for most researchers, the theoretical idea of **AI surpassing human intelligence** (often termed strong AI or superintelligence) raises ethical and societal questions.

docs/section1/Intoduction/Definition of AI.md

@@ -0,0 +1,35 @@
+## The World of Artificial Intelligence: An Overview
+
+Artificial intelligence (AI) is a broad field of computer science and technology focused on creating machines and computer systems that can **simulate or replicate human intelligence**. Its fundamental goal is to enable software and machines to **reason, learn, and act in ways that would normally require human cognitive abilities**.
+
+### Core Capabilities and Functions of AI
+
+AI systems are designed to perform a variety of advanced functions, including:
+
+- **Learning from data and experience**: AI systems learn and improve through exposure to vast amounts of data, identifying patterns and relationships that humans might miss. This learning process often involves algorithms trained on various datasets.
+- **Understanding and processing natural language (NLP)**: This capability allows computers to comprehend and generate human language, improving communication interfaces and enabling sophisticated tasks like translation, sentiment analysis, and conversational AI.
+- **Recognising objects and patterns**: This includes computer vision, which enables systems to extract, analyse, and comprehend spatial information from visual data for tasks such as image recognition, object detection, and classifying images.
+- **Making recommendations and predictions**: AI can mine data to forecast specific outcomes, such as the future performance of sports players or provide recommendations in e-commerce.
+- **Automating tasks and acting independently**: AI can automate routine, repetitive, and often tedious or dangerous tasks, freeing human capital for higher-value work. This autonomous action is exemplified by self-driving cars.
+
+### Relationship with other AI Concepts
+
+AI is a broad field that encompasses many different disciplines and technologies:
+
+- **Machine Learning (ML)**: This is a subset of AI that allows machines to extract knowledge from data and learn from it autonomously without being explicitly programmed for every scenario.
+- **Deep Learning**: A further specialisation within machine learning, deep learning uses multi-layered artificial neural networks (DNNs) that mimic the structure of the human brain to process information and learn complex patterns directly from large amounts of raw data. Many of the AI applications in use today are powered by some form of deep learning.
+- Other significant subfields and applications under the AI umbrella include **robotics, expert systems, and natural language processing**.
+
+### Types of AI Based on Capability
+
+AI systems are often categorised by their level of sophistication and capability:
+
+- **Narrow AI (ANI) / Weak AI**: This is the most common type of AI currently in existence and wide use. It is specialised in performing a specific task or a limited set of tasks within pre-determined parameters. Examples include virtual assistants like Siri or Amazon Alexa, facial recognition software, and recommendation engines. Even emerging generative AI applications, while powerful, are still considered weak AI because they cannot be repurposed for other domains.
+- **General AI (AGI) / Strong AI**: This is a hypothetical stage where an AI system would possess human-level intelligence. It would be capable of understanding, learning, and applying knowledge across a wide range of tasks without manual intervention, similar to a human being. AGI remains a theoretical concept and research goal, and true AGI does not currently exist.
+- **Super AI (ASI) / Artificial Superintelligence**: This theoretical level of AI would vastly surpass human intelligence in all aspects, including scientific creativity, general wisdom, and social skills. While not a prerequisite for AGI, some narrow AI models have already demonstrated "superintelligence" in specific domains by exceeding human performance (e.g., AlphaFold in protein structure prediction, IBM Deep Blue in chess, AlphaGo in Go). However, these are still considered narrow AI as they cannot autonomously learn new tasks or expand their problem-solving capabilities beyond their narrowly defined scope.
+
+### History and Evolution of AI
+
+The concept of thinking machines dates back to ancient myths and stories. The formal field of AI research was founded at the **Dartmouth College workshop in 1956**, where the term "artificial intelligence" was coined by John McCarthy. Early research included projects like ELIZA, the first chatbot (1966). Initial optimism led to ambitious predictions, but significant challenges and limited computing power resulted in periods known as "AI winters," marked by reduced funding and interest.
+
+The field saw renewed interest with the commercial success of "expert systems" in the 1980s and the establishment of deep learning's foundations. More recent breakthroughs include IBM Watson's victory on _Jeopardy!_ (2011) and AlphaGo's defeat of a Go champion (2016). The invention of the **transformer architecture in 2017** proved particularly impactful for language tasks and has enabled the development and public release of powerful generative AI models, such as ChatGPT in 2022, initiating the current "AI boom" and bringing AI into mainstream use.

docs/section1/Intoduction/Describe the history of AI from the past to the possible future.md

@@ -0,0 +1,76 @@
+# The Evolution of Artificial Intelligence: From Myth to Modernity
+
+The history of artificial intelligence (AI) stretches back to ancient myths and philosophical concepts of thinking machines, evolving through distinct periods of development, breakthroughs, and challenges, leading to its current impactful presence and speculative future.
+
+### Ancient and Early Foundations
+
+The idea of thinking machines dates back to antiquity, with myths, stories, and rumors of artificial beings possessing intelligence or consciousness. Philosophers like Aristotle laid conceptual groundwork for formal reasoning, which is fundamental to AI. By the 19th century, concepts of artificial men and thinking machines became popular themes in fiction, as seen in Mary Shelley's _Frankenstein_ and Karel Čapek's _R.U.R._ (Rossum's Universal Robots). Early automata were also built across civilizations, believed by some to be imbued with real minds.
+
+### Birth of Modern AI (1940s–1950s)
+
+The formal field of AI research was founded at the **Dartmouth College workshop in 1956**. Here, the term **"artificial intelligence" was coined by John McCarthy**. The workshop aimed to test the assertion that "every aspect of learning or any other feature of intelligence can be so precisely described that a machine can be made to simulate it".
+
+Before this, **Alan Turing**, often referred to as the "father of computer science," published his seminal paper "Computing Machinery and Intelligence" in **1950**. In this paper, Turing proposed the **Turing Test**, a game to determine if a machine could carry on a conversation indistinguishable from a human, suggesting that if it could, it would be reasonable to say the machine was "thinking". This test remains an important benchmark for AI progress. Inspired by theories in neurology, cybernetics, and information theory, scientists began to explore the possibility of building an "electronic brain". In **1955**, Allen Newell and Herbert A. Simon created the **Logic Theorist**, considered the first running AI computer program.
+
+### Early Successes and First AI Winter (1956–1980)
+
+The period after the Dartmouth Workshop saw significant optimism among AI researchers, who predicted fully intelligent machines within 20 years. Government agencies like the Defense Advanced Research Projects Agency (DARPA) heavily funded AI research, establishing laboratories at universities like MIT, Carnegie Mellon, and Stanford.
+
+Key developments during this time included:
+
+- **Symbolic Reasoning**: Early AI programs often used search algorithms to solve problems like proving theorems or playing games. The concept of manipulating symbols as the essence of human thought gained traction.
+- **Natural Language Processing**: **Joseph Weizenbaum's ELIZA**, the first AI chatbot, was unveiled in **1966**. While rudimentary, it could carry out conversations convincingly enough to fool users into thinking they were communicating with a human.
+- **Game AI**: Programs like Arthur Samuel's checkers program (1959) demonstrated early machine learning.
+
+However, by the **1970s**, early enthusiasm faded as the technology failed to meet the ambitious predictions, leading to growing discontent and reduced funding. This period, marked by critiques like the **Lighthill report (1973)** which highlighted the "combinatorial explosion" problem, became known as the **first "AI winter" (1974–1980)**. Challenges included limited computer power, the intractability of problems, **Moravec's paradox** (where AI struggled with "unintelligent" tasks like perception that humans find easy), and the vastness of commonsense knowledge.
+
+### Boom and Second AI Winter (1980s–2000s)
+
+Despite the setbacks, AI research continued. The **1980s** saw a resurgence with the adoption of **"expert systems"** by corporations. These systems used logical rules and large databases of specialist knowledge, saving companies millions. Japan's Fifth Generation Computer Project and the U.S. Strategic Computing Initiative further fueled investment, with the AI industry growing to billions of dollars. This period also saw the **revival of neural networks**, referred to as the **"connectionist" approach**, which aimed to replicate the human brain's structure. In **1986, Geoffrey Hinton** popularized **"backpropagation,"** a crucial training technique for neural networks.
+
+However, the limitations of expert systems led to another period of reduced funding and interest, the **second "AI winter" (1987–1993)**. Many fledgling AI companies went bankrupt. Despite this, AI algorithms continued to develop and were increasingly used **"behind the scenes"** in larger systems for tasks like data mining, speech recognition, and medical diagnosis, often without being explicitly called "AI". The **1990s** also saw milestones like **IBM Deep Blue defeating world chess champion Garry Kasparov in 1997**. A new paradigm of **"intelligent agents"** emerged, defining AI as the study of systems that perceive their environment and take actions to maximize success.
+
+### Big Data, Deep Learning, and AGI (2005–2017)
+
+The early 21st century witnessed a **renaissance in AI** due to increased access to **"big data,"** cheaper and faster computers (especially **GPUs**), and advanced machine learning techniques.
+
+- **Deep Learning Revolution**: This was a breakthrough technology, using **multilayered neural networks (deep neural networks)** to simulate the complex decision-making power of the human brain. Deep learning significantly improved performance in areas like image and video processing, text analysis, and speech recognition.
+- **Notable Achievements**: **IBM Watson defeated _Jeopardy!_ champions in 2011**. In **2012, AlexNet**, developed by Alex Krizhevsky (a student of Hinton), won the ImageNet computer vision competition by efficiently using GPUs to run deep networks, marking the beginning of the "deep learning era". Google DeepMind's **AlphaGo defeated a world Go champion in 2016**.
+- **Artificial General Intelligence (AGI)**: Around the mid-2010s, concerns that mainstream AI was too focused on "narrow AI" led to renewed interest and direct research into **Artificial General Intelligence (AGI)**, the hypothetical stage where an AI system could match or exceed human cognitive abilities across any task. Companies like OpenAI and Google's DeepMind were founded with this pursuit.
+
+### Large Language Models and Current AI Boom (2017–Present)
+
+A pivotal moment came in **2017** with the invention of the **transformer architecture** by Google researchers. This architecture, which exploits an **attention mechanism**, proved particularly useful for language tasks, enabling AIs to simultaneously handle translation, text generation, and document summarization. Most of today's leading AI models, including generative AI applications, rely on this architecture.
+
+The current **"AI boom"** was largely initiated by the public release of **Large Language Models (LLMs)** based on the transformer architecture.
+
+- **ChatGPT Launch**: On **November 30, 2022, OpenAI released ChatGPT**, powered by its GPT-3 model. It rapidly gained over 100 million users in two months, becoming the fastest-growing consumer software application in history. This brought AI into mainstream public consciousness.
+- **Generative AI (Gen AI)**: LLMs are a type of generative AI that can produce unique and realistic content like text, images, video, and audio from learned knowledge. They train on massive datasets, allowing them to respond to human queries in a human-like manner.
+- **Current Capabilities**: LLMs are proficient in tasks such as summarization, text generation, creative writing, question answering, and code generation. They can interpret human language even when vague, and associate words and concepts by meaning due to vast training. While powerful, current generative AI applications are still considered **Narrow AI** because they are limited to specific domains and cannot be repurposed for others.
+
+The rapid adoption of these AI technologies has sparked intense debate regarding ethical implications, including concerns about misinformation (deepfakes) and job displacement. Calls for a pause in advanced AI development have also emerged.
+
+### Possible Future of AI (Near-term and Long-term)
+
+Experts predict AI will become a fixture in personal and business lives within the next decade (by 2034).
+
+**Near-term (2020s–2030s)**:
+
+- **Multimodal AI**: Systems will integrate text, voice, images, and video to create more intuitive interactions, closely resembling how humans communicate. This could power advanced virtual assistants that understand complex queries and provide tailored responses across various media.
+- **Democratization of AI**: User-friendly platforms, including no-code and low-code solutions, will enable non-experts to develop custom AI solutions for business, individual tasks, and creative projects. Open-source AI will also foster transparency and innovation.
+- **Smaller, More Efficient Models**: There's a shift towards developing smaller, more cost-effective models (e.g., mini GPT 4o-mini) that deliver greater precision with fewer resources, making them suitable for embedding in devices like smartphones.
+- **AI in the C-Suite**: AI decision-making and prediction modeling will advance, allowing AI systems to function as strategic business partners, offering advice and automating complex tasks for executives.
+- **Regulations and Ethics**: Ethical considerations will shape regulations, including bans on systems posing unacceptable risks, with a focus on human oversight, fundamental rights, bias, and fairness.
+- **Agentic AI**: Systems composed of specialized agents that proactively anticipate needs and make autonomous decisions will become a core part of personal and business life. These agents adapt to real-time environments, use simpler decision-making algorithms, and learn from feedback.
+- **Synthetic Data**: As human-generated data becomes scarcer, artificial datasets that mimic real-world patterns will become standard for training AI, enhancing model accuracy and promoting data diversity.
+- **Hallucination Insurance**: Companies might offer "AI hallucination insurance" to protect against incorrect or misleading AI outputs, particularly in critical sectors like finance, medicine, and law.
+
+**Long-term and "Moonshots"**:
+
+- **Artificial General Intelligence (AGI)**: While speculative, the emergence of AGI by 2034 could lead to AI systems that autonomously generate, curate, and refine their own training datasets, enabling self-improvement without human intervention. True AGI does not currently exist and remains a theoretical concept and research goal.
+- **Post-Moore Computing**: Innovations in **neuromorphic computing** (mimicking the brain) and **optical computing** (using light) aim to move beyond traditional computing limits, potentially drastically reducing the time and resources needed to train and run large AI models.
+- **Distributed "Internet of AI"**: This envisions a decentralized AI infrastructure where AI operates across multiple devices and locations, processing data locally to enhance privacy and reduce latency.
+- **Overcoming Transformer Limitations**: Research is ongoing to improve the computational efficiency of transformer models, allowing them to handle larger context windows without exponential increases in resource usage, leading to more coherent and contextually relevant responses.
+- **Beyond Binary (Bitnet Models)**: Using ternary parameters (base-3) instead of binary data could enable AI to process information more efficiently, leading to faster computations and less power consumption.
+
+The history of AI is marked by alternating periods of optimism and skepticism, but its current trajectory points toward increasing influence across all sectors of society, with ongoing breakthroughs and new challenges on the horizon.

docs/section1/Intoduction/Describe three levels of artificial intelligence.md

@@ -0,0 +1,51 @@
+# Understanding AI: Types and Capabilities
+
+Artificial intelligence (AI) can be described across **different levels or types**, reflecting its capabilities and how it processes information. The sources provide two main frameworks for categorising AI: **capability-based levels** and **functionality-based levels**.
+
+### Capability-Based Levels
+
+This framework categorises AI into three main types based on their intelligence levels relative to human capabilities:
+
+1. **Artificial Narrow Intelligence (ANI) / Weak AI**
+    
+    - **Description**: ANI, also known as weak AI, is the most common type of AI found today. These systems are **specialized in a single task or domain** and cannot perform tasks outside their pre-programmed scope. Their "intelligence" is demonstrated only within these specific areas. Even emerging generative AI applications, despite their advancements, are considered weak AI because they cannot be repurposed for other domains.
+    - **Capabilities**: ANI systems are limited to computing specifications, algorithms, and the specific tasks they are designed for. They may have limited memories and rely only on real-time data to make decisions. They require substantial training with specific datasets to work reliably with unfamiliar data within their domain.
+    - **Examples**: Facial recognition software, AI summarizers, Google Search, predictive analytics, virtual assistants like Amazon Alexa, Apple Siri, or Google Assistant, and self-driving automobiles. IBM's Deep Blue, which defeated world chess champion Garry Kasparov in 1997, is also an example of a reactive machine, a form of narrow AI.
+2. **Artificial General Intelligence (AGI) / Strong AI**
+    
+    - **Description**: AGI, also referred to as strong AI or general AI, is a **hypothetical stage** in AI development. The aim of AGI research is to create software with human-like intelligence and the **ability to self-teach**, enabling it to perform tasks it was not specifically trained or developed for. It represents the fundamental goal of AI development: the artificial replication of human intelligence in a machine or software. AGI can solve complex problems in various domains, like a human being, without manual intervention.
+    - **Key Distinctions**: Unlike current AI that requires substantial training for related tasks within the same domain (e.g., fine-tuning an LLM with medical datasets to become a medical chatbot), an AGI system could learn and handle unfamiliar tasks without additional training. Strong AI, a concept often conflated with AGI, specifically refers to an AI system demonstrating consciousness and is considered a conscious mind, rather than merely a tool. Consciousness is usually implied as either a prerequisite or consequence of general intelligence.
+    - **Challenges in Research**: Achieving AGI is a **distant goal** and poses significant challenges. These include limited computer power, the intractability of problems leading to "combinatorial explosion" (where finding optimal solutions requires extraordinary computer time), and Moravec's paradox (where AI excels at "intelligent" tasks but struggles with "unintelligent" tasks like perception). It also requires overcoming the vastness of commonsense knowledge and representing commonsense reasoning, which is difficult to formalize with precise definitions. Furthermore, current neural networks struggle to replicate authentic human creativity and emotional thinking. AGI systems would also need to physically interact with the external environment and perceive the world as humans do, requiring advancements beyond existing computer technologies.
+    - **Theoretical Approaches**: AI experts have proposed several methods for AGI research:
+        - **Symbolic approach**: Assumes AGI can be developed by representing human thoughts with expanding logic networks, using if-else logic to interpret ideas at a higher thinking level. However, it struggles with lower-level cognitive abilities like perception.
+        - **Connectionist (or emergentist) approach**: Focuses on replicating the human brain's structure with neural-network architecture, hoping to achieve human-like and low-level cognitive capabilities by mimicking how brain neurons alter transmission paths based on external stimuli. Large language models (LLMs) are an example of AI using this method.
+        - **Universalists**: While not detailed in the provided sources, this suggests a broader, more integrated approach (implied by).
+    - **Technological Drivers**: Deep learning, generative AI, natural language processing (NLP), computer vision, and robotics are considered emerging technologies driving AGI research.
+    - **Current Status**: AGI remains a theoretical concept and research goal. No known AI systems currently approach this level of sophistication. However, some researchers have argued that advanced LLMs like GPT-4 could reasonably be viewed as an early, incomplete version of an AGI system, given their ability to solve novel and difficult tasks across various domains.
+3. **Artificial Superintelligence (ASI)**
+    
+    - **Description**: ASI is a **theoretical AI system** whose capabilities vastly exceed those of human beings. It surpasses human intelligence in all aspects, including creativity, problem-solving, and emotional intelligence.
+    - **Current Status**: ASI is still largely theoretical and a topic of debate and speculation. While true general superintelligence has not been achieved, there are narrow AI models that demonstrate what could be fairly called superintelligence in their specific tasks, as they exceed human performance.
+    - **Examples of "Narrow Superintelligence"**: AlphaFold (predicts protein structures better than humans), IBM Deep Blue (defeated world chess champion Garry Kasparov in 1997), IBM Watson (defeated _Jeopardy!_ champions in 2013), and AlphaGo (considered the world's greatest Go player). However, these models have not achieved artificial "general" intelligence as they cannot autonomously learn new tasks or expand their problem-solving capabilities beyond their narrowly defined scope.
+
+### Functionality-Based Levels
+
+This framework categorises AI based on how machines process information and react to stimuli, reflecting stages of AI development:
+
+1. **Reactive Machines**
+    
+    - **Description**: This is a limited form of AI that **only reacts to different kinds of stimuli based on pre-programmed rules**. It does not use memory and, therefore, cannot learn with new data.
+    - **Examples**: IBM's Deep Blue, which beat chess champion Garry Kasparov in 1997, is an example of a reactive machine. Spam filters are also mentioned as an example.
+2. **Limited Memory**
+    
+    - **Description**: Most modern AI systems fall into this category. They can **use memory to improve over time by being trained with new data**, typically through an artificial neural network or other training model. This capability allows them to use past data for a short period to inform decisions.
+    - **Connection to Modern AI**: Deep learning, a subset of machine learning, is considered limited memory AI. This type of AI learns and improves through exposure to vast amounts of data, identifying patterns and relationships that humans may miss.
+    - **Examples**: Self-driving cars (which consider recent past observations to navigate) and chatbots are examples of limited memory AI. Recommendation engines also fit this category.
+3. **Theory of Mind**
+    
+    - **Description**: Theory of mind AI describes AI that **does not currently exist**, but research is ongoing into its possibilities. It would be able to **emulate the human mind's decision-making capabilities**, including recognising and remembering emotions and reacting in social situations as a human would.
+4. **Self-Aware**
+    
+    - **Description**: A step above theory of mind AI, self-aware AI describes a **mythical machine** that is aware of its own existence and has the intellectual and emotional capabilities of a human. Like theory of mind AI, **self-aware AI does not currently exist** and remains purely hypothetical.
+
+In summary, while most existing AI is "narrow" and operates at the reactive or limited memory levels, Artificial General Intelligence, Artificial Superintelligence, Theory of Mind AI, and Self-Aware AI remain theoretical concepts and are subjects of ongoing research and speculation.

docs/section1/Intoduction/_category_.json

@@ -0,0 +1,7 @@
+{
+  "position": 1,
+  "link": {
+    "type": "generated-index",
+    "description": "5 minutes to learn the most important Docusaurus concepts."
+  }
+}

docs/section1/Pre-Trained Models/Pre_trained Models.md

@@ -0,0 +1,30 @@
+# Pre-trained Models: Benefits and Limitations
+
+**Pre-trained models (PTMs)** are **Machine Learning (ML) models that have been previously trained on a large dataset to solve a specific task or set of tasks**. These models learn patterns, features, and representations from their training data. Often, PTMs are **deep network models** that have undergone training on vast datasets to achieve a specific task, such as identifying cancer spots in medical images or detecting suspicious behavior in video surveillance feeds.
+
+The "large-scale" aspect of PTMs like Large Language Models (LLMs) refers to their **sophisticated pre-training objectives and huge model parameters**, which allow them to **effectively capture knowledge from massive labeled and unlabeled data**. By storing this knowledge in their parameters and then being fine-tuned for specific tasks, this rich, implicitly encoded knowledge can benefit a variety of downstream tasks. The AI community widely agrees on adopting PTMs as a backbone for downstream tasks rather than building models from scratch.
+
+### Benefits of Pre-trained Models
+
+PTMs offer numerous advantages for AI development and deployment:
+
+- **Accelerated Development and Accessibility**: PTMs **significantly reduce development time and computational resources**, as they eliminate the need to train a model from scratch, which can take months. This makes machine learning more accessible to a wider range of users, including those without extensive computational resources or labeled data. Developers can use PTMs and other predefined building blocks to bring new AI solutions to market faster and more cost-effectively.
+- **Leveraging Existing Expertise**: Since PTMs are typically trained on **large datasets curated by domain experts**, developers can leverage this inherent expertise without having to create their own specialized datasets. This is particularly useful in specialized domains where labeled data is scarce or difficult to create.
+- **Improved Performance**: PTMs can **improve AI application performance**. Studies show that pre-trained LLMs can perform as well as, or even better than, models specifically trained from scratch for custom tasks. This is partly due to their training on massive datasets and well-calibrated parameters.
+- **Flexibility and Customisation**: PTMs offer **flexibility** as they can be used as they are, or **customised for new use cases through transfer learning**. Transfer learning leverages the knowledge within PTMs for new, similar tasks with only a small amount of additional labeled data. For example, a single foundational model can be fine-tuned multiple times for different use cases. Developers can **engineer prompts and tweak parameters** to control model outputs significantly.
+- **Simplified Integration**: Many PTMs, particularly LLMs, come with **easy-to-use APIs** (Application Programming Interfaces), allowing developers to access their capabilities with just a few lines of code, thereby avoiding the hidden costs associated with building and hosting models from scratch.
+- **Diverse Applications**: PTMs are transforming various industries. For instance, pre-trained LLMs are used for **summarization** of long reports in financial and legal sectors, and **toxicity detection**. They are also foundational for generative AI applications like chatbots and content creation.
+
+### Limitations and Considerations of Pre-trained Models
+
+Despite their numerous benefits, PTMs also come with several limitations and important considerations:
+
+- **Bias**: PTMs **may carry biases present in their training data**, which can lead to unintended or discriminatory outcomes based on factors like race, gender, language, and cultural groups. For example, English data is overrepresented in many LLM training datasets, which may downplay non-English views.
+- **Domain Specificity**: While adaptable, models trained on general data **might not perform optimally on niche or highly domain-specific tasks without further fine-tuning**. Bridging the gap between general pre-training and domain-specific applications, especially with limited labeled data, remains a challenge.
+- **"Black-Box" Nature**: Many PTMs, especially deep learning models, are considered **"black boxes,"** making their internal workings and decision-making processes difficult to understand and interpret. Understanding _why_ a model made a specific decision can be challenging due to its complexity and the vast number of parameters.
+- **Computational Intensity and Cost**: Training and fine-tuning PTMs, particularly large ones, are **extremely hardware-intensive and expensive**, requiring substantial processing power, electricity, and specialized hardware like GPUs or TPUs. While training is often a one-time high cost, ongoing inference also demands significant compute power and can be expensive for models in active use.
+- **Data Quality and Hallucinations**: The effectiveness of PTMs heavily depends on the **quality of the training data**; "garbage in / garbage out" is a crucial factor. LLMs can **"hallucinate"**, confidently generating plausible-sounding but factually incorrect, nonsensical, or off-topic information not supported by their training data.
+- **Security and Misuse Risks**: PTMs are susceptible to **operational risks** like model drift and bias. They can also be targeted for **theft, reverse engineering, or unauthorized manipulation**. Malicious inputs can manipulate models to produce dangerous or unethical responses, and there's a concern about "sleeper agents"—hidden functionalities that can trigger insecure actions. Uploading confidential data for productivity can inadvertently expose it, as models are not designed as secure vaults.
+- **Copyright Concerns**: There are ongoing lawsuits and ethical considerations regarding whether PTMs are trained on **copyrighted material** without proper licensing, raising questions about fair use.
+- **Context Window Limitations**: Historically, transformer-based models faced limitations with a **fixed context window size and quadratic space complexity**. This hindered their ability to process and understand very long documents or extensive past interactions effectively. While advancements like larger context windows (e.g., Gemini 1.5 with up to 1 million tokens, Claude 2.1 with 200k tokens) are emerging, pushing these limits further remains a challenge.
+- **Over-confidence and Generalization**: PTMs can sometimes be **over-confident in their predictions**, even when they "do not know what they do not know". Preventing **overfitting** (where a model memorises training data but struggles to generalise to new data) is crucial and requires careful validation.

docs/section1/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "Section 1: Foundations of Artificial Intelligence",
+  "position": 2,
+  "link": {
+    "type": "generated-index",
+    "description": "5 minutes to learn the most important Docusaurus concepts."
+  }
+}

docs/section2/Building an AI server/Install Ollama on linux.md

@@ -0,0 +1,90 @@
+## How to install Ollama on a linux machine?
+
+To install **Ollama** on a Linux machine, follow these steps:
+
+**1. Check Prerequisites**
+
+- Ensure your system meets the minimum requirements: **64-bit CPU**, at least **8GB RAM** (16GB recommended), and **10–12GB free disk space**. A **GPU** (NVIDIA or AMD) is strongly recommended for optimal performance, but Ollama can run on CPU if necessary.
+- Make sure you have **curl** installed. If not, install it using:
+
+```bash
+sudo apt-get install curl
+```
+
+- For GPU support, verify your drivers (e.g., with `nvidia-smi` for NVIDIA GPUs).
+
+**2. Install Ollama**
+
+- The simplest way is to run the official installation script:
+
+```bash
+curl -fsSL https://ollama.com/install.sh | sh
+```
+
+This script will:
+    - Download and install Ollama to `/usr/local/bin`
+    - Create the `ollama` user and necessary groups
+    - Set up and start the Ollama systemd service
+
+**3. Verify Installation**
+
+- Check that Ollama is installed and running:
+
+```bash
+ollama --version
+systemctl status ollama
+```
+
+By default, Ollama runs as a service and listens on `localhost:11434`.
+
+**4. (Optional) Manual or Docker Installation**
+
+- For manual installation or advanced setups, you can download the binary or use Docker:
+    - Manual:
+
+```bash
+curl -fsSL https://ollama.com/download/ollama-linux-amd64.tgz | sudo tar zx -C /usr
+```
+
+Then set up the systemd service as described in the documentation.
+    - Docker:
+
+```bash
+docker pull ollama/ollama
+docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
+```
+
+This is useful for containerized environments.
+
+**5. Pull and Run a Model**
+
+- After installation, fetch a language model (e.g., Llama 2):
+
+```bash
+ollama pull llama2
+```
+
+- Run the model:
+
+```bash
+ollama run llama2
+```
+
+
+**6. Troubleshooting**
+
+- If GPU support is not detected, ensure your drivers and toolkits (e.g., `nvidia-cuda-toolkit`) are installed and repeat the installation.
+
+**Summary Table:**
+
+
+| Step | Command/Action |
+| :-- | :-- |
+| Update system | `sudo apt-get update && sudo apt-get upgrade` |
+| Install curl | `sudo apt-get install curl` |
+| Install Ollama | `curl -fsSL https://ollama.com/install.sh | sh` |
+| Verify installation | `ollama --version` and `systemctl status ollama` |
+| Pull a model | `ollama pull llama2` |
+| Run a model | `ollama run llama2` |
+
+After these steps, Ollama should be ready to use on your Linux system.

docs/section2/_category_.json

@@ -1,5 +1,5 @@
 {
-  "label": "Tutorial - Extras",
+  "label": "Section 2: Building and Using AI Systems",
   "position": 3,
   "link": {
     "type": "generated-index"

docusaurus.config.js

@@ -84,7 +84,7 @@ const config = {
         title: 'Wirediv AI',
         logo: {
           alt: 'My Site Logo',
-          src: 'img/logo.svg',
+          src: 'img/logo.jpg',
         },
         items: [
           {
@@ -144,7 +144,7 @@ const config = {
             ],
           },
         ],
-        copyright: `Copyright © ${new Date().getFullYear()} My Project, Inc. Built with Docusaurus.`,
+        copyright: `Copyright © ${new Date().getFullYear()} Wirediv.`,
       },
       prism: {
         theme: prismThemes.github,

src/pages/login.js

@@ -42,7 +42,7 @@ export default function Login() {
             boxShadow: '0 4px 10px rgba(0, 0, 0, 0.1)',
           }}
         >
-          <img src="img/combination-icon.png" width={'100px'} style={{ margin: "auto" }} />
+          <img src="img/logo.jpg" width={'100px'} style={{ margin: "auto" }} />
           <h1 style={{ marginLeft: 'auto', marginRight: 'auto' }}>Wirediv</h1>
           <h2 style={{ marginLeft: 'auto', marginRight: 'auto' }}>AI Course</h2>
           <label htmlFor="email" style={{ marginBottom: "10px" }}>

src/theme/ColorModeToggle/index.js

@@ -34,7 +34,7 @@ export default function ColorModeToggleWrapper(props) {
             marginTop: '10px',
             padding: '8px 12px',
             borderRadius: '6px',
-            color: 'var(--ifm-color-primary-dark)',
+            color: 'var(--ifm-color-primary-light)',
             border: 'none',
             cursor: 'pointer',
             marginRight: '5px',