Added 2 sections; added logo

docs/section1/Pre-Trained Models/Pre_trained Models.md

@@ -0,0 +1,30 @@
+# Pre-trained Models: Benefits and Limitations
+
+**Pre-trained models (PTMs)** are **Machine Learning (ML) models that have been previously trained on a large dataset to solve a specific task or set of tasks**. These models learn patterns, features, and representations from their training data. Often, PTMs are **deep network models** that have undergone training on vast datasets to achieve a specific task, such as identifying cancer spots in medical images or detecting suspicious behavior in video surveillance feeds.
+
+The "large-scale" aspect of PTMs like Large Language Models (LLMs) refers to their **sophisticated pre-training objectives and huge model parameters**, which allow them to **effectively capture knowledge from massive labeled and unlabeled data**. By storing this knowledge in their parameters and then being fine-tuned for specific tasks, this rich, implicitly encoded knowledge can benefit a variety of downstream tasks. The AI community widely agrees on adopting PTMs as a backbone for downstream tasks rather than building models from scratch.
+
+### Benefits of Pre-trained Models
+
+PTMs offer numerous advantages for AI development and deployment:
+
+- **Accelerated Development and Accessibility**: PTMs **significantly reduce development time and computational resources**, as they eliminate the need to train a model from scratch, which can take months. This makes machine learning more accessible to a wider range of users, including those without extensive computational resources or labeled data. Developers can use PTMs and other predefined building blocks to bring new AI solutions to market faster and more cost-effectively.
+- **Leveraging Existing Expertise**: Since PTMs are typically trained on **large datasets curated by domain experts**, developers can leverage this inherent expertise without having to create their own specialized datasets. This is particularly useful in specialized domains where labeled data is scarce or difficult to create.
+- **Improved Performance**: PTMs can **improve AI application performance**. Studies show that pre-trained LLMs can perform as well as, or even better than, models specifically trained from scratch for custom tasks. This is partly due to their training on massive datasets and well-calibrated parameters.
+- **Flexibility and Customisation**: PTMs offer **flexibility** as they can be used as they are, or **customised for new use cases through transfer learning**. Transfer learning leverages the knowledge within PTMs for new, similar tasks with only a small amount of additional labeled data. For example, a single foundational model can be fine-tuned multiple times for different use cases. Developers can **engineer prompts and tweak parameters** to control model outputs significantly.
+- **Simplified Integration**: Many PTMs, particularly LLMs, come with **easy-to-use APIs** (Application Programming Interfaces), allowing developers to access their capabilities with just a few lines of code, thereby avoiding the hidden costs associated with building and hosting models from scratch.
+- **Diverse Applications**: PTMs are transforming various industries. For instance, pre-trained LLMs are used for **summarization** of long reports in financial and legal sectors, and **toxicity detection**. They are also foundational for generative AI applications like chatbots and content creation.
+
+### Limitations and Considerations of Pre-trained Models
+
+Despite their numerous benefits, PTMs also come with several limitations and important considerations:
+
+- **Bias**: PTMs **may carry biases present in their training data**, which can lead to unintended or discriminatory outcomes based on factors like race, gender, language, and cultural groups. For example, English data is overrepresented in many LLM training datasets, which may downplay non-English views.
+- **Domain Specificity**: While adaptable, models trained on general data **might not perform optimally on niche or highly domain-specific tasks without further fine-tuning**. Bridging the gap between general pre-training and domain-specific applications, especially with limited labeled data, remains a challenge.
+- **"Black-Box" Nature**: Many PTMs, especially deep learning models, are considered **"black boxes,"** making their internal workings and decision-making processes difficult to understand and interpret. Understanding _why_ a model made a specific decision can be challenging due to its complexity and the vast number of parameters.
+- **Computational Intensity and Cost**: Training and fine-tuning PTMs, particularly large ones, are **extremely hardware-intensive and expensive**, requiring substantial processing power, electricity, and specialized hardware like GPUs or TPUs. While training is often a one-time high cost, ongoing inference also demands significant compute power and can be expensive for models in active use.
+- **Data Quality and Hallucinations**: The effectiveness of PTMs heavily depends on the **quality of the training data**; "garbage in / garbage out" is a crucial factor. LLMs can **"hallucinate"**, confidently generating plausible-sounding but factually incorrect, nonsensical, or off-topic information not supported by their training data.
+- **Security and Misuse Risks**: PTMs are susceptible to **operational risks** like model drift and bias. They can also be targeted for **theft, reverse engineering, or unauthorized manipulation**. Malicious inputs can manipulate models to produce dangerous or unethical responses, and there's a concern about "sleeper agents"—hidden functionalities that can trigger insecure actions. Uploading confidential data for productivity can inadvertently expose it, as models are not designed as secure vaults.
+- **Copyright Concerns**: There are ongoing lawsuits and ethical considerations regarding whether PTMs are trained on **copyrighted material** without proper licensing, raising questions about fair use.
+- **Context Window Limitations**: Historically, transformer-based models faced limitations with a **fixed context window size and quadratic space complexity**. This hindered their ability to process and understand very long documents or extensive past interactions effectively. While advancements like larger context windows (e.g., Gemini 1.5 with up to 1 million tokens, Claude 2.1 with 200k tokens) are emerging, pushing these limits further remains a challenge.
+- **Over-confidence and Generalization**: PTMs can sometimes be **over-confident in their predictions**, even when they "do not know what they do not know". Preventing **overfitting** (where a model memorises training data but struggles to generalise to new data) is crucial and requires careful validation.