3-Generative AI

Explore core principles, tools, and ethical use of Generative AI, and discover its real-world impact and foundational models powering creative applications.

Explore the fundamentals of generative AI, its key modalities, advanced capabilities, and essential ethical considerations shaping responsible AI development.

Explore real-world applications of Generative AI, including LLM-assisted coding, and learn to prompt, validate, and improve AI-generated code and tests.

Discover foundation models: large, versatile AI systems trained on massive datasets that generalize across tasks, surpassing traditional models in scalability and adaptability.

Learn to build text classifiers with foundation models, using zero-shot and few-shot prompt engineering for tasks like sentiment and spam detection, and evaluate classifier accuracy.

Learn how generative AI creates new data with architectures like Transformers and diffusion models, and how training enables creativity, reasoning, and task-specific abilities.

Learn how to assess generative AI using human evaluation, exact metrics, AI judges, and benchmarks, ensuring robust performance for open-ended, probabilistic model outputs.

Learn practical techniques to evaluate generative AI models, from Exact Match to ROUGE, semantic similarity, code correctness, Pass@k, and LLM-as-a-Judge scoring.

Explore neural networks from perceptrons to multilayer perceptrons, learning how they adapt via training, gradient descent, and backpropagation to solve complex AI tasks.

Learn to implement neural networks in PyTorch by mastering tensors, model building, loss functions, optimizers, data loading, and complete training loops for practical machine learning.

Explore AI model interpretability and ethics, including bias, misinformation, environmental impact, and fairness for responsible development and deployment of AI technologies.

Discover how LLMs generate text token by token using Hugging Face's Transformers, from tokenization to model use, and explore hands-on demos with efficient generation methods.

Explains the theory of using roles or personas to control the tone, style, and expertise of an LLM's output.

Provides hands-on practice in iteratively developing a role-based prompt to create a believable historical figure persona.

Learn to adapt foundation models for specialized tasks using prompt engineering, RAG, fine-tuning, model compression, and agentic AI tools for efficient, tailored AI solutions.

Learn to efficiently customize foundation models with PEFT and SFT, using LoRA to teach LLMs new skills like spelling via hands-on data preparation and fine-tuning.

Explore post-training for foundation models, including supervised and preference fine-tuning, to align AI with human values, improve usability, and ensure responsible interactions.

Learn to fine-tune LLMs for structured tasks like counting and spelling using GRPO and LoRA, applying reinforcement-based reward functions for targeted skill improvements.

Teaching an LLM to count the number of letters in a word using GRPO.

Introduces Large Language Models (LLMs), their core concepts, and the course structure. Covers prerequisites, environment setup, and defines Retrieval-Augmented Generation (RAG).

Explore the four core capabilities of LLMs: generation, summarization, classification, and reasoning. Real-world applications and the importance of RAG for building trust.

Learn to build a stateful chatbot using an LLM. Covers managing conversation history, using system prompts to define behavior, and understanding message roles (system, user, assistant).

Defines prompt engineering and its components. Explains how to control LLM outputs using inference parameters like temperature, top-p, max tokens, and stop sequences.

Apply prompting techniques hands-on. Implement Chain of Thought (CoT) prompting to improve reasoning and test how different inference parameters change model behavior.

Explains the theory of systematically refining prompt instructions by modifying components like Role, Task, Context, Examples, and Output Format.

Provides hands-on practice iteratively refining a prompt to transform a generic recipe analyzer into a precise dietary consultant that produces structured JSON.

Covers the foundations of NLP for LLMs. Defines tokenization, embeddings as semantic vectors, and vector search (similarity search) as the basis for finding relevant information.

Hands-on practice with tokenization. Implementing embedding generation and vector search to build a semantic search system from scratch.

Learn the business trade-offs of model selection. Covers performance, cost, speed, and control (TCO). Compares general-purpose (generation) vs. specialized (reasoning) models.

Apply model selection theory. Calculate Total Cost of Ownership (TCO) including error costs. Implement a hybrid model routing system to balance cost and quality.

Introduces the RAG architecture. Compares naive vs. advanced modular RAG. Covers the data ingestion pipeline, focusing on data formats and intelligent chunking strategies.

Explains semantic search and the role of vector databases. Covers indexing algorithms (HNSW) for speed and advanced retrieval techniques like HyDE and re-ranking.

Learn to write prompts for RAG. Covers grounding answers in context, handling conflicts, managing uncertainty, and enforcing verifiability by generating inline citations.

Build a complete RAG system. Practice vector database operations in ChromaDB, including adding documents, applying metadata filters, and implementing a retrieval and generation pipeline.

Learn to evaluate RAG system quality. Introduces key metrics: Context Precision, Context Recall, Faithfulness, and Answer Relevancy. Covers frameworks like RAGAS.

Implement a RAG evaluation pipeline using the RAGAS framework. Learn to calculate and interpret quality metrics to diagnose and improve your RAG system's performance.

Build an end-to-end RAG chatbot. Ingest NASA mission data, build a vector search pipeline, generate answers, and evaluate the system's quality.

Discover multimodal AI fundamentals and technologies, including models and use cases that process and generate text, images, audio, and video for richer, real-world applications.

Explore practical applications of multimodal AI by using APIs and open-source models for image captioning and audio transcription, with hands-on exercises and secure credential handling.

Explore how transformers unify text, images, audio, and video through attention, embeddings, and fusion strategies, powering state-of-the-art multimodal understanding and generation.

Explore practical tools for building multimodal AI apps, compare commercial and open-source options, and use Pydantic AI to create reliable, structured, vendor-agnostic workflows.

Explore enterprise visual content processing: core computer vision tasks, digital image representation, and real-world applications for efficiency, safety, and automation.

Explore vision data pipelines using HuggingFace, from dataset loading to resizing and normalization, with demos and hands-on exercises for effective image pre-processing.

Learn how embeddings convert images into compact vectors for efficient search, enable cross-modal tasks with models like CLIP, and power large-scale, robust computer vision systems.

Explore how to build text-to-image and image-to-image search using CLIP embeddings, combining theory, real-world demos, hands-on practice, and solution walkthroughs.

Explore multimodal vision APIs: prompt design, parameter tuning, structured outputs, cost control, integration, and best practices for robust, efficient image analysis.

Explore Gemini Vision API basics by practicing image moderation, learning to analyze images and implement moderation workflows using real-world examples and guided hands-on exercises.

Explore Vision Transformer models: core architecture, image tokenization, self- and cross-attention, and top models (SAM, RT-DETR, DINOv2) for segmentation, detection, and enterprise use.

Explore vision transformers with hands-on demos: extract image embeddings using DINOv2 and perform object detection and segmentation using RT-DETR and SAM2.1 models.

Learn how vision-language models align images and text for tasks like search, captioning, and VQA, with focus on architectures, applications, data needs, and deploying for enterprise use.

Explore zero-shot image classification and auto-labeling for driving scenes using CLIP, enabling efficient, scalable multimodal vision applications.

Explore how diffusion models generate images by reversing noise through iterative denoising, inspired by physical diffusion processes and key to modern generative AI developments.

Discover enterprise audio processing, core speech tasks (transcription, diarization, sentiment, TTS), key use cases, and strategies for value and integration in modern businesses.

Explore how audio is digitized for AI: sample rate, bit depth, channels, formats, and mel spectrograms for speech, plus challenges and best practices in audio preprocessing and analysis.

Explore audio processing with librosa: load, resample, convert, and analyze audio files; visualize with mel spectrograms and apply techniques through hands-on exercises.

Explore audio embeddings for efficient sound classification and retrieval, using models like CLAP to enable semantic search and robust text-based audio analysis at scale.

Explore using CLAP for sound retrieval, similarity, and zero-shot classification, then apply these skills to detect fan on/off states in real audio data.

Discover automatic speech recognition with Whisper: a robust, multilingual, open-source model for accurate transcription, translation, and speech processing in real-world audio.

Explore real-world speech transcription and translation with Whisper and Gemini, using Python to process, segment, and align audio with text, including multilingual support.

Explore advances in Audio Intelligence: multimodal systems, speech recognition, TTS, enterprise controls, creative workflows, and ethics for robust, secure, and accessible audio solutions.

Explore audio sentiment and command analysis using Pydantic AI and Gemini; learn to extract emotions and recognize spoken commands from audio with real-world datasets and hands-on exercises.

Explore voice content moderation: real-time and batch pipelines, compliance, privacy, layered detection, and operational excellence for secure and fair audio classification.

Learn to build a voice moderation system using Gemini to transcribe audio, detect personal data disclosures, and flag policy violations in customer service recordings.

Discover how enterprise video AI overcomes temporal complexity using smart frame selection for efficient understanding, search, classification, moderation, and generation at scale.

Explore key AI models like YOLO for real-time detection, CoTracker and TimeSformer for motion and temporal understanding, enabling advanced, scalable enterprise video analytics.

Learn how to detect and track objects in videos using YOLOv9, apply multi-object tracking, handle small objects, and count items crossing boundaries in practical scenarios.

Explore methods for video analysis and search using foundation models and CLIP4Clip, balancing temporal understanding, cost, and retrieval accuracy for enterprise applications.

Explore video understanding with Gemini and Clip4Clip: learn automated video description, key moment detection, and natural language video search using AI models and structured outputs.

Learn to classify and moderate video by modeling temporal patterns, handling real-world challenges, and combining automation with human oversight for scale, accuracy, and compliance.

Learn to build automated systems for video classification and moderation with Gemini and Pydantic AI, including action recognition and safety compliance in real-world scenarios.

Explore generative video AI tools and workflows that turn text, images, or footage into dynamic content for marketing, training, and creative use while ensuring quality and compliance.

Learn to generate marketing videos with Veo 3 using both text-to-video and image-to-video workflows, and understand their strengths, limitations, and real-world applications.

Explore deployment of multimodal AI systems for text, images, audio, video via unified APIs, multi-API orchestration, and custom solutions, balancing speed, cost, and control.

Explore tools and strategies for implementing, serving, and monitoring AI solutions, from rapid prototyping to production, including unified APIs, orchestration, and managed platforms.

Learn to build multimodal chatbots and analysis apps using Gradio and Pydantic AI, covering async programming, media inputs, rate limiting, and interface customization.

Learn to monitor and log multimodal AI systems, tracking performance, costs, and failures across modalities for optimized, reliable, and coherent production deployments.

Learn to implement logging and performance monitoring for multimodal AI chatbots using Gradio and Arize Phoenix, enabling robust analytics, debugging, and cost tracking.

Learn how to evaluate multimodal AI apps using user feedback systems and testing methods, blending human review, automated metrics, and continuous monitoring for quality improvement.

Learn to build robust testing frameworks for multimodal AI apps using Pydantic Evals, covering structured outputs, semantic evaluation, custom evaluators, and hands-on exercises.

Learn strategies to scale multimodal AI: unified APIs, multi-API pipelines, and custom deployments, focusing on performance, cost, reliability, and architectural trade-offs.

In this project, students will create an AI agent that simulates customer service scenarios and specialized monitoring agents that analyze communications across text, images, videos, and audio.