Name: Evaluating LLM-powered applications
Start: 2025-02-25T19:05:00Z
End: 2025-02-25T19:05:51.000Z
Location: BrightTALK

Tackle AI challenges and find out why leading organizations of all sizes and industries use the Weights & Biases AI developer platform to build AI models and develop AI applications with confidence.

For modern quant firms, model and compute speed have become a new form of alpha. Whether developing AI models that trade in real time or power financial products and services, having the right AI tool stack is essential. In this webinar, you’ll learn how AI tools from Weights & Biases compress the entire quantitative research loop, from hypothesis to production, so quant firms can discover strategies sooner and deploy models with confidence. You’ll come away understanding how to use a modern AI tech stack that delivers tighter governance and greater confidence.

Accelerate AI model performance for quantitative trading

AI agents are hard to deploy, and their actions are often opaque. In this joint AWS and Weights & Biases session, we’ll demonstrate how to use Amazon Bedrock AgentCore to easily deploy agent prototypes to production in a scalable and safe manner. We’ll also show you how to use it with  W&B Weave to unify real-time tracing and performance views across development and production, providing you with clear visibility into agent behavior and decision-making from a single view. Finally, we’ll close the loop: turn production traces filtered by user/expert feedback into evaluation datasets, use Weave evals to measure what matters, and promote improved versions with governance and lineage.

Inside the black box: AI agent observability with Amazon Bedrock AgentCore and W&B Weave

Legacy contact center software can be costly and inconsistent in meeting customer needs. In this demo, see how multi-agent AI contact centers, built and monitored with Weights & Biases, address these challenges by integrating autonomous agents and real-time monitoring. Gain insights into tracing, quality assessment, and achieving consistent support across multiple communication channels.

Build and monitor multi-agent contact centers using Weights & Biases

Reinforcement learning (RL) is one of the most effective ways to fine-tune an AI agent for reliability, speed, and cost-effectiveness. Until now, getting started meant spending hours or days evaluating GPU providers and wiring up scripts and infrastructure. With our launch of the industry’s first Serverless RL, that’s changed. In this webinar, we’ll introduce the new capabilities and show you how to get your RL training job running in minutes without thinking about GPUs.

Introducing serverless reinforcement learning: Train reliable AI agents without worrying about GPUs

Building effective multi-agent AI systems requires an eval-driven development process and an end-to-end stack from infrastructure and frameworks to protocols and evaluation tools. Google Cloud and Weights & Biases have partnered to offer a cohesive stack that makes architecting multi-agent systems easier. In this webinar, we’ll show you how to use it to build your first real-world agentic AI application.

Architecting multi-agent workflows with Google Cloud and Weights & Biases

Even if you’re fluent in Python, building production-ready AI agents still involves a lot of boilerplate, orchestration, and debugging overhead. This webinar dives deep into how LlamaIndex’s modular, composable framework and Weights & Biases’ eval-driven development platform can help you move faster and smarter. You’ll learn how to streamline data pipelines, chain together reasoning steps, and trace agent behavior end-to-end—all while keeping full control of the stack. Whether you're experimenting with RAG pipelines or deploying autonomous agents, you'll come away with practical techniques to evaluate, monitor, and iterate with confidence. You'll learn:
- How to assemble agent pipelines using LlamaIndex’s intuitive APIs and modular architecture
- Strategies to instrument your agents with W&B Weave for tracing, evaluation, guardrails, and continuous improvement
- Tips for building, deploying, and monitoring agents without writing complex code

Zero to agent: Low-code AI development with LlamaIndex and Weights & Biases

AI agents introduce powerful automation opportunities and unique development challenges. With large language models (LLMs) producing highly variable outputs and user expectations on the rise, developers need robust tools for evaluation and monitoring. This session explores how W&B Weave supports reproducible testing workflows purpose-built for modern AI development, enabling teams to iterate faster and deploy with confidence.

Deploy AI agents confidently with a proven evaluation framework

AI agents have become essential for streamlining and enhancing research tasks with precise, contextually relevant information. For instance, they can compile real-time market trends from financial websites or generate concise summaries of the latest clinical research findings. Join us to discover how Tavily's Search API, combined with W&B Weave's evaluation and monitoring capabilities, creates a powerful toolkit for developing production-ready AI web research agents. You'll learn how to:
- Enhance your AI applications by adding customizable, high-precision search to your agentic workflows
- Prevent harmful AI responses by combining Tavily’s trusted, authoritative data sources with Weave guardrails
- Debug and monitor AI agents with detailed tracing and observability tools

Build AI research agents with Tavily and W&B Weave

W&B Inference provides API and playground access to leading open-source foundation models, without the additional overhead of dealing with multiple model providers or hosting on your own. In this demo, we'll walk you through how to leverage W&B Inference for prototyping and evaluating AI applications using popular open-source LLMs.

W&B Inference: Access CoreWeave-hosted open-source models in W&B Weave

Discover how to bring CoreWeave’s deep infrastructure observability directly into your W&B training workflows. When training on CoreWeave, W&B Models automatically captures key events from CoreWeave’s Mission Control, giving you powerful insights and expert-backed remediation hints to debug your training and fine-tuning runs faster.

CoreWeave infrastructure observability in W&B Models

Dans ce dernier volet, nous passons à l’analyse technique. Apprenez à construire, tester et superviser techniquement un agent de recrutement conforme en utilisant W&B Weave. Nous démontrerons deux techniques complémentaires de mitigation des risques – l’évaluation hors-ligne avant le déploiement, et les garde-fous en temps réel en production – tout en montrant comment capturer des métriques quantitatives et des traces qualitatives grâce à une surveillance granulaire. Découvrez comment intégrer le fine-tuning, l'entraînement du modèle, l’ingénierie système et le code applicatif dans un pipeline unifié et auditable.

À quoi s’attendre:
- Une session approfondie de 30 minutes sur notre pipeline agentique avec Weave, en continuité avec les Parties 1 & 2
- Une double stratégie de mitigation des risques : benchmarks hors-ligne pour tester les décisions, garde-fous en ligne pour faire respecter les politiques à l’exécution
- Démonstration de la collecte de traces dans W&B : dashboards de performance et biais, ainsi que journaux de conversations pour audits qualitatifs
- Modèles d’intégration pour unifier fine-tuning (W&B Models), workflows Weave et composants applicatifs

Ce que vous apprendrez:
- Comment concevoir et exécuter des benchmarks hors-ligne pour valider le comportement des agents selon les critères de conformité
- Comment configurer les garde-fous Weave et les hooks de validation W&B pour détecter les violations en temps réel
- Les meilleures pratiques pour capturer, visualiser et trier les métriques quantitatives et les traces qualitatives
- Comment architecturer un pipeline complet, de bout en bout, qui fournit une piste d’audit vivante et exploitable

Partie 3: Conception d'un agent d'embauche conforme : Une plongée technique

Poursuivant l’introduction aux agents dans le cadre de l’AI Act, cette session se concentre sur le « qui » et le « comment ». Après un rappel des jalons de l’AI Act et de ses exigences pour les cas à haut risque, nous partagerons une feuille de route détaillée pour coordonner la conformité entre les responsables de la gouvernance, les ingénieurs ML, les chefs de produit, les juristes et autres intervenants. Vous verrez comment les outils MLOps tels que Weights & Biases standardisent les workflows, automatisent la génération de preuves, et imposent des étapes de validation. La session se conclura par une démonstration en direct de notre pipeline de recrutement agentique, accompagnée d’un rapport dynamique W&B servant de dossier de conformité.

À quoi s'attendre:
- Un rappel rapide des jalons de l’AI Act et des défis techniques/gouvernance de notre prototype d’agent de recrutement
- Des définitions de rôles concrètes, des processus de passation, et des stratégies de réduction des risques pour des équipes pluridisciplinaires
- Une exploration de W&B Models et Weave pour automatiser le reporting, appliquer les standards, et réduire la documentation manuelle
- Démo en direct : pipeline de bout en bout pour un agent de recrutement et rapport de conformité dynamique dans Weights & Biases

Ce que vous apprendrez:
- Comment traduire les exigences de l’AI Act en tâches spécifiques pour chaque groupe d’intervenants
- Les risques courants liés à l’utilisation d’agents dans le recrutement (ex : biais, hallucinations) et les techniques techniques de mitigation
- Comment configurer les workflows et rapports W&B pour centraliser la traçabilité et imposer des validations
- Comment adapter notre dépôt de démonstration comme modèle pour vos propres cas d’usage IA à haut risque

Partie 2: Rôles, processus et plateformes pour un agent de recrutement conforme

As agentic AI applications become increasingly powerful and complex, building and deploying them with confidence requires the right tools and processes. In this webinar, we’ll guide you through an end-to-end workflow that combines Dify’s open-source LLM app development platform with W&B Weave’s developer-friendly evaluation and monitoring toolkit. Learn how to accelerate your path from prototype to production while ensuring reliability and performance at every stage, including how to:
- Streamline agentic AI application development using Dify’s workflow, RAG pipeline, and model management features
- Use best practices to continuously evaluate, monitor, and iterate your AI agents 
- Build production-ready AI workflows by combining an intuitive development interface with powerful observability tools

Developing and monitoring AI agents using Dify and W&B Weave

La montée en puissance rapide des agents IA – des systèmes autonomes agissant pour le compte des utilisateurs – ouvre un potentiel considérable en matière de productivité et de personnalisation. En particulier, dans les cas d’usage complexes ou à haut risque, ces agents peuvent apporter une valeur ajoutée essentielle. Toutefois, dans le cadre du nouvel AI Act de l’Union européenne, les cas d’usage à haut risque sont soumis à des exigences strictes en matière de gouvernance, de documentation et de transparence. Lors de cette session, vous découvrirez comment transformer ces défis de conformité en un avantage stratégique en définissant clairement les rôles et processus, en associant les articles juridiques aux tâches d’ingénierie, et en tirant parti de Weights & Biases pour faire collaborer les équipes techniques et non techniques.

À quoi s’attendre:
- Une analyse en deux volets des agents IA : les opportunités majeures et les contraintes réglementaires liées aux cas d’usage à haut risque
- Une présentation approfondie de notre étude de cas sur un agent de recrutement (fine-tuning + workflows agents) conforme aux articles pertinents de l’AI Act
- Des cadres pratiques pour définir les rôles d’équipe, organiser les transferts de responsabilité, et assurer une auditabilité rapide

Ce que vous apprendrez:
- Comment les articles 5 à 15 de l’AI Act s’appliquent aux systèmes agents et comment les traduire en tâches concrètes d’ingénierie et de reporting
- Des stratégies pour structurer un développement rapide et conforme sans freiner l’innovation
- Comment Weights & Biases centralise la traçabilité des modèles, la documentation et les rapports pour les ingénieurs comme pour les responsables conformité

Partie 1: Les agents dans le cadre de l'AI Act européen: équilibre entre opportunité, risque et conformité

At the Fully Connected 2025 AI conference, we introduced three new integrations between CoreWeave and Weights & Biases and the public preview of online evaluations in W&B Weave. We'll dive deep into these new capabilities and how they help ML engineers and developers iterate faster on their AI agents and applications. Watch this webinar to learn how to:
- Continuously monitor AI applications with online evaluations to identify and troubleshoot issues in production
- Analyze monitors over time to discover trends and patterns to maintain your application’s quality over time
- Use W&B Inference to explore, evaluate, and choose the best open-source models for your use case
- Easily run public evaluations with a single click to compare performance and keep leaderboards up to date

How to iterate AI faster: Lessons from Fully Connected 2025

Dive deep into the world of AI agent communication and observability in this comprehensive session. Learn how the Model Context Protocol (MCP) solves communication challenges for distributed AI agents, acting as a universal standard for connecting Large Language Models (LLMs) with the tools they need.

This video will guide you through practical, hands-on demonstrations using W&B Weave to bring transparency to your AI applications. You will learn how to:
- Get started with MCP and W&B Weave with just a few lines of code to track and trace interactions within your applications
- Monitor your applications in production, gather feedback, and continuously improve
- Debug agentic workflows by identifying and resolving issues like duplicated processes and errors in real-time
- Orchestrate and monitor multiple AI agents built with frameworks like Crew AI, analyzing their performance and costs

Furthermore, we will share invaluable best practices for using W&B Weave in both development and production environments. These include constructing meaningful trace trees, tagging traces with structured metadata, managing PII, and more to ensure your AI applications are robust, scalable, and observable.

Gaining transparency in AI agents: From MCP communication to W&B Weave observability

Join Weights & Biases Co-Founder Lukas Biewald and CoreWeave SVP of Engineering Chen Goldberg as they unveil the first of many collaborative projects. Lukas shares insights on the rapid evolution of AI over the past two years, highlighting major advances in generative models, the shift to a new computing era, and the growing complexity of agentic applications. He also introduces the latest updates to W&B Models and Weave. Chen then takes the stage to present three powerful new features the team has been working on together.

Weights & Biases and CoreWeave: Fully Connected 2025 State of the AI Industry Keynote

Over the past two years, the AI landscape has undergone a remarkable transformation. Large Language Models (LLMs) have moved to the forefront, powering applications like ChatGPT and driving an open revolution of models spearheaded by Llama. Now, we’re witnessing agentic systems entering the mainstream. 

Despite these advances, significant challenges persist as we transition into a generative AI and agent-first world. These challenging questions require a collective community working towards common goals to address them effectively.  

PyTorch has become that central place bringing together diverse perspectives with a community collectively building a comprehensive framework that integrates all layers of AI development. Joe Spisak, a long-time leader of PyTorch and Llama open source, talks about the next phase of PyTorch, the growing ecosystem around Foundation 2.0 and how PyTorch continues to evolve as the open language of AI.

PyTorch: The open language of AI

LLMs are non-deterministic, which means you need rock-solid evaluations to make sure your AI apps behave the way you (and your users) want. In this recorded webinar, we’ll show you how to run rigorous evals on LLM-powered applications, letting you iterate fast and ship with confidence.

Evaluating LLM-powered applications

GenAI

LLMs

Software Development

Development

Artificial Intelligence

The application development community features top thought leadership focusing on optimal practices in software development, SDLC methodology, mobile app development and application development platforms and tools. Join top software engineers and coders as they cover emerging trends in everything from enterprise app development to developing for mobile platforms such as Android and iOS.

Application Development

Enterprise applications have become a crucial piece of infrastructure for many businesses. The enterprise applications community on BrightTALK features the latest insights in enterprise application integration, enterprise application architecture and EAS software. Join the community to gain access to thousands of videos and webinars presented by recognized enterprise information systems experts and arm yourself with the knowledge you need to succeed.

Enterprise Applications

Practicing business intelligence allows your company to transform raw data into sets of insights for targeted business growth. The business intelligence and analytics community on BrightTALK is made up of thousands of data scientists, database administrators, business analysts and other data professionals. Find relevant webinars and videos on business analytics, business intelligence, data analysis and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Business Intelligence and Analytics

Get powerful marketing data and analytics insights from influential experts. Connect with thought leaders and colleagues to get the most up-to-date knowledge on the analytics that quantify your investments.

Data and Analytics

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Evaluating LLM-powered applications

Presented by

About this talk

Weights & Biases