Name: AI Starter Kit: Using Composition of Experts Models with Langchain
Start: 2024-05-06T12:30:00Z
End: 2024-05-06T12:30:13.000Z
Location: BrightTALK
Rating: 0

Customers turn to SambaNova to quickly deploy state-of-the-art generative AI capabilities within the enterprise. Our purpose-built enterprise-scale AI platform is the technology backbone for the next generation of AI computing.

The inference process in Large Language Models (LLMs) is often limited due to the absence of parallelism in the auto-regressive decoding process, resulting in most operations being restricted by the memory bandwidth of accelerators. While methods such as speculative decoding have been suggested to address this issue, their implementation is impeded by the challenges associated with acquiring and maintaining a separate draft model. 

Recorded at SambaNova Systems, this seminar covers Medusa, an efficient method that augments LLM inference by adding extra decoding heads to predict multiple subsequent tokens in parallel.

Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

Learn how to build complex reasoning applications that run 10x faster in a 10x smaller footprint on our new architecture that unlocks blazing fast handoffs for chained models on chip. We'll review the current state of the generative AI application market and dig into some of the pain points our customers were seeing at SambaNova Systems. We'll then cover what we built to solve those problems and how it works, what our customers are doing with it, and how you can easily try it yourself.

The New Way To Deliver Enterprise AI Applications

A walkthrough of our AI Starter Kit available on GitHub using Composition of Experts (CoE) models with Langchain. 

The repository provides a Python script and a Jupyter Notebook that demonstrate how to call SambaNova CoE models using the Langchain framework. The script offers different approaches for calling CoE models, including using SambaStudio with a named expert or routing and Sambaverse - a free tool where you can discover, compare and evaluate the best open source models.

AI Starter Kit: Using Composition of Experts Models with Langchain

Researchers across a variety of scientific disciplines are developing new approaches to merge deep learning with their traditional computational science techniques. One such approach is to replace part of a complex, computational demanding, simulation with a surrogate AI model to accelerate the simulation. Attend this session to learn about experimentation with this approach at Lawrence Livermore National Labs.

The Convergence of Machine Learning and HPC for Cognitive Simulation

As the usage of large language models (LLMs) grows, performing efficient inference with these models becomes increasingly important. While speculative decoding has recently emerged as a promising direction for speeding up inference, existing methods are limited in their ability to scale to larger speculation budgets, and adapt to different hyperparameters and hardware.

Recorded at SambaNova Systems, this seminar covers Sequoia - a scalable, robust, and hardware-aware algorithm for speculative decoding. To attain better scalability, Sequoia introduces a dynamic programming algorithm to find the optimal tree structure for the speculated tokens. To achieve robust speculative performance, Sequoia uses a novel sampling and verification method that outperforms prior work across different decoding temperatures. Finally, Sequoia introduces a hardware-aware tree optimizer that maximizes speculative performance by automatically selecting the token tree size and depth for a given hardware platform.

Evaluation shows that Sequoia improves the decoding speed of Llama2-7B, Llama2-13B, and Vicuna-33B on an A100 by up to 4.04×, 3.73×, and 2.27×. For offloading setting on RTX4090, Sequoia achieves as low as 0.56 s/token for exact Llama2-70B inference latency, which is 9.96× on our optimized offloading system (5.6 s/token), 9.7× than DeepSpeed-Zero-Inference, 19.5× than Huggingface Accelerate.

Scalable, Robust, and Hardware-aware Speculative Decoding

Machine Learning

Machine Learning Models

Artificial Intelligence

Big Data

Generative AI

Language Model

Computer Vision

The application development community features top thought leadership focusing on optimal practices in software development, SDLC methodology, mobile app development and application development platforms and tools. Join top software engineers and coders as they cover emerging trends in everything from enterprise app development to developing for mobile platforms such as Android and iOS.

Application Development

Enterprise applications have become a crucial piece of infrastructure for many businesses. The enterprise applications community on BrightTALK features the latest insights in enterprise application integration, enterprise application architecture and EAS software. Join the community to gain access to thousands of videos and webinars presented by recognized enterprise information systems experts and arm yourself with the knowledge you need to succeed.

Enterprise Applications

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Information Technology

Get powerful emerging technology insights from influential experts. Connect with thought leaders and colleagues to get the most up-to-date knowledge on technologies like 3D printing, pervasive robotics, natural interfaces, connected everything and cognitive systems.

Emerging Technologies

The research and development community on BrightTALK brings together leading professionals in healthcare, technology and manufacturing to present on current topics in their fields. From big data in healthcare to online forensics training and medical research practices, the community is an exceptional resource for those looking to expand their knowledge. Be part of the conversation by joining the community and participating in interactive live webinars.

Research and Development

Get powerful insights to run your business. CEOs, CFOs, CMOs, COOs, CTOs and CIOs are among the leaders in this community who connect with peers and experts to grow their businesses.

CxO Strategy

In the business management community, commercial experts will be sharing webinars and videos on an array of subjects. From sales and marketing, to finance, productivity and growth, this content will help you stay up-to-date with the current economic environment and provide timely management insight to drive business growth.

AI Starter Kit: Using Composition of Experts Models with Langchain

Presented by

About this talk

More from this channel