| Page 659 | Kisaco Research

Section Manager, Tokyo Electron

I had the privilege of attending the event, and it was truly an unforgettable experience. From the moment I walked in, I was greeted with warmth and enthusiasm. The organization and execution were impeccable, making it clear that a tremendous amount of effort and dedication had gone into planning every aspect. The speakers were not only knowledgeable but also engaging, leaving the audience inspired and motivated. The topics covered were not only relevant but thought-provoking, sparking meaningful conversations among attendees.

CEO, Efabless Corporation

It's a great conference including the entire innovation chain from chips and software to systems and end-users. The Gen AI sessions were particularly interesting as we track from ground zero the emergence of this important trend.

CEO and Co-Founder, NeuEdge Ltd

The summit has been an unparalleled event for meeting brilliant people working at every stage of the AI hardware stack. As a founder of an AI hardware startup, I was grateful for the willingness of all attendees to share their insights, and I left with many new friends and connections.

Senior Full Stack Data Scientist, Capital One

I personally though this was a great mix of hardware companies and software companies coming together to solve cutting-edge problems. As an attendee - there was a lot of useful learning.

 

Dylan Patel

Chief Analyst
Semi Analysis

Dylan Patel

Chief Analyst
Semi Analysis

Dylan Patel

Chief Analyst
Semi Analysis

Author:

Xavier Soosai

Chief Information Officer
Center for Information Technology/National Institute of Health

As the Director of the Office of Information Technology Services of the Center for Information Technology (CIT), Soosai oversees ten service areas and the delivery of scientific research and business operations across the institutes and centers (ICs) at NIH. This includes maintaining the high-performance computing environment used by NIH intramural scientists; maintaining NIH’s secure, high-speed network; ensuring the viability and availability of collaboration services, compute hosting and storage services, identity and access management services, service desk support, and more for the NIH community. 

Soosai works with CIT leadership and internal service area managers and collaborates with NIH ICs to define scope and provide technical expertise, strategic planning, and leadership for local and enterprise IT projects that drive efficiency and innovation across NIH. Additionally, Soosai is responsible for directing the evaluation and adoption of rapidly evolving technology and forecasting future technology needs.

 

... read more

Xavier Soosai

Chief Information Officer
Center for Information Technology/National Institute of Health

As the Director of the Office of Information Technology Services of the Center for Information Technology (CIT), Soosai oversees ten service areas and the delivery of scientific research and business operations across the institutes and centers (ICs) at NIH. This includes maintaining the high-performance computing environment used by NIH intramural scientists; maintaining NIH’s secure, high-speed network; ensuring the viability and availability of collaboration services, compute hosting and storage services, identity and access management services, service desk support, and more for the NIH community. 

Soosai works with CIT leadership and internal service area managers and collaborates with NIH ICs to define scope and provide technical expertise, strategic planning, and leadership for local and enterprise IT projects that drive efficiency and innovation across NIH. Additionally, Soosai is responsible for directing the evaluation and adoption of rapidly evolving technology and forecasting future technology needs.

 

Emerging Memory Innovations
Systems Infrastructure/Architecture
HBM/CXL

Author:

SangJoon Hwang

Corporate EVP, Head of DRAM Product & Technology Samsung Electronics
Samsung Electronics

SangJoon Hwang received B.S, M.S., and Ph.D. degrees in electric engineering from the Korea University in 1994, 1996, and 2008, respectively.

He joined the Samsung Electronics, Hwaseong, South Korea in 1996, where he had successfully led a DRAM design group in 2014 and the Flash design team in 2017 as a Vice President and the Memory Product Planning team in 2019 as as a Senior Vice President. Through leading various backgounds from product planning to design, his experience enhances the overall quality of Samsung DRAM products.

 

Since 2023, he has been leading the DRAM Product & Technology of the Samsung memory division. His current research interests include architecture for next-generation DRAM and product development utilizing new process technology for new product line-up.

... read more

SangJoon Hwang

Corporate EVP, Head of DRAM Product & Technology Samsung Electronics
Samsung Electronics

SangJoon Hwang received B.S, M.S., and Ph.D. degrees in electric engineering from the Korea University in 1994, 1996, and 2008, respectively.

He joined the Samsung Electronics, Hwaseong, South Korea in 1996, where he had successfully led a DRAM design group in 2014 and the Flash design team in 2017 as a Vice President and the Memory Product Planning team in 2019 as as a Senior Vice President. Through leading various backgounds from product planning to design, his experience enhances the overall quality of Samsung DRAM products.

 

Since 2023, he has been leading the DRAM Product & Technology of the Samsung memory division. His current research interests include architecture for next-generation DRAM and product development utilizing new process technology for new product line-up.

Author:

Yang Seok Ki

CXL Board of Director, VP and CTO of Memory Solutions Lab
Samsung Electronics

Dr. Yang Seok Ki is a Vice President and CTO of the Memory Solutions Lab (MSL) at Samsung Semiconductor Inc. in San Jose, California. Since joining Samsung in 2011, he has led various advanced development projects including SmartSSD, Key-Value SSD, CXL Memory Expander, Memory Semantic SSD, etc. In addition, he led the NVMe Key Value Standard, SNIA Key Value API, SNIA Computational Storage Architecture and API. He is a CXL board of director and a technical chair of Data Centric Computing workstream of the Open Computing Project (OCP) Future Technology Initiative (FTI). Prior to joining Samsung, he worked for Oracle's Server Technology Group. Prior to his industrial career, he was involved in High Performance Computing (HPC), Grid Computing, and Cloud research at the Institute of Information Sciences at the University of Southern California and the Center for Networked Systems at the University of California, San Diego. He received his Ph.D. in Electrical Engineering and Computer Engineering from Seoul National University, and his Master's and Bachelor's degrees in Computer Engineering from Seoul National University. He also completed the Engineering Leadership Professional Program (ELPP) from the University of California, Berkeley.

... read more

Yang Seok Ki

CXL Board of Director, VP and CTO of Memory Solutions Lab
Samsung Electronics

Dr. Yang Seok Ki is a Vice President and CTO of the Memory Solutions Lab (MSL) at Samsung Semiconductor Inc. in San Jose, California. Since joining Samsung in 2011, he has led various advanced development projects including SmartSSD, Key-Value SSD, CXL Memory Expander, Memory Semantic SSD, etc. In addition, he led the NVMe Key Value Standard, SNIA Key Value API, SNIA Computational Storage Architecture and API. He is a CXL board of director and a technical chair of Data Centric Computing workstream of the Open Computing Project (OCP) Future Technology Initiative (FTI). Prior to joining Samsung, he worked for Oracle's Server Technology Group. Prior to his industrial career, he was involved in High Performance Computing (HPC), Grid Computing, and Cloud research at the Institute of Information Sciences at the University of Southern California and the Center for Networked Systems at the University of California, San Diego. He received his Ph.D. in Electrical Engineering and Computer Engineering from Seoul National University, and his Master's and Bachelor's degrees in Computer Engineering from Seoul National University. He also completed the Engineering Leadership Professional Program (ELPP) from the University of California, Berkeley.

Large Language Models (LLMs) have revolutionized natural language processing but have posed significant challenges in training and inference due to their enormous memory requirements. In this talk, we delve into techniques and optimizations to mitigate memory constraints across the entire lifecycle of LLMs.

The first segment explores Memory Optimized LLM Training. We discuss Training challenges and cover different techniques under Parameter Efficient Fine Tuning (PEFT). like prompt tuning with LoRA, and adapters.

LLMs inference is more memory bound rather than compute bound, In this section we will explore inference optimizations mostly for transformer architectures like Paged Key-Value (KV) Cache, Speculative Decoding, Quantization, Inflight Batching strategies, Flash Attention, each contributing to enhanced inference speed and efficiency.

Finally, we explore the concept of Coherent Memory, and how it helps with Inference optimizations by KV Cache offloading and LoRA weight re-computation.

By illuminating these advancements, this talk aims to provide a comprehensive understanding of state-of-the-art memory optimization techniques for LLMs, empowering practitioners to push the boundaries of natural language processing further.

Systems Infrastructure/Architecture
AI/ML Compute

Author:

Arun Raman

Deep Learning Solutions Architect
NVIDIA

Arun Raman is an AI solution architect at NVIDIA, adept at navigating the intricate challenges of deploying AI applications across edge, cloud, and on-premises environments within the consumer Internet industry. In his current role, he works on the design of end-to-end accelerated AI pipelines, for consumer internet customers meticulously addressing preprocessing, training, and inference optimizations.  His experience extends beyond AI, having worked with distributed systems and multi-cloud infrastructure. He shares practical strategies and real-world experiences, empowering organizations to leverage AI effectively.

... read more

Arun Raman

Deep Learning Solutions Architect
NVIDIA

Arun Raman is an AI solution architect at NVIDIA, adept at navigating the intricate challenges of deploying AI applications across edge, cloud, and on-premises environments within the consumer Internet industry. In his current role, he works on the design of end-to-end accelerated AI pipelines, for consumer internet customers meticulously addressing preprocessing, training, and inference optimizations.  His experience extends beyond AI, having worked with distributed systems and multi-cloud infrastructure. He shares practical strategies and real-world experiences, empowering organizations to leverage AI effectively.

HBM
CXL
Interconnects
AI/ML Compute

Author:

Jin-Hyeok Choi

EVP, Memory Solution & Product Development
Samsung Electronics

Jin-Hyeok Choi leads Device Solution’s R&D – Memory division, which develops new memory technologies and enables memory products.

Jin-Hyeok joined Samsung Electronics in 2003 as a SoC design engineer, working on the development of mobile storage. From 2012 to 2019, he was in charge of the development team for controllers, a core component of SoCs based on NAND Flash. He developed and commercialized the world's first eMMC and UFS products, as well as various controllers for SATA/SAS/NVMe SSDs. He also developed the first-ever enterprise premium SSD with high endurance VNAND and has contributed significantly to the expansion of the storage market.

Jin-Hyeok received his B.S., M.S., and Ph. D. degrees in Electronics Engineering from Seoul National University in 1989, 1991, and 1996, respectively. He also studied low-power circuits at the University of Tokyo's Institute of Industrial Science.

... read more

Jin-Hyeok Choi

EVP, Memory Solution & Product Development
Samsung Electronics

Jin-Hyeok Choi leads Device Solution’s R&D – Memory division, which develops new memory technologies and enables memory products.

Jin-Hyeok joined Samsung Electronics in 2003 as a SoC design engineer, working on the development of mobile storage. From 2012 to 2019, he was in charge of the development team for controllers, a core component of SoCs based on NAND Flash. He developed and commercialized the world's first eMMC and UFS products, as well as various controllers for SATA/SAS/NVMe SSDs. He also developed the first-ever enterprise premium SSD with high endurance VNAND and has contributed significantly to the expansion of the storage market.

Jin-Hyeok received his B.S., M.S., and Ph. D. degrees in Electronics Engineering from Seoul National University in 1989, 1991, and 1996, respectively. He also studied low-power circuits at the University of Tokyo's Institute of Industrial Science.

Author:

SangJoon Hwang

Corporate EVP, Head of DRAM Product & Technology
Samsung Electronics

SangJoon Hwang received B.S, M.S., and Ph.D. degrees in electric engineering from the Korea University in 1994, 1996, and 2008, respectively.

He joined the Samsung Electronics, Hwaseong, South Korea in 1996, where he had successfully led a DRAM design group in 2014 and the Flash design team in 2017 as a Vice President and the Memory Product Planning team in 2019 as as a Senior Vice President. Through leading various backgounds from product planning to design, his experience enhances the overall quality of Samsung DRAM products.

 

Since 2023, he has been leading the DRAM Product & Technology of the Samsung memory division. His current research interests include architecture for next-generation DRAM and product development utilizing new process technology for new product line-up.

... read more

SangJoon Hwang

Corporate EVP, Head of DRAM Product & Technology
Samsung Electronics

SangJoon Hwang received B.S, M.S., and Ph.D. degrees in electric engineering from the Korea University in 1994, 1996, and 2008, respectively.

He joined the Samsung Electronics, Hwaseong, South Korea in 1996, where he had successfully led a DRAM design group in 2014 and the Flash design team in 2017 as a Vice President and the Memory Product Planning team in 2019 as as a Senior Vice President. Through leading various backgounds from product planning to design, his experience enhances the overall quality of Samsung DRAM products.

 

Since 2023, he has been leading the DRAM Product & Technology of the Samsung memory division. His current research interests include architecture for next-generation DRAM and product development utilizing new process technology for new product line-up.

Author:

Paul Turner

VP
Broadcom/VMWare

 

Paul Turner, is the Vice President of the vSphere Product Management team, covering vCenter, ESXi, vMotion and Project Pacific. He is leading our next generation of vSphere and moving the platform to become the leading infrastructure platform for all apps – VMs, Containers and Machine Learning applications. Paul brings more than 20 years expertise in enterprise software product management and marketing - having held leadership roles at VMware, NetApp, Oracle, Cloudian and Scality. Under his leadership, Scality was recognized as a leader in Gartner's Magic Quadrant and also by IDC in their Marketscape report for Object Storage. Prior to this at NetApp, he led the product management and technical marketing for their management software and also ran the Product Strategy Office, where he guided their investments into all-flash, Iongrid, CacheIQ, Onaro and Akorri.   

 

Paul holds an computer science degree from Trinity College in Ireland. He lives in Los Altos, Silicon Valley with his wife Kristy and their children Conor and Aoife.. 

 

 

 

... read more

Paul Turner

VP
Broadcom/VMWare

 

Paul Turner, is the Vice President of the vSphere Product Management team, covering vCenter, ESXi, vMotion and Project Pacific. He is leading our next generation of vSphere and moving the platform to become the leading infrastructure platform for all apps – VMs, Containers and Machine Learning applications. Paul brings more than 20 years expertise in enterprise software product management and marketing - having held leadership roles at VMware, NetApp, Oracle, Cloudian and Scality. Under his leadership, Scality was recognized as a leader in Gartner's Magic Quadrant and also by IDC in their Marketscape report for Object Storage. Prior to this at NetApp, he led the product management and technical marketing for their management software and also ran the Product Strategy Office, where he guided their investments into all-flash, Iongrid, CacheIQ, Onaro and Akorri.   

 

Paul holds an computer science degree from Trinity College in Ireland. He lives in Los Altos, Silicon Valley with his wife Kristy and their children Conor and Aoife.. 

 

 

 

Author:

Gunnar Hellekson

VP & GM
RedHat

Gunnar Hellekson is Vice President and General Manager for the Red Hat Enterprise Linux business. Before that, he was Chief Strategist for Red Hat’s US Public Sector group.  He is a founder of Open Source for America, one of Federal Computer Week’s Fed 100 for 2010, and was voted one of the FedScoop 50 for industry leadership. He was a founder of the Military Open Source working group, a member of the SIIA Software Division Board, the Board of Directors for the Public Sector Innovation Group, the Open Technology Fund Advisory Council, New America’s California Civic Innovation Project Advisory Council, and the CivicCommons Board of Advisors. He perks up when people talk about commoditization and the industrial mobilization of World War II. He is also co-host of the Dave and Gunnar Show.

Prior to joining Red Hat, he worked as a developer, systems administrator, and IT director for a number of Internet businesses. He has also been a business and IT consultant to not-for-profit organizations in New York City. During that time, he spearheaded the reform of safety regulations for New York State’s electrical utilities through the Jodie Lane Project.

Gunnar’s CV is available in HTML, PDF, and on GitHub.

 

... read more

Gunnar Hellekson

VP & GM
RedHat

Gunnar Hellekson is Vice President and General Manager for the Red Hat Enterprise Linux business. Before that, he was Chief Strategist for Red Hat’s US Public Sector group.  He is a founder of Open Source for America, one of Federal Computer Week’s Fed 100 for 2010, and was voted one of the FedScoop 50 for industry leadership. He was a founder of the Military Open Source working group, a member of the SIIA Software Division Board, the Board of Directors for the Public Sector Innovation Group, the Open Technology Fund Advisory Council, New America’s California Civic Innovation Project Advisory Council, and the CivicCommons Board of Advisors. He perks up when people talk about commoditization and the industrial mobilization of World War II. He is also co-host of the Dave and Gunnar Show.

Prior to joining Red Hat, he worked as a developer, systems administrator, and IT director for a number of Internet businesses. He has also been a business and IT consultant to not-for-profit organizations in New York City. During that time, he spearheaded the reform of safety regulations for New York State’s electrical utilities through the Jodie Lane Project.

Gunnar’s CV is available in HTML, PDF, and on GitHub.