Recent developments in machine learning research have the potential to create a lasting impact in academic research. From associative memory mechanisms to large language models, researchers are pushing the boundaries of what is possible with machine learning. This newsletter will present recent developments in machine learning research, with a focus on potential breakthroughs. This paper presents a model for associative memory mechanisms based on high-dimensional matrices. It provides precise scaling laws and statistical efficiency of different estimators, and is supported by extensive numerical experiments. The potential for these techniques to create a lasting impact in academic research is clear, as they can help to better understand and optimize learning processes. This paper presents experiments to benchmark the computational and energy costs of large language model inference. Results show potential for cost-savings, scaling performance, efficient hardware usage, and optimal inference strategies. The findings of this paper could have a lasting impact on academic research of LLMs and their usage in various domains. ECoFLaP is a two-stage coarse-to-fine layer-wise pruning approach for large Vision-Language Models (LVLMs) that can
This paper presents a model for associative memory mechanisms based on high-dimensional matrices. It provides precise scaling laws and statistical efficiency of different estimators, and is supported by extensive numerical experiments. The potential for these techniques to create a lasting impact in academic research is clear, as they can help to better understand and optimize learning processes.
This paper presents experiments to benchmark the computational and energy costs of large language model inference. Results show potential for cost-savings, scaling performance, efficient hardware usage, and optimal inference strategies. The findings of this paper could have a lasting impact on academic research of LLMs and their usage in various domains.
ECoFLaP is a two-stage coarse-to-fine layer-wise pruning approach for large Vision-Language Models (LVLMs) that can efficiently compress model weights while maintaining performance. It leverages global importance scores to determine sparsity ratios, and local layer-wise unstructured weight pruning to achieve significant performance improvements in the high-sparsity regime. This technique has the potential to create a lasting impact in academic research by enabling efficient deployment of LVLMs.
This paper demonstrates that Transformers and LLMs can learn to learn discrete functions, leading to potential lasting impacts in academic research. Results show that Transformers can nearly match the optimal learning algorithm for simpler tasks, while their performance decreases on more complex tasks. Additionally, Transformers can learn to implement two distinct algorithms to solve a single task, and can adaptively select the more sample-efficient algorithm. Lastly, LLMs can compete with nearest-neighbor baselines on prediction tasks.
xVal is a numerical encoding scheme that uses a single token to represent any real number, allowing large language models to be adapted for the analysis of scientific datasets. This strategy renders the model end-to-end continuous, leading to improved generalization and token-efficiency, with potential to create a lasting impact in academic research.
This paper demonstrates that pretraining with data-driven priors can lead to dramatic gains in performance across multiple architectures, and reduce the gap between Transformers and state space models. This has the potential to create a lasting impact in academic research, as it provides a reliable way to evaluate different architectures on supervised tasks.
This paper presents a novel approach to combining retrieval and long context large language models (LLMs) to achieve better performance on downstream tasks. Results show that retrieval-augmented LLMs with 4K context window can outperform finetuned LLMs with 16K context window, while taking less computation. The best model outperforms GPT-3.5-turbo-16k and Davinci003 on seven long context tasks, providing a potential for lasting impact in academic research.
This paper proposes using Large Language Models (LLMs) as decision makers for autonomous driving, providing improved safety, efficiency, generalizability, and interoperability. Experiments demonstrate that LLMs consistently outperform baseline approaches, and can handle complex driving behaviors even in multi-vehicle coordination scenarios. This could have a lasting impact in academic research of autonomous driving.
This paper presents a new attack, Shadow Alignment, which can subvert safely-aligned language models with only a small amount of data. This attack has the potential to create a lasting impact in academic research by demonstrating the need for improved safety measures for open-source language models.
This paper presents a novel technique, Mixture of Prompts (MoPs), to adapt large language models to heterogeneous tasks and data distributions. MoPs are associated with smart gating functionality to identify relevant skills and dynamically assign combined experts. Results show that MoPs can reduce perplexity by up to 70% in federated scenarios and up to 30% in centralized scenarios, demonstrating the potential for lasting impact in academic research.