Nicolas Augusti

𝐃𝐚𝐲 7 𝐈𝐧 𝐆𝐞𝐧𝐀𝐈: 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐁𝐄𝐑𝐓'𝐬 𝐂𝐨𝐫𝐞 𝐚𝐧𝐝 𝐅𝐢𝐧𝐞-𝐓𝐮𝐧𝐢𝐧𝐠 Day 7 of my Generative AI journey was packed with insights as I wrapped up the BERT paper, "𝐁𝐄𝐑𝐓: 𝐏𝐫𝐞-𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐨𝐟 𝐃𝐞𝐞𝐩 𝐁𝐢𝐝𝐢𝐫𝐞𝐜𝐭𝐢𝐨𝐧𝐚𝐥 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫𝐬 𝐟𝐨𝐫 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠" by Devlin et al. I also delved into Umair Jamil’s video, "𝐁𝐄𝐑𝐓 𝐞𝐱𝐩𝐥𝐚𝐢𝐧𝐞𝐝: 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠, 𝐈𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞, 𝐁𝐄𝐑𝐓 𝐯𝐬 𝐆𝐏𝐓/𝐋𝐋𝐌, 𝐅𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠, [𝐂𝐋𝐒] 𝐭𝐨𝐤𝐞𝐧", which added another layer of clarity to my understanding. ( Check it out: https://lnkd.in/dGZg6mQw ) 𝐔𝐧𝐩𝐚𝐜𝐤𝐢𝐧𝐠 𝐭𝐡𝐞 𝐁𝐄𝐑𝐓 𝐏𝐚𝐩𝐞𝐫 The architecture and pre-training strategies in BERT are a masterclass in optimization and versatility. Here’s what stood out: → [𝐂𝐋𝐒] 𝐓𝐨𝐤𝐞𝐧: Used for sentence-level tasks, the token aggregates information from the input sequence and is pivotal for classification models. → 𝐌𝐚𝐬𝐤𝐞𝐝 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠 (𝐌𝐋𝐌): This clever approach of masking random tokens teaches BERT to predict context-sensitive words, enabling a bidirectional understanding of language. → 𝐍𝐞𝐱𝐭 𝐒𝐞𝐧𝐭𝐞𝐧𝐜𝐞 𝐏𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐨𝐧 (𝐍𝐒𝐏): A task that helps BERT understand sentence relationships, making it effective for tasks like Q&A and summarization. 𝐈𝐧𝐬𝐢𝐠𝐡𝐭𝐬 𝐟𝐫𝐨𝐦 𝐔𝐦𝐚𝐢𝐫 𝐉𝐚𝐦𝐢𝐥’𝐬 𝐕𝐢𝐝𝐞𝐨 Umair Jamil’s explanation bridged the gap between theory and practice. He not only demystified BERT’s architecture but also touched on: → 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐓𝐞𝐜𝐡𝐧𝐢𝐪𝐮𝐞𝐬: How large-scale datasets and compute are utilized for pre-training. → 𝐅𝐢𝐧𝐞-𝐓𝐮𝐧𝐢𝐧𝐠: The ease of adapting BERT for tasks like sentiment analysis and named entity recognition by tweaking the final layers. → 𝐂𝐨𝐦𝐩𝐚𝐫𝐢𝐬𝐨𝐧𝐬: Distinguishing BERT from GPT and other LLMs, emphasizing its pre-training methodology and bidirectional nature. And that was it for today 🤷‍♂️ 𝐖𝐡𝐚𝐭’𝐬 𝐍𝐞𝐱𝐭? Now it’s time to put theory into practice. I'll try implementing the BERT tomorrow. #GenAI #BERT_learnings

10

3 Comments

Sometimes I just can't share all my thoughts on LinkedIn. That's why I've launched my own newsletter : https://lnkd.in/eVukhBqB I've just published my first post, where I dive into what TL;DR is and how to craft them effectively : https://lnkd.in/eKTNkeiA Give it a read, and if you enjoy it, be sure to subscribe for more exciting content in the future! #newsletter #datascience

18

These days I’ve been in Montpellier, France, to present our last research results on combinatorial optimization problems over temporal graphs to the 31st symposium on temporal representation and reasoning (TIME2024). Check out our paper at https://lnkd.in/da66ref4 Here we discuss a new heuristic-based approach for minimum timeline cover over temporal graphs. The paper provides theoretical results to a problem largely adopted in social network analysis as interpretable network summarization framework to find important events. Particularly, we used it to analyze job postings, to find the most required skills by companies and understand how the market is evolving. We are already working on evolutions of this approach with Deep Learning. Stay tuned for next publications. #knowledgegraph #combinatorialoptimization #networksummarization

15

1 Comment

Fine-tuning LLMs: Aligning models with specific customer needs is crucial for real-world success, yet it requires precision to ensure the model adapts effectively without compromising quality. Fine-tuning often introduces risks, such as underfit or overfit layers, which can lead to unpredictable behavior in production. Tools like WeightWatcher are essential for addressing these challenges, offering a data-free diagnostic to evaluate model quality by analyzing the spectral properties of weight matrices. Grounded in the Heavy-Tailed Self-Regularization (HT-SR) theory, WeightWatcher provides insights into layer performance and potential issues, helping developers optimize instruction-tuned models like Llama3.1 and Falcon2 for robust, production-ready use cases. #InstructionFineTuning #LLMEngineering #ModelQuality #CustomerFocusedAI

2

𝐃𝐞𝐚𝐫 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐬 🔥 Ready for another PySpark Transformation Challenge? 📣 𝐈𝐧𝐟𝐨: You have a dataset with daily stock prices for various stocks. For each stock, you need to flag rows where the price has increased, decreased, or remained the same compared to the previous day. The first day for each stock should have a NULL value in the new column, as there is no previous day to compare. 𝐂𝐫𝐞𝐚𝐭𝐞 𝐒𝐚𝐦𝐩𝐥𝐞 𝐃𝐚𝐭𝐚𝐟𝐫𝐚𝐦𝐞 (Input Data): # 𝘚𝘢𝘮𝘱𝘭𝘦 𝘥𝘢𝘵𝘢 𝘥𝘢𝘵𝘢 = [   ("𝘈", "2024-01-01", 100),   ("𝘈", "2024-01-02", 105),   ("𝘈", "2024-01-03", 104),   ("𝘉", "2024-01-01", 200),   ("𝘉", "2024-01-02", 200),   ("𝘉", "2024-01-03", 201) ] # 𝘊𝘳𝘦𝘢𝘵𝘦 𝘋𝘢𝘵𝘢𝘍𝘳𝘢𝘮𝘦 𝘥𝘧 = 𝘴𝘱𝘢𝘳𝘬.𝘤𝘳𝘦𝘢𝘵𝘦𝘋𝘢𝘵𝘢𝘍𝘳𝘢𝘮𝘦(𝘥𝘢𝘵𝘢, ["𝘴𝘵𝘰𝘤𝘬_𝘪𝘥", "𝘥𝘢𝘵𝘦", "𝘱𝘳𝘪𝘤𝘦"]) 𝐏𝐲𝐒𝐩𝐚𝐫𝐤 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧 𝐂𝐨𝐝𝐞? (Expected Data) 𝘥𝘧_𝘵𝘳𝘢𝘯𝘴𝘧𝘰𝘳𝘮𝘦𝘥 = ????? Let me know the answer in the comments 😀 #pyspark #dataengineering

72

12 Comments

Hugging Face x LangChain : A new partner package in LangChain - We are thrilled to announce the launch of langchain_huggingface, a partner package in LangChain jointly maintained by Hugging Face and LangChain. This new Python package is designed to bring the power of the latest development of Hugging Face into LangChain and keep it up to date. langchain-huggingface integrates seamlessly with LangChain, providing an efficient and effective way to utilize Hugging Face models within the LangChain ecosystem. This partnership is not just about sharing technology but also about a joint commitment to maintain and continually improve this integration. Getting started with langchain-huggingface is straightforward. Here’s how you can install and begin using the package: pip install langchain-huggingface #llm #huggingface #langchain #genai #generativeai #artificialintelligence

25

🚀 Mistral AI: Pixtral Large and Le Chat Platform Introducing Pixtral Large: Mistral AI, the French AI startup, has launched Pixtral Large, a groundbreaking 124-billion parameter multimodal model designed to process both text and visual data. - 🧠 Core Architecture: Combines a 123-billion parameter text decoder with a 1-billion parameter vision encoder. - 📖 Expanded Context Window: Supports up to 128K tokens, enabling it to handle 30 high-resolution images or a 300-page book. - 🏆 Benchmark Leader: Outperforms GPT-4o, Gemini 1.5 Pro, and Claude-3.5 Sonnet on key benchmarks like MathVista (69.4% accuracy), ChartQA, and DocVQA. Le Chat Platform: Unlocking New Possibilities Mistral’s Le Chat platform integrates Pixtral Large, introducing innovative features for diverse applications: -🔍 Web Search with Citations: Real-time data retrieval with source transparency. -🖋️ Canvas: Tools for live document creation, collaborative editing, and version control. -📄 Advanced OCR: Efficiently processes PDFs, tables, and equations to extract insights. - 🎨 Image Generation: Powered by Flux Pro from Black Forest Labs. - 🤖 Task Agents: Automates tasks like summarization and invoice processing. - 🌟 Performance Highlights Pixtral Large sets new standards on benchmarks like ChartQA, DocVQA, and VQAv2, showcasing cutting-edge capabilities in document analysis and visual data interpretation. The model is available under the Mistral Research License for non-commercial use, with commercial licenses for enterprise adoption. 📢 Additional Updates: Mistral AI also unveiled Mistral Large 2.1, a 123-billion parameter model optimized for general-purpose tasks, along with enhanced APIs to streamline developer integration.

6

🚀 I’m happy to announce that our paper “E-commerce product categorization with LLM-based dual-expert classification paradigm” has been accepted at EMNLP 2024 Customizable NLP! Thanks to the co-authors Zhu Cheng, Vincent (Wen) Zhang, Chih-Chi Chou, You-Yi Jau, Archita Pathak, Penny Gao, Aziz Umit Batur for collaboration! We present a dual-expert classification system that utilizes the power of large language models (LLMs) for accurate product classification. This framework integrates domain-specific knowledge and pre-trained LLM’s general knowledge through effective model fine-tuning and prompting techniques. First, the fine-tuned domain-specific expert recommends top K candidate categories for a given input product. Then, the more general LLM-based expert, through prompting techniques, analyzes the nuanced differences between candidate categories and selects the most suitable target category. We introduce a new in-context learning approach that utilizes LLM self-generated summarization to provide clearer instructions and enhance its performance. Experiments on e-commerce datasets demonstrate the effectiveness of our LLM-based Dual-Expert classification system. If you happen to be at EMNLP, feel free to check out our work on 11/16 Saturday 11:00 AM - 12:30 PM! #amazon #amazonscience #LLM #LargeLanguageModel #classification #categorization #GenAI #ecommerce

61

4 Comments

🚨 Just out - a new take on space control! ⚽ In this paper, we started out with a simple question: Why do data models always treat space as a ressource shared across all 22 players? To challenge this perspective, we investigated how a single team is managing space on the pitch. How is a team allocating space internally? Is triangular positioning the name of the game? And how do attackers switch between marking assignments? Looking at ~130.000 attacks we found how clever positioning and disruption of marking assingments is related to success ✔️ 👉 https://lnkd.in/eVC-TXmA

113

Check out my latest talk on "Peak Performance in Python" at this year's PyConDE Open to feedback!

19

𝐁𝐫𝐢𝐥𝐥𝐢𝐚𝐧𝐭 𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧 𝐟𝐨𝐫 𝐢𝐧𝐭𝐞𝐫𝐚𝐜𝐭𝐢𝐯𝐞 𝐚𝐧𝐚𝐥𝐲𝐬𝐢𝐬 𝐚𝐧𝐝 𝐌𝐋 𝐚𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬 The architecture is simply brilliant - e2b runtime - LLM middle layer- - notebook style rendering Massive applications. - fully scalable. - open source - customize - build - integrate Next up - prompt-driven iterative analytics and machine learning. Now exploring ways to incorporate similar solution into REX (rex.tigzig.com / realtime.tigzig.com - my open source analytics toolkit), to provide interactive analysis and machine learning. Check out the video demo on X (I actually saw it on X first): https://lnkd.in/gQXqMRiZ App & Code base : https://lnkd.in/eGmyXh-q

2

Forget about LLMs Ranking Leaderboards🥇, Welcome to the Era of Model Routing 🔀! This shift in the market is truly transformative. The idea of dynamically selecting the most suitable model for each specific task is a game-changer. It's like having a garage full of specialized vehicles, each ready to take on a different terrain. Whether it's the speed of a Ferrari 🏎 for complex technical analyses or the reliability of a Range Rover 🚗 for varied tasks, model routing ensures the optimal performance for every scenario. Amazon Q for Business is a perfect example, dynamically choosing the best model based on query complexity to maximize user experience. This approach heralds a future where model selection is flexible, task-specific, and efficient, marking a significant departure from static model use. Exciting times ahead in the world of generative AI! #AI #MachineLearning #ModelRouting #Innovation #TechRevolution #FutureOfAI #GenerativeAI #SmartTech 🚀🔍🧠

1

It works on any subject/topic ! 😳 Who said synthetic data are not easy / useful ? 💥 Search + Save Datasets generated by a LLM in real time 💥 Try my new app on Hugging Face : https://lnkd.in/epvgZcvD The LLM invents dataset names and themes and generates full datasets in just one click ! You can use this app to bootstrap all sorts of machine learning projects, whether it is to train a language model or small machine learning models. Let me know what you think ! (btw, of course it's all free and open source) Thanks for the gradio tips Abubakar Abid Freddy Boulton and Vishrav Chaudhary and team for Phi-3 mini #synthetic #data #datasets #llm #opensource

441

25 Comments

Everyone is diving into learning about LLMs, and this is great because it's leading to the creation of exceptional resources. . . . #mlops #ai #LanguageModel #MachineLearning #DeepLearning #ArtificialIntelligence #TechInnovation #LLM

8

Create table ##stock (stock_id varchar(10), date date, price int) Insert into ##stock values  ('A', '2024-01-01', 100),  ('A', '2024-01-02', 105),  ('A', '2024-01-03', 104),  ('B', '2024-01-01', 200),  ('B', '2024-01-02', 200),  ('B', '2024-01-03', 201)  with cte as  (  select * , LAG(price)over(partition by stock_id order by date)lprice from ##stock  )  select stock_id,date,price ,   case when lprice IS null then 'NULL'  when price>lprice then 'UP'  when price<lprice then 'Down'  else 'Same' end price_change  from cte

5

1 Comment

Want to test Meta, Google, and Mistral models all on one platform ? **Groq.com** Why Groq? Because speed matters. GroqCloud supports a diverse array of models, enabling you to select the best for your needs: ➡ llama3-8b-8192 by Meta (Context Window: 8,192 tokens) ➡ llama3-70b-8192 by Meta (Context Window: 8,192 tokens) ➡ mixtral-8x7b-32768 by Mistral (Context Window: 32,768 tokens) ➡ gemma-7b-it by Google (Context Window: 8,192 tokens) Groq's commitment to GenAI inference speed is transforming real-time AI applications with its LPU. What is an LPU? ➡ Language Processing Unit, is a revolutionary processing system designed specifically for LANGUAGE applications. Unlike traditional CPUs (Central Processing Units) and GPUs (Graphics Processing Units), which handle a broad range of tasks. LPUs are optimized for the rapid processing demands of LLMs This specialized architecture allows for more efficient handling of sequential data, significantly reducing the time per word processed. ➡ That means you get faster outputs You can run Llama-2 70B at over 300 tokens per second per user!

24

How to handle private data in ML datasets ? Here is out latest exciting experiment at Hugging Face 🤗 : Many datasets contain undocumented private information, making them difficult for ML practitioners to use. We started to use #Presidio to detect the presence of private data in HF Datasets, in particular Personal Identifiable Information (#PII). We now show a Presidio Report that estimates the amount of PII on dataset pages. This is useful e.g. to help further filter datasets or to take more informed decisions before training a model. More info on Presidio + examples in our latest blog post: https://lnkd.in/e5_mt7j7 We built this with the help of the community (special thanks Omri Mendels for the help with Presidio, Sara Hooker for the feedback on multilingual datasets, Margaret Mitchell and team for the help on the blog post !) and from our work with CNIL - Commission Nationale de l'Informatique et des Libertés and their guidance.

51