pre-training

The official GitHub page for the survey paper "A Survey of Large Language Models".

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo

Awesome resources for in-context learning and prompt engineering: Mastery of the LLMs such as ChatGPT, GPT-3, and FlanT5, with up-to-date and cutting-edge updates.

Papers about pretraining and self-supervised learning on Graph Neural Networks (GNN).

Unified Training of Universal Time Series Forecasting Transformers

Code for TKDE paper "Self-supervised learning on graphs: Contrastive, generative, or predictive"

An Open-sourced Knowledgable Large Language Model Framework.

Awesome list for research on CLIP (Contrastive Language-Image Pre-Training).

A professional list on Large (Language) Models and Foundation Models (LLM, LM, FM) for Time Series, Spatiotemporal, and Event Data.

Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo

Oscar and VinVL

Pre-training of Deep Bidirectional Transformers for Language Understanding: pre-train TextCNN

Research code for ECCV 2020 paper "UNITER: UNiversal Image-TExt Representation Learning"

Large Language Model-enhanced Recommender System Papers

Code for ICLR 2020 paper "VL-BERT: Pre-training of Generic Visual-Linguistic Representations".

The repository of ET-BERT, a network traffic classification model on encrypted traffic. The work has been accepted as The Web Conference (WWW) 2022 accepted paper.

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

[ICLR 2024] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

[NeurIPS 2020] "Graph Contrastive Learning with Augmentations" by Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, Yang Shen