Give me some time
🤗🤗

See that purple banner on the Llama 4 models? It's Xet storage, and this is actually huge for anyone building with AI models. Real numbers: ~25% deduplication on Llama 4 models, hitting ~40% for finetunes

Because of this, different repositories can share bytes we store. That opens up something cool - we can draw a graph of which repos actually share data at the chunk level, where:

- Nodes = repositories
- Edges = shared chunks
- Edge thickness = how much they overlap

When you upload a file through Xet, the contents are split into ~64KB chunks and deduplicated, but what if those same chunks already exist in another repo on the Hub?

As 📷 xet-team infrastructure begins backing hundreds of repositories on the Hugging Face Hub, we’re getting to put on our researcher hats and peer into the bytes. 👀 🤓

IMO, one of the most interesting ideas Xet storage introduces is a globally shared store of data.

Need to convert CSV to Parquet?

Use chatdb.ai/tools/csv-to-p…. It does the job instantly.

@cfahlgren1 provides many other tools on his website. Approved and bookmarked!