If you’ve got multimodal datasets, Claude Code on terminal isn’t going to work — but we’ve got you. We’re excited to be at GTC announcing multimodal support in Hyperparam: images, audio and geospatial data. Drop a dataset on Hyperparam to check it out.

Apache Parquet added an Implementation Status page for parquet clients. But Hyparquet had a shameful red ❌ on Variant Encoding.

But JavaScript deserves first-class Parquet support! So I stayed up late, and now Hyparquet 1.24.0 has full Variant support ✅

Peer pressure works!

Now sponsoring bundlephobia on github. They are a doing a fantastic service for the JS community.

Browser-first data tools > Python-first. Hyperparam opens Parquet datasets in the browser with no backend, so your time-to-first-row is ~instant

#BurnTheBackEnd

Everyone working with LLM data has been there: thousands of conversations and no way to find the one that matters.

Existing data tools were not built for the massive scale of LLM data.

Hyperparam was built to solve this problem. Free while in beta starting 11/19.

Every chat log is a lesson in customer experience — but how do you analyze enough of them to make it meaningful? Tweak prompts, test different AI models, score for sycophancy, and explore everything blazing fast in the browser.

Fastest way to get insights from your data? Sort it. Don’t believe me? Go sort some data with Hyperparam.

If you can’t see your data, you’re just guessing.

Hyperparam pairs LLMs with a data UI so humans score, slice, and ship with confidence.

Cool parquet metadata visualizer by @J_, powered by Hyparquet

Use Hyperparam to annotate your chat logs with subjective labels (like sycophancy score...)

Hyparquet on its way to being the most-starred parquet reader for the browser (js, typescript, wasm)

Sick of depending on python and spark for your data lake? Wouldn't it be nice if you could browse your Iceberg tables using JAVASCRIPT?

No April Fools. We actually wrote an Apache Iceberg parser in 100% javascript. Checkout `icebird` on github!

Very satisfying to see the performance improvements in hyparquet 1.9.0!

30%+ speedups on many cases! (lower is better, log scale)

Hyparquet is the world's most compliant parquet parser! Hyparquet successfully opens more parquet files than even the arrow reference implementation.

hyparquet: 100%
arrow: 88%
duckdb: 80%
fastparquet: 58%
parquet-wasm: 78%
dsnp/parquetjs: 37%

"I came across a nice dependency-free library today, hyparquet, for parsing Parquet files in JavaScript.

It supports many compression formats including Snappy and Zstd, and reads Parquet files in chunks using range requests rather than loading the entire file. You can read just ular rows and columns.

It’s a good companion to @jeffrey_heer's flechette, a lightweight library for reading and writing Arrow files."

-- @yurivish

@severo_dev @TadViewer @dbeaver_news Works with @huggingface urls too!

View parquet files in the browser with "npx hyperparam"

Become one with the data. The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. This step is critical. - @karpathy

Published hyparquet v0.7.9 with an 11% speedup when parsing the TPCH benchmark dataset 🚀

PS- code interpreter rocks for evaluating the statistical significance of test runs!

hyllama achievement unlocked: 100% test coverage! 🦙💯

"We found that previous generations of llama are surprisingly good at identifying high-quality data"

Published hyparquet v0.7.4 to npm -- bug fixes and performance optimization for parquet files with big row groups.

There is no "good" way to concatenate js arrays:
- for is slow
- concat wastes memory
- a.push(...b) chrome fails at length 32k

Fastest concat js:

Every project needs a mascot!

Snappy Penguin

Why Snappy WebAssembly?

Because we're making the world's fastest Parquet parser in the browser.

Most parquet files use snappy compression, and about 60% of the runtime is spent on decompression. WASM can help here.

Documenting LLM training tricks:

How did LLMs improve at math? Better tokenizers helped. Imagine how hard it would be to learn math if numbers didn't have constant number-base across all digits?

LLaMA: base 10
GPT 3.5+: base 1000
GPT 3-: chaos