𝐃𝐞𝐭𝐞𝐜𝐭𝐢𝐨𝐧, 𝐫𝐞𝐦𝐞𝐝𝐢𝐚𝐭𝐢𝐨𝐧, 𝐜𝐡𝐞𝐜𝐤𝐩𝐨𝐢𝐧𝐭𝐢𝐧𝐠 — 𝐧𝐞𝐜𝐞𝐬𝐬𝐚𝐫𝐲, 𝐛𝐮𝐭 𝐧𝐨𝐭 𝐞𝐧𝐨𝐮𝐠𝐡 𝐟𝐨𝐫 𝐬𝐭𝐫𝐨𝐧𝐠 𝐠𝐨𝐨𝐝𝐩𝐮𝐭.

At AI Infra Summit, Clark Zinzow of @togethercompute laid out the four goodput pain points they see most:

• 𝑐𝑜𝑚𝑝𝑢𝑡𝑒 — even on high-performance scale-out storage systems, restoring can take minutes, while recomputing progress since the last checkpoint adds tens more.

• 𝐻𝑜𝑡-𝑠𝑝𝑎𝑟𝑒 𝑡𝑟𝑎𝑑𝑒𝑜𝑓𝑓𝑠 — always-on spares are expensive to hold idle; cold spares are too slow to pull into the cluster.

• 𝐶ℎ𝑒𝑐𝑘𝑝𝑜𝑖𝑛𝑡 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 — checkpoint more often to lose less, and the overhead eats into your goodput.

• 𝐺𝑟𝑎𝑦 𝑓𝑎𝑖𝑙𝑢𝑟𝑒𝑠 — every health check passes, yet the job quietly slows or stalls.

How we close all four is in Part 4, linked in the comments.

#AIInfraSummit