Data-driven design for the Dask scheduler
Historically, changes in the scheduling algorithm of Dask have often been based on theory, single use cases, or even gut feeling. Coiled has now moved to using hard, comprehensive performance metrics for all changes - and it's been a turning point!
I come from a 12 years career in orchestrating Monte Carlo simulations for finance, sized at 1500+ CPU hours each. For the last two years I've been an OSS engineer at Coiled, building up the foundations of the Dask library.
visit the speaker at: Github