Demos from the September community meeting are now up on YouTube. See how Dask is being used for scalable computing in Astronomy and the latest updates from dask-jobqueue.
youtu.be/afmus2EvccU
youtu.be/afmus2EvccU
0
0
3
Dask DataFrame is now 20x faster. Some of most prominent changes include:
- Apache Arrow support in @pandas_dev
- Better shuffling algorithm for faster joins
- Automatic query optimization
Learn more: docs.coiled.io/blog/dask-data…
- Apache Arrow support in @pandas_dev
- Better shuffling algorithm for faster joins
- Automatic query optimization
Learn more: docs.coiled.io/blog/dask-data…
0
8
30
We had some great talks at last month's Dask Demo Day like dask on @databricks, scaling embedding pipelines with @llama_index and deploying @PrefectIO on the cloud.
Join the next one this Thursday, 11am ET github.com/dask/community…
Watch the recording: youtube.com/watch?v=07e1JL…
Join the next one this Thursday, 11am ET github.com/dask/community…
Watch the recording: youtube.com/watch?v=07e1JL…
0
1
7
Thanks all for joining! The recording is up on youtube: youtube.com/watch?v=wkQzVN…
0
1
10
Thanks all for joining! The recording is up on the Dask YouTube channel youtube.com/watch?v=5YXiRl…
0
0
1
The July Dask Demo Day recording is up, we had great talks from the community. Go check it out.
youtu.be/dTsTkRD6rbY
youtu.be/dTsTkRD6rbY
0
1
4
Latest recording for the Dask Demo Day is up.
Topics where:
- Dask-GeoPandas
- Gil montoring in Dask
- FIne performance dask metrics
- Dask spans
youtube.com/watch?v=eUnvdA…
Topics where:
- Dask-GeoPandas
- Gil montoring in Dask
- FIne performance dask metrics
- Dask spans
youtube.com/watch?v=eUnvdA…
1
3
5
In the next release of dask distributed we will be renaming the dask-worker-space temporary directory to dask-scratch-space to better reflect that it is used by more than just workers.
Prepare to update your .gitignore files!
Prepare to update your .gitignore files!
0
0
1
April Dask Demo Day recording is up. We had great talks, check them out! youtube.com/watch?v=7ja_YH…
0
1
7
March's Dask Demo day recording! Incredible talks!
youtube.com/watch?v=-clDr7…
Want to present at one of the upcoming Dask Demo Days?
Sign up here:
github.com/dask/community…
youtube.com/watch?v=-clDr7…
Want to present at one of the upcoming Dask Demo Days?
Sign up here:
github.com/dask/community…
0
4
3
Learn how to use Dask and Optuna for Hyper Parameter Optimization from @CoiledHQ
youtube.com/watch?v=euT6_h…
youtube.com/watch?v=euT6_h…
0
1
7
Video for February's Dask Demo Day: youtube.com/watch?v=R0Hdnh…
Want to present in one of the upcoming Dask Demo Days? Sign up here: github.com/dask/community…
Want to present in one of the upcoming Dask Demo Days? Sign up here: github.com/dask/community…
0
2
1
It is now possible to manage Dask workloads using @flyteorg 🎉!
🖥 Flyte tasks spin up their own ephemeral Dask cluster
🏢 Uses existing Kubernetes infrastructure
💰 Spot/Preemtible instances are natively supported
📦 The whole Dask task can be cached
blog.dask.org/2023/02/13/das…
🖥 Flyte tasks spin up their own ephemeral Dask cluster
🏢 Uses existing Kubernetes infrastructure
💰 Spot/Preemtible instances are natively supported
📦 The whole Dask task can be cached
blog.dask.org/2023/02/13/das…
0
7
16
Video for this month's demo day available here:
youtu.be/_x7oaSEJDjA
Want to learn about the next one? We have a newsletter now at newsletter+subscribe@dask.org
youtu.be/_x7oaSEJDjA
Want to learn about the next one? We have a newsletter now at newsletter+subscribe@dask.org
Dask Demo Day this Thursday!
10am US Central
Quick demos on ...
1. 2,000,000,000 lightning strikes by @nywxguy
2. Dask Awkward by @ Mx_ddavis
3. Dask CLI by @ Mx_ddavis
4. Profiling with PySpy by Gabe Joseph
5. XGBoost and Optuna by @__jrbourbeau__
github.com/dask/community…
0
3
4
Video for this month's demo day is here: youtu.be/_x7oaSEJDjA
Dask Demo Day this Thursday!
10am US Central
Quick demos on ...
1. 2,000,000,000 lightning strikes by @nywxguy
2. Dask Awkward by @ Mx_ddavis
3. Dask CLI by @ Mx_ddavis
4. Profiling with PySpy by Gabe Joseph
5. XGBoost and Optuna by @__jrbourbeau__
github.com/dask/community…
0
0
0
The Dask Kubernetes Operator is now generally available 🎉!
📦 Clusters are now native custom resources
💻 Manage clusters with kubectl or Python
🐍 Reuse clusters between Python processes
🎁 and much more...
Check out the full write up blog.dask.org/2022/11/09/das…
📦 Clusters are now native custom resources
💻 Manage clusters with kubectl or Python
🐍 Reuse clusters between Python processes
🎁 and much more...
Check out the full write up blog.dask.org/2022/11/09/das…
0
4
16
@bsmithlite @_JacobTomlinson @danrothenberg @mrocklin @TomAugspurger @FlorianJetter @natt941 @al_merose Video! youtu.be/VlTgcLqb1DQ
0
1
1
Dask Demo Day this Thursday!
Quick demos on ...
1. Scalable Geoscience by @TomAugspurger
2. Task Queuing by @FlorianJetter
3. Prometheus by @natt941
4. Kubernetes updates by @_JacobTomlinson
5. Apache Beam by @al_merose
10am US Central
github.com/dask/community…
0
0
1
![from collections import namedtuple
import dask
a = dask.delayed(5)
b = dask.delayed(5)
Point = namedtuple("Point", ["x", "y"])
p = Point(a, b)
dask.compute(p)
# Output: (Point(x=5, y=5),)](/pic/orig/media%2FFa8iAt2UUAA-uxg.jpg)
High-cardinality groupby aggregations on Dask DataFrames are much more performant. Thank you, @_rjzamora!
They now use a shuffle-based algorithm, learn more: github.com/dask/dask/pull…
/2
They now use a shuffle-based algorithm, learn more: github.com/dask/dask/pull…
/2
2
1
9
The Dask team released version 2022.8.1 last week!
It includes contributions from 21 people, including 2 new contributors.
Some release highlights: 🧵
/1
It includes contributions from 21 people, including 2 new contributors.
Some release highlights: 🧵
/1
![Install from conda-forge: conda install -c conda-forge dask
Install from PyPI: python -m pip install "dask[complete]"](/pic/orig/media%2FFa8iBwSVsAILCXx.jpg)
2
3
34
Dask Clusters have a new '.get_client()' method -- It returns the connected client, or initializes a new one if no client is connected.
Thanks for adding this, @JSignell!
/4
Thanks for adding this, @JSignell!
/4
1
0
3
Dask now calculates the 'npartitions' and 'divisions' more accurately+intuitively while creating a DataFrame using 'dd.from_pandas()'.
Thanks for revising this logic, @_rjzamora!
/3
Thanks for revising this logic, @_rjzamora!
/3
![import dask.dataframe as dd
import pandas as pd
df = pd.DataFrame(
{
"x": [1, 2, 3],
"y": ["a", "b", "c"],
}
)
ddf = dd.from_pandas(df, npartitions=2)
# BEFORE
ddf.npartitions # Output: 1
ddf.divisions # Output: (0, 2)
# DASK>=2022.8.0
ddf.npartitions # Output: 2
ddf.divisions # Output: (0, 2, 2)](/pic/orig/media%2FFZ6NoeWaUAE1FVo.jpg)
1
0
4
You can now start a Jupyter server in the Dask Scheduler process. ✨
Thank you @mrocklin, @Mbussonn, and @zrsailer for working on this!
For more details, see: github.com/dask/distribut…
/2
Thank you @mrocklin, @Mbussonn, and @zrsailer for working on this!
For more details, see: github.com/dask/distribut…
/2
1
0
9
Sharing a quick win:
We noticed an increase in worker memory usage in a benchmark a few days ago, and merged a fix yesterday. Thanks, Guido Imperiale!
Learn how Dask handles memory usage in tricky situations: distributed.dask.org/en/stable/work…
Relevant issue: github.com/dask/distribut…
We noticed an increase in worker memory usage in a benchmark a few days ago, and merged a fix yesterday. Thanks, Guido Imperiale!
Learn how Dask handles memory usage in tricky situations: distributed.dask.org/en/stable/work…
Relevant issue: github.com/dask/distribut…
0
1
11
We have an awesome community blog post from folks at @metoffice, @ECMWF and @eumetsat around keeping your computations close to your data when you have data in multiple datacenters or cloud platforms that you want to include in a single calculation.
blog.dask.org/2022/07/19/das…
blog.dask.org/2022/07/19/das…
1
19
41
Dask has a new show_versions() function that helps share all the important package versions while reporting bugs. Thanks for working on this, Sultan Orazbayev!
/4
/4
1
0
3
We can now document known inconsistencies between Dask APIs and pandas/NumPy more easily, thanks to @richardpelgrim!
Help us document more inconsistencies here: github.com/dask/dask/issu…
/3
Help us document more inconsistencies here: github.com/dask/dask/issu…
/3
1
1
4
Dask now works in Pyodide and PyScript!
Thank you @IanRRose for adding preliminary support for Dask’s single machine threaded and multiprocessing schedulers!
/2
Thank you @IanRRose for adding preliminary support for Dask’s single machine threaded and multiprocessing schedulers!
/2
![The following code running in JupyterLite with the task graph visualization:
import piplite
await piplite.install("dask[array]")
await piplite.install("ipycytoscape")
await piplite.install("jinja2")
import dask.array as da
x = da.random.random(size=(100,100), chunks=10)
y = x + x.T
y.visualize()](/pic/orig/media%2FFWRo6dqacAAH6pC.jpg)
1
0
14
The Dask team released version 2022.6.1 on Friday!
It includes contributions from 21 people, including 4 new contributors.
Some release highlights: 🧵
/1
It includes contributions from 21 people, including 4 new contributors.
Some release highlights: 🧵
/1
![Install from conda-forge: conda install -c conda-forge dask
Install from PyPI: python -m pip install "dask[complete]"](/pic/orig/media%2FFWRnbsVakAAoW09.jpg)
1
4
34
Dask’s high-level graph representations in Jupyter notebooks now include layer dependencies, thank you Angelos Omirolis for working on this!
/4
/4
1
0
3
All the Dask documentation sites have a fresh new look, aligned with our new brand! ✨
Thank you, @scharlottej13!
Check it out: docs.dask.org/en/stable/
/3
Thank you, @scharlottej13!
Check it out: docs.dask.org/en/stable/
/3
1
0
5
The Dask team released version 2022.6.0 last week!
It includes contributions from 22 people, including 5 new contributors.
Some release highlights: 🧵
/1
It includes contributions from 22 people, including 5 new contributors.
Some release highlights: 🧵
/1
![Install from conda-forge: conda install -c conda-forge dask
Install from PyPI: python -m pip install "dask[complete]"](/pic/orig/media%2FFVOt2P_VEAA-2Tx.jpg)
1
4
30
As part of the @dask_dev rebrand we have refreshed our YouTube channel youtube.com/c/Dask-dev. Here is a new video from a recent event @odsc youtube.com/watch?v=Uza3k_…
0
2
2
We now have two new ways to create Dask DataFrames!
- from_map() to create DataFrames using a custom function map, and
- from_dict() to create DataFrames from Python dictionaries.
Thanks, @_rjzamora and @_mrpowers_ for adding these!
- from_map() to create DataFrames using a custom function map, and
- from_dict() to create DataFrames from Python dictionaries.
Thanks, @_rjzamora and @_mrpowers_ for adding these!
![import pandas as pd
import dask.dataframe as dd
func = lambda x, size=0: pd.Series([x] * size)
inputs = ["A", "B"]
dd.from_map(func, inputs, size=2).compute()
# 0 A
# 1 A
# 0 B
# 1 B
# dtype: object](/pic/orig/media%2FFTx7BLNaIAAaien.jpg)
![import dask.dataframe as dd
dd.DataFrame.from_dict(
{"num1": [1, 2, 3, 4], "num2": [7, 8, 9, 10]}, npartitions=2
).compute()
# num1 num2
# 0 1 7
# 1 2 8
# 2 3 9
# 3 4 10](/pic/orig/media%2FFTx7RKAaUAAsmbu.jpg)
2
0
10
The Dask team released versions 2022.05.1 and 2022.05.2 this week!
These include contributions from 33 people, including 7 new contributors.
Some release highlights: 🧵
These include contributions from 33 people, including 7 new contributors.
Some release highlights: 🧵
![Install from conda-forge: conda install -c conda-forge dask
Install from PyPI: python -m pip install "dask[complete]"](/pic/orig/media%2FFTx57HUaUAEQIQ5.jpg)
1
5
40
The distributed dashboard has some new plots showing cluster-wide bandwidths for memory, disk, and network. It also has a new plot tracking event loop health. Thanks @mrocklin!
1
1
2
The Dask team released version 2022.4.0 today! It includes contributions from 22 people, including 3 new contributors.
Some release highlights 🧵:
Some release highlights 🧵:
2
7
48
Other doc improvements include better API docs of string, categorical, and datetime accessors for DataFrames, and updated explanations of how Dask’s distributed scheduler assigns tasks to workers. Thanks to @scharlottej13, @jcristharif, keewis, and Gabe Joseph!
1
0
4
There have been a number of improvements to documentation. There is a new new page documenting the interactive Dask dashboard thanks to @ncclementi
1
0
2
The Dask team is happy to announce the release of 2022.02.1 today!
docs.dask.org/en/stable/chan…
Some highlights 🧵:
docs.dask.org/en/stable/chan…
Some highlights 🧵:
1
4
36
Are you deploying Dask on Kubernetes with the Helm Chart?
If so you can now configure multiple worker groups with different hardware configurations.
This enables running high memory or GPU workers and using annotations to run specific tasks on them.
blog.dask.org/2022/02/17/hel…
If so you can now configure multiple worker groups with different hardware configurations.
This enables running high memory or GPU workers and using annotations to run specific tasks on them.
blog.dask.org/2022/02/17/hel…
0
5
22
Dask has new documentation on visualizing High Level Graphs, check it out here: docs.dask.org/en/latest/grap…
Thanks for this, @DataNerdery!
Thanks for this, @DataNerdery!
1
0
2
"FugueSQL - The Enhanced SQL Interface for Pandas, Spark, and Dask DataFrames" by Chengxuan Wang
Video: youtube.com/watch?v=OBpnGY…
Description: We will introduce FugueSQL, an enhanced SQL interface that allows SQL lovers to express end-to-end workflows
pydata.org/global2021/sch…
8/9
Video: youtube.com/watch?v=OBpnGY…
Description: We will introduce FugueSQL, an enhanced SQL interface that allows SQL lovers to express end-to-end workflows
pydata.org/global2021/sch…
8/9
1
1
5
"Snowflake & Dask: How to scale workloads using distributed fetch capabilities" by Miles Adkins, James Bourbeau, Mark Keller
Video: youtube.com/watch?v=WJ5yam…
Description: We will cover how Snowflake, a cloud data solution, integrates with Dask
pydata.org/global2021/sch…
7/9
Video: youtube.com/watch?v=WJ5yam…
Description: We will cover how Snowflake, a cloud data solution, integrates with Dask
pydata.org/global2021/sch…
7/9
1
0
1
"Getting started with Dask using Saturn Cloud" by Mitali Sanwal
Video: youtube.com/watch?v=G303aR…
Description: In this talk we'll show you how to harness the power of Dask with Saturn Cloud.
pydata.org/global2021/sch…
6/9
Video: youtube.com/watch?v=G303aR…
Description: In this talk we'll show you how to harness the power of Dask with Saturn Cloud.
pydata.org/global2021/sch…
6/9
1
0
2
"Dask for Everyone" by Hugo Bowne-Anderson
Video: youtube.com/watch?v=nsYPez…
Description: This talk describes Coiled, a company based around Dask, and how it strives to enable the use of Dask by everyone, everywhere.
pydata.org/global2021/sch…
5/9
Video: youtube.com/watch?v=nsYPez…
Description: This talk describes Coiled, a company based around Dask, and how it strives to enable the use of Dask by everyone, everywhere.
pydata.org/global2021/sch…
5/9
1
1
4
"Dask: From POC to Production" by April Rathe from Arrowstreet Capital
Video: youtube.com/watch?v=5KN5aB…
Description: In this talk, we’ll describe our experiences integrating Dask into our environment. We'll provide a roadmap of decisions, development..
pydata.org/global2021/sch…
4/9
Video: youtube.com/watch?v=5KN5aB…
Description: In this talk, we’ll describe our experiences integrating Dask into our environment. We'll provide a roadmap of decisions, development..
pydata.org/global2021/sch…
4/9
1
1
2
"Spatial Analytics using Dask & Numba" by Brendan Collins
Video: youtube.com/watch?v=9PufD9…
Description: This talk will use the Xarray-Spatial library for raster-based geospatial analysis to showcase how Dask and Numba speed up spatial processing
pydata.org/global2021/sch…
3/9
Video: youtube.com/watch?v=9PufD9…
Description: This talk will use the Xarray-Spatial library for raster-based geospatial analysis to showcase how Dask and Numba speed up spatial processing
pydata.org/global2021/sch…
3/9
1
4
5
"High Performance Python With Numba, Dask, and Rapids For the Absolute Beginner" by Gus Cavanaugh
Video: youtube.com/watch?v=ewaY9C…
Description: We step through a complete example while also covering core concepts so you can generalize to your own work
pydata.org/global2021/sch…
2/9
Video: youtube.com/watch?v=ewaY9C…
Description: We step through a complete example while also covering core concepts so you can generalize to your own work
pydata.org/global2021/sch…
2/9
1
3
9
Dask DataFrame read_parquet's performance for remotely-stored-data has been improved. This will provide faster reads for both the "pyarrow" and "fastparquet" engines, thanks to @_rjzamora!
1
2
9
Deploying Dask clusters is now even easier thanks to some new and refactored top-level documentation from @_JacobTomlinson.
1
1
8
The Dask Dashboard has a new panel displaying a stacked area plot of your computation’s progress through time. This is particularly useful as an alternative to the Task Stream when the number of tasks or workers becomes unwieldy:
1
1
15
The Dask docs have some shiny new diagrams for illustrating key concepts for Dask and parallel computing in Python:
1
2
10
Use Dask? Use JupyterLab?
We strongly recommend using the Dask JupyterLab extension, which brings the dashboard into your Jupyter session
youtube.com/watch?v=EX_voq…
We strongly recommend using the Dask JupyterLab extension, which brings the dashboard into your Jupyter session
youtube.com/watch?v=EX_voq…
1
24
79
Version 2021.06.1 of me is out! Thank you to all the community members who contributed. In particular, thanks to @_JacobTomlinson and @DataNerdery for making me look a lot nicer in @ProjectJupyter
3
11
63
A new blogpost from @DataNerdery : Skeleton network analysis with Dask:
blog.dask.org/2021/05/07/ske…
blog.dask.org/2021/05/07/ske…
0
1
3
Join us for #DaskSummit2021, 19-21 May. Tickets available now at summit.dask.org/tickets/ (pay what you can)
0
0
6
Join us for the #DaskSummit2021 online conference, May 19-21. The schedule is now live:
summit.dask.org/schedule/
summit.dask.org/schedule/
2
18
47
Come and talk at the distributed @dask_dev summit! Submit proposals at summit.dask.org/present/. Deadline for submission is March 21, 2021.
1
16
42
Version 2.23.0 of me was just released!
You get a Dask release, you get a Dask release, everybody gets a Dask release!
Some notable features include:
- A new DataFrame.items() method
- More flexibility around Python versions and compression
- Docs on using SQL with DataFrames
You get a Dask release, you get a Dask release, everybody gets a Dask release!
Some notable features include:
- A new DataFrame.items() method
- More flexibility around Python versions and compression
- Docs on using SQL with DataFrames
2
19
63
Just wrapped up our August Community Meeting.
Topics:
📊 An official Grafana template
👥 A potential online user conference
🌙 Nightly package builds
➕ and more...
Meetings first Thurday of each month. All are welcome!
Schedule, agenda and notes 👇🏻
docs.dask.org/en/latest/supp…
Topics:
📊 An official Grafana template
👥 A potential online user conference
🌙 Nightly package builds
➕ and more...
Meetings first Thurday of each month. All are welcome!
Schedule, agenda and notes 👇🏻
docs.dask.org/en/latest/supp…
0
3
26
Curious about Dask but not yet an active user? We'd love to hear what's important to you on the Dask Survey.
dask.org/survey
Currently we're mostly hearing from active users, and we'd like to change that.
dask.org/survey
Currently we're mostly hearing from active users, and we'd like to change that.
0
9
16
A fun, hopefully useful example to quickly train an ensemble of learners on the partitions of a Dask Array or DataFrame:
* Static: examples.dask.org/machine-learni…
* Binder: mybinder.org/v2/gh/dask/das…
* Static: examples.dask.org/machine-learni…
* Binder: mybinder.org/v2/gh/dask/das…
1
15
26
An image of a Cholesky decomposition from a recent task-ordering benchmarking PR by @eriknwelch
github.com/dask/dask-benc…
github.com/dask/dask-benc…
0
6
36
New example showing how Dask and Prophet can be used together for time series forecasting at scale.
4
24
159
Earlier this year 50 Dask maintainers and expert users from 30 institutions gathered to share their experiences scaling Python.
This post contains slides a summary of the experience.
blog.dask.org/2020/04/28/das…
This post contains slides a summary of the experience.
blog.dask.org/2020/04/28/das…
2
24
80
Love Dask's dashboard, but want to make a static version that you can share with team mates?
Dask version 2.9 includes a new performance_report function that does just this.
youtube.com/watch?v=nTMGbk…
Dask version 2.9 includes a new performance_report function that does just this.
youtube.com/watch?v=nTMGbk…
1
7
44
Talk: "Is Spark still relevant?" by @Chemisist at @PyData NYC 2019 comparing Dask, Spark, and RAPIDS, mostly from the perspective of enterprise support.
youtube.com/watch?v=obKZzF…
youtube.com/watch?v=obKZzF…
0
8
38
The Dask dashboard integrates natively with JupyterLab. This screencast shows how to set up and use these two tools together.
youtu.be/EX_voquHdk0
youtu.be/EX_voquHdk0
4
32
145
Confused about the difference between compute and persist? Curious how Dask reasons about when it can release data?
This is the seven-minute screencast for you.
youtube.com/watch?v=Msnzpz…
This is the seven-minute screencast for you.
youtube.com/watch?v=Msnzpz…
0
5
38
Dask's adaptive scheduling automatically scales clusters based on active load. It's a great way to scale up when you need resources, and down when you don't.
It's probably the best way to control cloud costs with Dask.
Here is a 4-minute screencast:
youtube.com/watch?v=dViyEq…
It's probably the best way to control cloud costs with Dask.
Here is a 4-minute screencast:
youtube.com/watch?v=dViyEq…
1
32
112
Dask can serve as an asynchronous task queue, simliar to Celery. Here is an example where a Tornado web application offloads expensive queries on to Dask.
examples.dask.org/applications/a…
examples.dask.org/applications/a…
2
24
78
Dask partners with Xarray and Rasterio/GDAL to make it easy to analyze large satellite imagery datasets.
See this executable example from @scotty_h_q computing NDVI from Landsat geotiff files:
examples.dask.org/applications/s…
See this executable example from @scotty_h_q computing NDVI from Landsat geotiff files:
examples.dask.org/applications/s…
1
22
72
Better and faster hyperparameter optimization with Dask:
blog.dask.org/2019/09/30/das…
A new blogpost from @stsievert
blog.dask.org/2019/09/30/das…
A new blogpost from @stsievert
0
34
124
@jejjohnson Sure! Have you seen jobqueue.dask.org ?
This screencast might also be helpful: youtu.be/FXsgmwpRExM
This screencast might also be helpful: youtu.be/FXsgmwpRExM
1
0
1
Dask maintainers regularly deliver tutorials at Python conferences.
If you're interested, you may want to watch this video from SciPy 2018
youtube.com/watch?v=mqdglv…
If you're interested, you may want to watch this video from SciPy 2018
youtube.com/watch?v=mqdglv…
0
8
71
I enjoyed this talk by @msalvaris using Dask, @PyTorch, and GPUs on @Azure to accelerate computer vision workloads:
developer.download.nvidia.com/video/gputechc…
developer.download.nvidia.com/video/gputechc…
1
14
57
Understanding performance in distributed systems can be hard.
Our documentation page "Understanding Performance" can help here: docs.dask.org/en/latest/unde…
In particular, we often point people to this video of Dask's dashboard.
youtu.be/N_GqzcuGLCY
Our documentation page "Understanding Performance" can help here: docs.dask.org/en/latest/unde…
In particular, we often point people to this video of Dask's dashboard.
youtu.be/N_GqzcuGLCY
0
15
62
Many people looking at large volumes of HDF5 and NetCDF data today use Dask through Xarray (@xarray_dev).
In many cases we recommend that people use Xarray over Dask Array, in particular when they have labeled and indexed dimensions.
xarray.pydata.org/en/stable/
In many cases we recommend that people use Xarray over Dask Array, in particular when they have labeled and indexed dimensions.
xarray.pydata.org/en/stable/
1
26
69
Read from @bnaul how Sidewalk labs uses Dask to model human behavior through modern cities, and the various challenges that arose.
stories.dask.org/en/latest/side…
stories.dask.org/en/latest/side…
1
6
18
Dask isn't only about Arrays and Dataframes.
Many people use Dask Futures to parallelize existing workloads, and build advanced real-time systems easily that adapt to changing situations.
docs.dask.org/en/latest/futu…
Many people use Dask Futures to parallelize existing workloads, and build advanced real-time systems easily that adapt to changing situations.
docs.dask.org/en/latest/futu…
3
34
118
Did you know that you can launch Dask as an MPI job with mpirun or mpiexec?
This makes it trivial to integrate with many HPC systems today.
mpi.dask.org/en/latest/
This makes it trivial to integrate with many HPC systems today.
mpi.dask.org/en/latest/
2
19
51
Read from @alimanfoo about how they combat the spread of malaria by analyzing mosquito genomes with Dask and Zarr at @bdi_oxford.
stories.dask.org/en/latest/mosq…
stories.dask.org/en/latest/mosq…
1
12
25
Did you know that you can participate in the Dask user survey even if you aren't a Dask user today?
Just let us know that you're "just looking" and let us know what you would like to see.
forms.gle/PFviQTmxtDLthg…
Just let us know that you're "just looking" and let us know what you would like to see.
forms.gle/PFviQTmxtDLthg…
1
5
11
Announcing the Dask User Survey for 2019 !
This takes 5-10 minutes and impacts our future direction. Please spread widely!
forms.gle/PFviQTmxtDLthg…
This takes 5-10 minutes and impacts our future direction. Please spread widely!
forms.gle/PFviQTmxtDLthg…
0
33
36
New blogpost on Dask and GPU arrays.
@PeterEntschev discusses recent work to integrate NumPy, Dask, and CuPy, and shows results with more complex algorithms like SVD and GLM.
blog.dask.org/2019/03/18/das…
@PeterEntschev discusses recent work to integrate NumPy, Dask, and CuPy, and shows results with more complex algorithms like SVD and GLM.
blog.dask.org/2019/03/18/das…
1
54
135
Dask release 0.19.0
Stability improvements, Actors, TPOT, Dashboard improvements, and more
matthewrocklin.com/blog/work/2018…
Stability improvements, Actors, TPOT, Dashboard improvements, and more
matthewrocklin.com/blog/work/2018…
1
69
169
@twiecki I like that you were able to do this by mixing dask.array and delayed. You might be able to simplify it further using atop.
gist.github.com/mrocklin/289bf…
rawgit.com/mrocklin/289bf…
gist.github.com/mrocklin/289bf…
rawgit.com/mrocklin/289bf…
1
0
5
New Blogpost: Building SAGA Optimization for Dask Arrays
matthewrocklin.com/blog/work/2018…
This was an educational experience building an optimization algorithm using dask.delayed with @fpedregosa at @UCBIDS
matthewrocklin.com/blog/work/2018…
This was an educational experience building an optimization algorithm using dask.delayed with @fpedregosa at @UCBIDS
0
18
44
Scalable Machine Learning with Dask @SciPyConf 2018 by @ogrisel and @TomAugspurger
youtube.com/watch?v=ccfsbu…
youtube.com/watch?v=ccfsbu…
0
25
60
Dask development status update:
matthewrocklin.com/blog/work/2018…
Yarn, incremental machine learning, and more examples
matthewrocklin.com/blog/work/2018…
Yarn, incremental machine learning, and more examples
0
16
57