{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2019 Dask User Survey Results\n",
    "\n",
    "This notebook presents the results of the 2019 Dask User Survey,\n",
    "which ran earlier this summer. Thanks to everyone who took the time to fill out the survey!\n",
    "These results help us better understand the Dask community and will guide future development efforts.\n",
    "\n",
    "The raw data, as well as the start of an analysis, can be found in this binder:\n",
    "\n",
    "[![2019 Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/dask/dask-examples/main?urlpath=%2Ftree%2Fsurveys%2F2019.ipynb)\n",
    "\n",
    "Let us know if you find anything in the data.\n",
    "\n",
    "## Highlights\n",
    "\n",
    "We had 259 responses to the survey. Overall, we found that the survey respondents really care about improved documentation, and ease of use (including ease of deployment), and scaling. While Dask brings together many different communities (big arrays versus big dataframes, traditional HPC users versus cloud-native resource managers), there was general agreement in what is most important for Dask.\n",
    "\n",
    "Now we'll go through some individual items questions, highlighting particularly interesting results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import textwrap\n",
    "import re\n",
    "\n",
    "api_choices = ['Array', 'Bag', 'DataFrame', 'Delayed', 'Futures', 'ML', 'Xarray']\n",
    "cluster_manager_choices = [\n",
    "    \"SSH\",\n",
    "    \"Kubernetes\",\n",
    "    \"HPC\",\n",
    "    \"My workplace has a custom solution for this\",\n",
    "    \"Hadoop / Yarn / EMR\",\n",
    "]\n",
    "\n",
    "def shorten(label):\n",
    "    return textwrap.shorten(label, 50)\n",
    "\n",
    "\n",
    "def fmt_percent(ax):\n",
    "    ticklabels = ['{:,.2f}%'.format(x) for x in ax.get_xticks()]\n",
    "    ax.set_xticklabels(ticklabels)\n",
    "    sns.despine()\n",
    "    return ax\n",
    "\n",
    "\n",
    "df = (\n",
    "    pd.read_csv(\"data/2019-user-survey-results.csv.gz\", parse_dates=['Timestamp'])\n",
    "      .replace({\"How often do you use Dask?\": \"I use Dask all the time, even when I sleep\"}, \"Every day\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How do you use Dask?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For learning resources, almost every respondent uses the documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = (\n",
    "    df['What Dask resources have you used for support in the last six months?']\n",
    "    .str.split(\";\").explode()\n",
    "    .value_counts().head(6)\n",
    "    .div(len(df)).mul(100).plot.barh()\n",
    ");\n",
    "fmt_percent(ax).set(title=\"Support Resource Usage\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most respondents use Dask at least occasionally. Fortunately we had a decent number of respondents who are just looking into Dask, yet still spent the time to take the survey."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "usage_order = [\n",
    "    'Every day',\n",
    "    'Occasionally',\n",
    "    'Just looking for now',\n",
    "]\n",
    "ax = df['How often do you use Dask?'].value_counts().loc[usage_order].div(len(df)).mul(100).plot.barh()\n",
    "fmt_percent(ax).set(title=\"How often do you use Dask?\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm curiuos about how learning resource usage changes as users become more experienced. We might expect those just looking into Dask to start with `examples.dask.org`, where they can try out Dask without installing anything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resources = df['What Dask resources have you used for support in the last six months?'].str.split(\";\").explode()\n",
    "top = resources.value_counts().head(6).index\n",
    "resources = resources[resources.isin(top)]\n",
    "\n",
    "m = (\n",
    "    pd.merge(df[['How often do you use Dask?']], resources, left_index=True, right_index=True)\n",
    "      .replace(re.compile(\"GitHub.*\"), \"GitHub\")\n",
    ")\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 10))\n",
    "\n",
    "sns.countplot(hue=\"What Dask resources have you used for support in the last six months?\",\n",
    "              y='How often do you use Dask?',\n",
    "              order=usage_order,\n",
    "              data=m, ax=ax)\n",
    "sns.despine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall, documentation is still the leader across user user groups.\n",
    "\n",
    "The usage of the [Dask tutorial](https://github.com/dask/dask-tutorial) and the [dask examples](https://examples.dask.org) are relatively consistent across groups. The primary difference between regular and new users is that regular users are more likely to engage on GitHub.\n",
    "\n",
    "From StackOverflow questions and GitHub issues, we have a vague idea about which parts of the library are used.\n",
    "The survey shows that (for our respondents at least) DataFrame and Delayed are the most commonly used APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_counts = (\n",
    "    df['Dask APIs'].str.split(\";\").explode().value_counts()\n",
    "    .div(len(df)).mul(100)\n",
    ")\n",
    "ax = api_counts.sort_values().nlargest(8).plot.barh()\n",
    "fmt_percent(ax).set(xlabel=\"Percent\")\n",
    "sns.despine();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"About {:0.2%} of our respondests are using Dask on a Cluster.\".format(df['Local machine or Cluster?'].str.contains(\"Cluster\").mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the majority of respondents *also* use Dask on their laptop.\n",
    "This highlights the importance of Dask scaling down, either for\n",
    "prototyping with a `LocalCluster`, or for out-of-core analysis\n",
    "using `LocalCluster` or one of the single-machine schedulers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "order = [\n",
    "    'Personal laptop',\n",
    "    'Large workstation',\n",
    "    'Cluster of 2-10 machines',\n",
    "    'Cluster with 10-100 machines',\n",
    "    'Cluster with 100+ machines'\n",
    "]\n",
    "df['Local machine or Cluster?'].str.split(\";\").explode().value_counts().loc[order].plot.barh();\n",
    "sns.despine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most respondents use Dask interactively, at least some of the time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapper = {\n",
    "    \"Interactive:  I use Dask with Jupyter or IPython when playing with data;Batch: I submit scripts that run in the future\": \"Both\",\n",
    "    \"Interactive:  I use Dask with Jupyter or IPython when playing with data\": \"Interactive\",\n",
    "    \"Batch: I submit scripts that run in the future\": \"Batch\",\n",
    "    \n",
    "}\n",
    "\n",
    "ax = df[\"Interactive or Batch?\"].map(mapper).value_counts().div(len(df)).mul(100).plot.barh()\n",
    "sns.despine()\n",
    "fmt_percent(ax)\n",
    "ax.set(title='Interactive or Batch?');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most repondents thought that more documentation and examples would be the most valuable improvements to the project. This is especially pronounced among new users. But even among those using Dask everyday more people thought that \"More examples\" is more valuable than \"New features\" or \"Performance improvements\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help_by_use = (\n",
    "    df.groupby(\"How often do you use Dask?\")['Which would help you most right now?']\n",
    "    .value_counts()\n",
    "    .unstack()\n",
    ")\n",
    "\n",
    "s = (\n",
    "    help_by_use\n",
    "        .style\n",
    "        .background_gradient(axis=\"rows\")\n",
    "        .set_caption(\"Normalized by row. Darker means that a higher proporiton of \"\n",
    "                     \"users with that usage frequency prefer that priority.\")\n",
    ")\n",
    "s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perhaps users of certain dask APIs feel differenlty from the group as a whole? We perform a similar analysis grouped by API use, rather than frequency of use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help_by_api = (\n",
    "    pd.merge(\n",
    "        df['Dask APIs'].str.split(';').explode(),\n",
    "        df['Which would help you most right now?'],\n",
    "        left_index=True, right_index=True)\n",
    "    .groupby('Which would help you most right now?')['Dask APIs'].value_counts()\n",
    "    .unstack(fill_value=0).T\n",
    "    .loc[['Array', 'Bag', 'DataFrame', 'Delayed', 'Futures', 'ML', 'Xarray']]\n",
    "    \n",
    ")\n",
    "(\n",
    "    help_by_api\n",
    "        .style\n",
    "        .background_gradient(axis=\"columns\")\n",
    "        .set_caption(\"Normalized by row. Darker means that a higher proporiton of \"\n",
    "                     \"users of that API prefer that priority.\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nothing really stands out. The \"futures\" users (who we expect to be relatively advanced) may prioritize features and performance over documentation. But everyone agrees that more examples are the highest priority."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Common Feature Requests\n",
    "\n",
    "For specific features, we made a list of things that we (as developers) thought might be important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "common = (df[df.columns[df.columns.str.startswith(\"What common feature\")]]\n",
    "          .rename(columns=lambda x: x.lstrip(\"What common feature requests do you care about most?[\").rstrip(r\"]\")))\n",
    "\n",
    "counts = (\n",
    "    common.apply(pd.value_counts)\n",
    "    .T.stack().reset_index()\n",
    "    .rename(columns={'level_0': 'Question', 'level_1': \"Importance\", 0: \"count\"})\n",
    ")\n",
    "\n",
    "order = [\"Not relevant for me\", \"Somewhat useful\", 'Critical to me']\n",
    "g = (\n",
    "    sns.FacetGrid(counts, col=\"Question\", col_wrap=2, aspect=1.5, sharex=False, height=3)\n",
    "    .map(sns.barplot, \"Importance\", \"count\", order=order)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The clearest standout is how many people thought \"Better NumPy/Pandas support\" was \"most critical\". In hindsight, it'd be good to have a followup fill-in field to undertand what each respondent meant by that. The parsimonious interpretion is \"cover more of the NumPy / pandas API\".\n",
    "\n",
    "\"Ease of deployment\" had a high proportion of \"critical to me\". Again in hindsight, I notice a bit of ambiguity. Does this mean people want Dask to be easier to deploy? Or does this mean that Dask, which they currently find easy to deploy, is critically important? Regardless, we can prioritize simplicity in deployment.\n",
    "\n",
    "Relatively few respondents care about things like \"Managing many users\", though we expect that this would be relatively popular among system administartors, who are a smaller population.\n",
    "\n",
    "And of course, we have people pushing Dask to its limits for whom \"Improving scaling\" is critically important."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What other systems do you use?\n",
    "\n",
    "A relatively high proportion of respondents use Python 3 (97% compared to 84% in the most recent [Python Developers Survey](https://www.jetbrains.com/research/python-developers-survey-2018/))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Python 2 or 3?'].dropna().astype(int).value_counts(normalize=True).apply(\"{:0.2%}\".format)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We were a bit surprised to see that SSH is the most popular \"cluster resource manager\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['If you use a cluster, how do you launch Dask? '].dropna().str.split(\";\").explode().value_counts().head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does cluster-resource manager compare with API usage?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "managers = (\n",
    "    df['If you use a cluster, how do you launch Dask? '].str.split(\";\").explode().dropna()\n",
    "        .replace(re.compile(\"HPC.*\"), \"HPC\")\n",
    "    .loc[lambda x: x.isin(cluster_manager_choices)]\n",
    ")\n",
    "\n",
    "apis = (\n",
    "    df['Dask APIs'].str.split(\";\").explode().dropna()\n",
    "    .loc[lambda x: x.isin(api_choices)]\n",
    ")\n",
    "wm = pd.merge(apis, managers, left_index=True, right_index=True).replace(\"My workplace has a custom solution for this\", \"Custom\")\n",
    "\n",
    "x = wm.groupby(\"Dask APIs\")[\"If you use a cluster, how do you launch Dask? \"].value_counts().unstack().T\n",
    "x.style.background_gradient(axis=\"columns\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HPC users are relatively heavy users of `dask.array` and xarray."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Somewhat surprisingly, Dask's heaviest users find dask stable enough. Perhaps they've pushed past the bugs and found workarounds (percentages are normalized by row)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(9, 6))\n",
    "sns.countplot(x=\"How often do you use Dask?\", hue=\"Is Dask stable enough for you?\", data=df, ax=ax,\n",
    "              order=reversed(usage_order));\n",
    "sns.despine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Takeaways\n",
    "\n",
    "1. We should prioritize improving and expanding our documentation and examples. This may be\n",
    "   accomplished by Dask maintainers seeking examples from the community. Many of the examples\n",
    "   on https://examples.dask.org were developed by domain specialist who use Dask.\n",
    "2. Improved scaling to larger problems is important, but we shouldn't\n",
    "   sacrifice the single-machine usecase to get there.\n",
    "3. Both interactive and batch workflows are important.\n",
    "4. Dask's various sub-communities are more similar than they are different.\n",
    "\n",
    "Thanks again to all the respondents. We look forward to repeating this process to identify trends over time."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}