Parallel Sourcing #28336
Replies: 4 comments 3 replies
-
|
Hey there 👋 I (and my team at New Relic) are super interested in this feature, but had a couple questions before we dig in too deep. We are in the process of migrating our docs site over to open source and using Gatsby in the process! Due to the size of the new site (thousands of markdown and MDX files + many other auto generated pages), we find that our boot-up time for development is a bit slow. We've developed quite a few custom plugins that handle various requirements of the site. While we haven't taken too much time (yet) to diagnose why our boot-up time is so long, I suspect its likely due to one or more of our custom plugins. The new parallel sourcing feature sounds super intriguing. We haven't tried it out yet, but I'd be curious what kind of limitations there are (if any) for these. Does this feature require that all of the source plugins operate independently from data created from other plugins? And does this feature mainly benefit async plugins? Or is this feature designed to help across all plugins? For reference, here are 2 examples from our site that rely on data created from other plugins:
I would love to hear more about this! We will try this feature out soon. Patrick had asked me to get the discussion going! |
Beta Was this translation helpful? Give feedback.
-
|
👋 Looking at your links above, it doesn't seem like this flag would affect you. What it changes is pretty small. If you have two plugins which both implement the Why this can speed up your site is sometimes source plugins can block Gatsby as they're making slow network requests. Running source plugins in parallel then means that all the slow network requests can be happening at the same time. But in sites like yours where all the data seems local, the problem is more just the pure computational expense of sorting through MDX/md files. MDX in particular is quite expensive. We do hope to speed up these sorts of operations in the future by spreading CPU intensive work across multiple node.js processes but that hasn't happened yet. Are you on the latest Gatsby? You might have seen Query on Demand get released — this helps speed up dev server startup time as it delays most of the markdown work until you actually visit the page. The MDX plugin still does a lot of work upfront so QoD is less effective but you should notice some effect https://twitter.com/GatsbyJS/status/1338929948975898624 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for getting back! That is good to know and makes total sense. We definitely do a lot of computationally expensive work. We do have the new features enabled! Query on demand and lazy images helped a TON with our boot-up time, so thanks so much for those features! We are still sitting about ~1min+ for boot-up from a cold start. I'd love to get that down if we can, but it may just be that we have too much going on at the moment to really help that. I'm hoping with the release of MDX v2, there is some help in terms performance. I know the underlying remark core just got a HUGE update, something we are hoping to try out soon (we are on an alpha version of v2 right before the big update). We'll keep watch for new features. Thanks for all the great work thus far! |
Beta Was this translation helpful? Give feedback.
-
|
I have a lot of source plugin and when I use |
Beta Was this translation helpful? Give feedback.
-
|
This has been working absolutely fine for me for several weeks now 🚀 |
Beta Was this translation helpful? Give feedback.
-
|
This was working fine when I was only using: but started getting errors when I've added Sometimes I get error Disabling Gatsby Info System:
OS: Linux 5.10 Ubuntu 20.04.2 LTS (Focal Fossa)
CPU: (8) x64 Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
Shell: 5.8 - /usr/bin/zsh
Binaries:
Node: 14.17.3 - ~/.nvm/versions/node/v14.17.3/bin/node
Yarn: 1.22.10 - ~/.nvm/versions/node/v14.17.3/bin/yarn
npm: 6.14.13 - ~/.nvm/versions/node/v14.17.3/bin/npm
npmPackages:
gatsby: ^3.9.1 => 3.9.1
gatsby-plugin-eslint: ^3.0.0 => 3.0.0
gatsby-plugin-feed: ^3.9.0 => 3.9.0
gatsby-plugin-gatsby-cloud: ^2.9.1 => 2.9.1
gatsby-plugin-google-tagmanager: ^3.9.0 => 3.9.0
gatsby-plugin-image: ^1.9.0 => 1.9.0
gatsby-plugin-manifest: ^3.9.0 => 3.9.0
gatsby-plugin-netlify: ^3.9.0 => 3.9.0
gatsby-plugin-react-helmet: ^4.9.0 => 4.9.0
gatsby-plugin-react-helmet-canonical-urls: ^1.4.0 => 1.4.0
gatsby-plugin-react-i18next: ^1.1.1 => 1.1.1
gatsby-plugin-sharp: ^3.9.0 => 3.9.0
gatsby-plugin-sitemap: ^4.5.0 => 4.5.0
gatsby-plugin-theme-ui: ^0.10.1 => 0.10.1
gatsby-source-filesystem: ^3.9.0 => 3.9.0
gatsby-source-google-spreadsheets: 3.0.8 => 3.0.8
gatsby-source-wordpress: ^5.9.2 => 5.9.2
gatsby-source-youtube-v3: ^3.0.1 => 3.0.1
gatsby-transformer-sharp: ^3.9.0 => 3.9.0
(Side note: I'm on Windows 11 Insider with WSL2, sometimes when changing the port |
Beta Was this translation helpful? Give feedback.
-
|
we experienced the same, but with google sheets api. Google doesn't like when you're hitting them too frequently. the fix for us was to add an artificial sleep between each google request. 1 second for us was sufficient in getting past that timeout |
Beta Was this translation helpful? Give feedback.
-
Original PR
Gatsby runs APIs sequentially. That's probably what we want as running many APIs is CPU bound. But sourcing is almost entirely bounded by the speed of the external APIs so we save quite a bit of time for sites with multiple source plugins by sourcing in parallel.
This change speeds up cold sourcing for gatsbyjs.com (which has 4 external sources) from 78s -> 45s or a 42% improvement.
One worry with changing this is it could increase memory pressure from more nodes being created and transformed at the same time. This is a separate issue that needs addressed but until is, sites might see increased memory usage from adding this flag.
There's a chance someone is depending on the serial behavior of API running so we'd need to wait for Gatsby v3 to enable it for everyone.
To enable this in your site, modify your
gatsby-config.jsto add the following:Beta Was this translation helpful? Give feedback.
All reactions