Running two instances of `yarn` causing issues?

[jodeleeuw]

On a clean install of pushkin, attempting to run pushkin install site I get this kind of error:

I think this is fixable by just running the command again, but I suspect the root cause of this is that pushkin-cli is trying to run multiple parallel instances of yarn when it builds the api and front-end as async events.

pushkin-cli/src/commands/sites/index.js

Lines 159 to 160 in d0c3750

	const apiPromise = promiseFolderInit(path.join(initDir, 'pushkin'), 'api').catch((err) => { console.error(err); });
	const frontendPromise = promiseFolderInit(path.join(initDir, 'pushkin'), 'front-end').catch((err) => { console.error(err); });

Before I make a PR on this, I just wanted to see if I'm missing anything. Clearly there was some deliberate choice to make all of the fs operations async in this part of the code, so I'm wondering if there's a reason to not await the completion of each of those before moving on to the next.

[jkhartshorne]

@jessestorbeck this is definitely related to #110, it's maybe a different problem.

So the --mutex flag is supposed to prevent race conditions in yarn. But I used it on yarn install but not yarn build. Also, it looks like that in at least some cases the -mutex flag must come immediately after yarn. Currently testing.

What I can't tell in @jodeleeuw is whether the problem is that --mutex wasn't working or that it was. The fact that the error says it's waiting for another instance to finish suggests to me that perhaps once instance was waiting for the freeze on yarn to be released, but it never was.

See also pushkin-consortium/pushkin-exptemplates-basic#35

[jkhartshorne]

@jodeleeuw -- addressing another thing you raised above, the CLI definitely needs to be more consistent in terms of what is done synchronously and what is asynchronous. I've tried to use async wherever possible in order to speed things up. But there are definitely things that could be async and aren't. And there may be things that shouldn't be async. Usually I use async fs when we won't read the updated file any time soon.

One complication is that Node isn't very good about knowing when terminal processes have finished. I remember I spent a lot of time on this two years ago and hacked things to the point where they mostly don't crash as a result, but no doubt it's less than ideal. And since it's been a couple years since I did that work, it's quite possible there are options now that weren't available then.

We do accept pull requests :)

[jkhartshorne]

@jessestorbeck @ayang21 have either of you noticed whether this issue has been run into lately?

[ayang21]

Pushkin install site has been working just fine for me. No errors have appeared.

[jessestorbeck]

@jkhartshorne -- I missed this earlier, but I also haven't seen this problem in a while.

[ellissc]

@jkhartshorne I've been running into a couple errors lately, though I'm not sure if they're due to Docker or Yarn. Do you think they may be related?

Pushkin version 3.4.0
Recommended site and experiment versions when installing.
Docker version 4.24.0

[AryaanMishra]

@jkhartshorne we installed the basic version of the site template and experiment template and we have been running into issue on docker. We tried prep twice, using stop, kill and armageddon and also docker system prune, but it doesn't seem to help.

Pushkin version 3.4.0
Docker version 4.24.0

[ellissc]

docker system prune worked (even though I had been running stop kill armageddon), but the localhost then gives a network error, similar to the stochastic issue (pushkin-consortium/pushkin#225 (comment)).

Update: this network error is stochastic as well. The network error went away after running stop kill armageddon and prep start --nocache a few times. Leaving the error message in case other people run into the same issue.

[jessestorbeck]

@ellissc @AryaanMishra -- Can we do a bit of housekeeping here first? This issue is about an error during pushkin install site. @ellissc, I can't tell if your original screenshot is from prep or start, but I can see that @AryaanMishra's is from prep. If these are errors you haven't seen before, they should probably start as discussions. We can decide from there -- after some attempt to diagnose the problem -- whether to elevate it to an issue. The last general thing is that v4.24.0 is your version of Docker Desktop. The more helpful version number is for Docker Engine. You can get this version number with the command docker --version.

@ellissc, it looks like the top error for you is this ENOSPC thing. I can't say I really know what's causing it, but taking it at face value, it kind of makes sense that clearing out Docker would fix the problem of not having enough space.

@AryaanMishra, looking back here, I have actually seen this issue with the --load flag before, and upgrading Docker Engine (and restarting my machine) apparently fixed it. Can you find out first what version of Docker Engine you're using? I'm on v24.0.5. If you need to upgrade, it may be easiest to just re-install Docker Desktop and that should update Docker Engine too.

[ellissc]

@jessestorbeck @AryaanMishra Thank you! Yeah, it looks like Docker Engine was causing the issue. For some reason, our lab computer was running Docker Engine v20, even though we had the newest version of Docker Desktop. I was able to get Docker Engine v24.0.6, and that fixed the issue. Thank you for your help!