Extensions causes lots of kubectl processes to spawn that never finish

[jeroenlandheer]

Expected behaviour

It looks like the extension is using kubectl for watching resources in kubernetes but spawns a process for each and every single item in the workloads / sources list. This is very inefficient because it causes lots of processes to run and some do not terminate when I close VSCode.

Pressing the refresh button in the workloads immediately spawns quite a lot of them (a little less than a 100 in my case) and some of them stop afterwards, some of them don't.

Actual behaviour

When I run this in powershell...

> get-process | ? {$_.name -eq "kubectl"}

 NPM(K)    PM(M)      WS(M)     CPU(s)      Id  SI ProcessName
 ------    -----      -----     ------      --  -- -----------
     13    15,02      15,00       0,03    2720   3 kubectl
     13    14,99      14,98       0,09    4656   3 kubectl
     23    41,20      41,16       0,62    6464   3 kubectl
     22    41,05      41,01       0,47    8080   3 kubectl
     13    14,90      14,90       0,00    9700   3 kubectl
     13    15,00      14,98       0,03   11964   3 kubectl
     22    40,78      41,04       0,67   12904   3 kubectl
     22    41,49      41,61       0,66   15016   3 kubectl
     22    40,35      40,91       0,55   15244   3 kubectl
     22    39,89      40,26       0,62   17116   3 kubectl
     22    40,96      41,21       0,64   21004   3 kubectl
     13    14,91      14,91       0,00   23652   3 kubectl
     22    39,40      40,02       0,50   23824   3 kubectl
     23    40,25      40,50       0,42   25200   3 kubectl
... and another 500+ entries ....

Steps to reproduce

Just run the extension with a Flux deployment on Windows.

We have about:

• 100 kustomizations
• 50 helmrelease objects
• 17 helm repositories
• 1 git source

Versions

kubectl version: 1.26.1
Flux version: 2.1.0
Git version: N/A
Azure version: N/A
Extension version: v0.25.3
VSCode version: 1.82.2
Operating System (OS) and its version: Win 11 22H2

[kingdonb]

The kubectl entries are proxy connections. They are intended to be one per cluster, and they should be managed by the extension - there's some care taken to ensure that they are cleaned up after the extension shuts down, so this is definitely not expected. Is the extension working otherwise? How many clusters do you have, and what type of cluster are they - bare metal clusters, or how do you authenticate to the cluster is really what I'm looking to know.

If you run kubectl proxy yourself, in the same kubeconfig context as the extension gets, what result do you see?

[kingdonb]

In unix, with ps w I would usually be able to see the arguments. If kubectl proxy fails then you might be falling back to the old way, which might be performing quite poorly as we haven't tested fallback thoroughly. The proxy behavior and serverside connection is much more performant, so if you are falling back to that mode, the solution is hopefully to find out why the proxy mode fails and fix it so we don't fall back if possible, since kubectl forking mode is very inefficient and slow, imperative, (not responsive and reactive like the serverside notify mode that avoids running kubectl for each query, or forking processes or repeated connections altogether.)

If they are not kubectl proxy processes then it is probably the fallback behavior, and if that's the case, we should really figure out how to rate limit that forking behavior - even if the behavior is a lot slower, and takes more than seconds to complete a refresh in the fallback mode, it definitely shouldn't ever fork 100 processes.

[juozasg]

@jeroenlandheer Thank you for the bug report! This should be fixed in the new 0.25.4 release. The extension will still spawn kubectl processes to query the cluster, but none of them should remain after vscode is closed.

[kingdonb]

Please let us know if this resolves your issue or if it has any impact at all! It's not really clear from our testing if these processes are spawned all at once, or if they built up over a long time of opening and closing VSCode due to platform-specific issues.

It is also possible that something is timing out, causing a new proxy to be spawned (due to the large number of resources) - you might try changing the timeout in the extension settings.

We'd love to get some more details about this scenario, I'm going to suggest as well if you still see this issue at all, you try increasing the timeout and please report back if you're able to test again.

I have the version 0.25.4 and still meet the issue. Once I use the extension, kubectl processes with kubectl proxy -p 0 command are spawn until my machine OOM. Those processes are not closed when I exit vscode and I have to end those manually.

I use rancher to manage my kubernetes clusters and this is the output I get from kubectl proxy command

kubectl proxy
W1123 10:48:58.227589   51446 proxy.go:172] Your kube context contains a server path /k8s/clusters/c-m-dnc9dmdp, use --append-server-path to automatically append the path to each request
Starting to serve on 127.0.0.1:8001

If I use a direct connect method (without rancher proxying my requests), I don't have any issue.

[stalb]

Every time I start VSCode (or when I refresh the cluster view), a new kubectl proxy -p 0 command is started:

ps faux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
stalb    1236874  0.0  0.0  16516  8252 ?        Ss    2023   0:03 /lib/systemd/systemd --user
stalb    1349019  1.2  0.9 1177782656 147596 ?   Ssl  15:24   0:03  \_ /snap/codium/365/usr/share/codium/codium --ms-enable-electron-run-as-node --no-sandbox --unity-launch
stalb    1349107  3.0  1.2 1177568204 206084 ?   Sl   15:24   0:09  |   \_ /snap/codium/365/usr/share/codium/codium --type=utility --utility-sub-type=node.mojom.NodeService --lang=en-US --service-sandbox-type=none --no-sandbox --dns-resul
stalb    1354240  0.0  0.0   2480   584 ?        S    15:28   0:00  |       \_ /bin/sh -c kubectl proxy -p 0
stalb    1354268  0.2  0.2 4955780 40588 ?       Sl   15:28   0:00  |           \_ /snap/kubectl/3149/kubectl proxy -p 0
stalb    1349500  0.0  0.2 4955780 38940 ?       Sl   15:24   0:00  \_ /snap/kubectl/3149/kubectl proxy -p 0
stalb    1352530  0.1  0.2 4956036 41840 ?       Sl   15:28   0:00  \_ /snap/kubectl/3149/kubectl proxy -p 0
stalb    1353361  0.2  0.2 4955780 40324 ?       Sl   15:28   0:00  \_ /snap/kubectl/3149/kubectl proxy -p 0

When I exit VSCode the proxy commands are not killed:

ps faux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
stalb    1236874  0.0  0.0  16516  8252 ?        Ss    2023   0:03 /lib/systemd/systemd --user
stalb    1349500  0.0  0.2 4955780 38560 ?       Sl   15:24   0:00  \_ /snap/kubectl/3149/kubectl proxy -p 0
stalb    1352530  0.0  0.2 4956036 39728 ?       Sl   15:28   0:00  \_ /snap/kubectl/3149/kubectl proxy -p 0
stalb    1353361  0.0  0.2 4955780 39024 ?       Sl   15:28   0:00  \_ /snap/kubectl/3149/kubectl proxy -p 0
stalb    1354268  0.0  0.2 4955780 38660 ?       Sl   15:28   0:00  \_ /snap/kubectl/3149/kubectl proxy -p 0

The /bin/sh -c kubectl proxy -p 0 command is stopped and restarted each time I refresh the cluster view, but not its kubectl subprocess. Same problem when I quit VSCode (/bin/sh -c kubectl proxy -p 0 command is stopped but not its subprocess).

Config:

• System : debian 11.7 (bullseye)
• VSCodium:
  • Version: 1.84.2
    Release: 23319
    Commit: 1fc52b725b3f76b97eadfd988cea42a739ae923f
    Date: 2023-11-15T21:57:44.436Z
    Electron: 25.9.2
    ElectronBuildId: undefined
    Chromium: 114.0.5735.289
    Node.js: 18.15.0
    V8: 11.4.183.29-electron.0
    OS: Linux x64 5.10.0-22-amd64 snap
• GitOps Tools for Flux v0.26.0

Remark:
I also meet the issue on Windows 11 with VSCode 1.85.1 and GitOps Tools for Flux v0.26.0.

Could this be related to kubernetes-client/javascript ?

[kingdonb]

Thank you for reporting the issue persists. It would be good to try to reproduce the issue with steps since it's not happening on my machine. Thank you for providing details about the runtime and operating system hosting VSCode. This may help.

Development is on a bit of a hiatus, we have taken steps to ensure that new releases can continue to go out if fixes are made available, but we need volunteers urgently. Anyone with a desire to learn about VSCode and Kubernetes internals, free time to volunteer, and any inkling of relevant experience with TypeScript or VSCode extension development should apply within.

For the time being you can still email kingdon at weave dot works, but it is possible that email will stop working soon, so if you don't hear from me, try again at kingdon at tuesdaystudios dot com. I can't say until a more formal announcement is made much about the future of the extension, but I would like to resolve this issue and have become somewhat intimately familiar with the debugger, so it is possible that with a 30 minute zoom call we could diagnose the issue and begin to triage a patch or fix for your system.

Do let me know if you've uncovered anything additional which may help to reproduce it. It sounds like multiple people are able to reproduce the issue and the latest 0.26.0 didn't resolve it.

[kingdonb]

If I use a direct connect method (without rancher proxying my requests), I don't have any issue.

That is interesting. I've used loft.sh to proxy requests before and it uncovered some interesting problems. Maybe I can get the issue to reproduce by using a proxy for kubectl auth. I'll try it! Thanks again.

[stalb]

With v0.27.0 release the problem has stop. So I guess the problem was connected to the live updated UI feature (#500 #503)

[kingdonb]

Thanks for the report @stalb this information may help us piece together how to move forward, and where to look for bugs before we make another release. 👍