-
Notifications
You must be signed in to change notification settings - Fork 180
Update Troubleshooting Guide with v1.21 and Feedback from #726 #729
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -1,70 +1,135 @@ | ||||||||||||||||||
|
|
||||||||||||||||||
| ````markdown | ||||||||||||||||||
| --- | ||||||||||||||||||
| title: TroubleShooting | ||||||||||||||||||
| title: Troubleshooting | ||||||||||||||||||
| sidebar_position: 7 | ||||||||||||||||||
| --- | ||||||||||||||||||
|
|
||||||||||||||||||
| This page contains a few commonly occurring questions. | ||||||||||||||||||
| # Troubleshooting | ||||||||||||||||||
|
|
||||||||||||||||||
| This page addresses common issues encountered during the setup and operation of KubeEdge. | ||||||||||||||||||
|
|
||||||||||||||||||
| --- | ||||||||||||||||||
|
|
||||||||||||||||||
| ## keadm init failed to download release | ||||||||||||||||||
|
|
||||||||||||||||||
| If you have issue about connection to github, please follow below guide with proxy before do setup, take `v1.12.1` as example: | ||||||||||||||||||
| If you experience connectivity issues with GitHub during `keadm init`, you can manually download the required KubeEdge release and CRD files before proceeding with the setup. This approach is particularly useful in environments with restricted internet access. | ||||||||||||||||||
|
|
||||||||||||||||||
| - download release pkgs from [release page](https://github.com/kubeedge/kubeedge/releases/tag/v1.12.1) | ||||||||||||||||||
| - download crds yamls matches the release version you downloaded, links as below: | ||||||||||||||||||
| - [devices_v1alpha1_device.yaml](https://raw.githubusercontent.com/kubeedge/kubeedge/v1.12.1/build/crds/devices/devices_v1alpha2_device.yaml) | ||||||||||||||||||
| - [devices_v1alpha1_devicemodel.yaml](https://raw.githubusercontent.com/kubeedge/kubeedge/v1.12.1/build/crds/devices/devices_v1alpha2_devicemodel.yaml) | ||||||||||||||||||
| - [cluster_objectsync_v1alpha1.yaml](https://raw.githubusercontent.com/kubeedge/kubeedge/v1.12.1/build/crds/reliablesyncs/cluster_objectsync_v1alpha1.yaml) | ||||||||||||||||||
| - [objectsync_v1alpha1.yaml](https://raw.githubusercontent.com/kubeedge/kubeedge/v1.12.1/build/crds/reliablesyncs/objectsync_v1alpha1.yaml) | ||||||||||||||||||
| - put them under `/etc/kubeedge` as below: | ||||||||||||||||||
| ```bash | ||||||||||||||||||
| $ tree -L 3 | ||||||||||||||||||
| . | ||||||||||||||||||
| ├── crds | ||||||||||||||||||
| │ ├── devices | ||||||||||||||||||
| │ │ ├── devices_v1alpha2_devicemodel.yaml | ||||||||||||||||||
| │ │ └── devices_v1alpha2_device.yaml | ||||||||||||||||||
| │ └── reliablesyncs | ||||||||||||||||||
| │ ├── cluster_objectsync_v1alpha1.yaml | ||||||||||||||||||
| │ └── objectsync_v1alpha1.yaml | ||||||||||||||||||
| └── kubeedge-v1.12.1-linux-amd64.tar.gz | ||||||||||||||||||
|
|
||||||||||||||||||
| 3 directories, 5 files | ||||||||||||||||||
| ### Example using KubeEdge v1.21.0 | ||||||||||||||||||
|
|
||||||||||||||||||
| ``` | ||||||||||||||||||
| #### Download the release package | ||||||||||||||||||
|
|
||||||||||||||||||
| Get it from the [KubeEdge v1.21.0 release page](https://github.com/kubeedge/kubeedge/releases/tag/v1.21.0). | ||||||||||||||||||
|
|
||||||||||||||||||
| Make sure to select the correct architecture for your edge node (e.g., `linux-amd64`, `linux-arm64`). | ||||||||||||||||||
|
|
||||||||||||||||||
| #### Download the CRD YAMLs | ||||||||||||||||||
|
|
||||||||||||||||||
| For v1.21, use the **v1beta1 CRDs** available under `build/crds`: | ||||||||||||||||||
|
|
||||||||||||||||||
| - [devices_v1beta1_device.yaml](https://github.com/kubeedge/kubeedge/blob/v1.21.0/build/crds/devices/devices_v1beta1_device.yaml) | ||||||||||||||||||
| - [devices_v1beta1_devicemodel.yaml](https://github.com/kubeedge/kubeedge/blob/v1.21.0/build/crds/devices/devices_v1beta1_devicemodel.yaml) | ||||||||||||||||||
| - [cluster_objectsync_v1alpha1.yaml](https://github.com/kubeedge/kubeedge/blob/v1.21.0/build/crds/reliablesyncs/cluster_objectsync_v1alpha1.yaml) | ||||||||||||||||||
| - [objectsync_v1alpha1.yaml](https://github.com/kubeedge/kubeedge/blob/v1.21.0/build/crds/reliablesyncs/objectsync_v1alpha1.yaml) | ||||||||||||||||||
|
Comment on lines
+30
to
+33
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The links to the CRD YAML files point to the GitHub HTML view (
Suggested change
|
||||||||||||||||||
|
|
||||||||||||||||||
| #### Place the files under `/etc/kubeedge` | ||||||||||||||||||
|
|
||||||||||||||||||
| ```bash | ||||||||||||||||||
| tree -L 3 /etc/kubeedge | ||||||||||||||||||
|
|
||||||||||||||||||
| . | ||||||||||||||||||
| ├── crds | ||||||||||||||||||
| │ ├── devices | ||||||||||||||||||
| │ │ ├── devices_v1beta1_device.yaml | ||||||||||||||||||
| │ │ └── devices_v1beta1_devicemodel.yaml | ||||||||||||||||||
| │ └── reliablesyncs | ||||||||||||||||||
| │ ├── cluster_objectsync_v1alpha1.yaml | ||||||||||||||||||
| │ └── objectsync_v1alpha1.yaml | ||||||||||||||||||
| └── kubeedge-v1.21.0-linux-amd64.tar.gz | ||||||||||||||||||
| ```` | ||||||||||||||||||
|
|
||||||||||||||||||
| **Run keadm init with version flag** | ||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||||||||||||||||||
|
|
||||||||||||||||||
| ```bash | ||||||||||||||||||
| keadm init --kubeedge-version v1.21.0 | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| This ensures that `keadm` detects the local CRDs and release files, skipping the download process. | ||||||||||||||||||
|
|
||||||||||||||||||
| --- | ||||||||||||||||||
|
|
||||||||||||||||||
| Then you can do setup without any network issue, `keadm` would detect them and not download again(make sure you specify `v1.12.1` with option `--kubeedge-version v1.12.1`). | ||||||||||||||||||
| ## Container keeps Pending or Terminating | ||||||||||||||||||
|
|
||||||||||||||||||
| ## Container keeps pending/ terminating | ||||||||||||||||||
| If your pods remain in a **Pending** or **Terminating** state, try the following troubleshooting steps: | ||||||||||||||||||
|
|
||||||||||||||||||
| 1. Check the output of `kubectl get nodes`, whether the node is running healthy. Note that nodes in unreachable, offline status cannot complete graceful/soft pod deletion until they come back to normal. | ||||||||||||||||||
| 2. Check the output of `kubectl describe pod <your-pod>`, whether the pod is scheduled successfully. | ||||||||||||||||||
| 3. Check the `edgecore` logs for any errors. | ||||||||||||||||||
| 4. Check the architecture of the node running `edgecore` and make sure that container image you are trying to run is of the same architecture. | ||||||||||||||||||
| For example, if you are running `edgecore` on Raspberry Pi 4, which is of `arm64v8` architecture, the nginx image to be executed would be `arm64v8/nginx` from the docker hub. | ||||||||||||||||||
| 1. **Check node health** | ||||||||||||||||||
| Run `kubectl get nodes` and ensure all nodes are in a `Ready` state. | ||||||||||||||||||
|
|
||||||||||||||||||
| 5. Also, check that the `podSandboxImage` is correctly set as defined in [Modification in edgecore.yaml](../setup/config#modification-in-edgecoreyaml). | ||||||||||||||||||
| 2. **Describe the pod** | ||||||||||||||||||
| Run `kubectl describe pod <your-pod>` to get details about scheduling and other events. | ||||||||||||||||||
|
|
||||||||||||||||||
| 6. If all of the above is correctly set, login manually to your edge node and run your docker image manually by | ||||||||||||||||||
| 3. **Inspect EdgeCore logs** | ||||||||||||||||||
|
|
||||||||||||||||||
| ```shell | ||||||||||||||||||
| docker run <your-container-image> | ||||||||||||||||||
| - **If using systemd**: | ||||||||||||||||||
| ```bash | ||||||||||||||||||
| journalctl --unit edgecore.service -f | ||||||||||||||||||
| ``` | ||||||||||||||||||
| - **If started manually**: | ||||||||||||||||||
| Check the log file specified at startup or stdout. | ||||||||||||||||||
|
|
||||||||||||||||||
| 4. **Verify architecture compatibility** | ||||||||||||||||||
| Ensure the container image architecture matches your edge node's architecture. For example, use `arm64v8/nginx` for a Raspberry Pi 4 (arm64) instead of the default `nginx` image. | ||||||||||||||||||
|
|
||||||||||||||||||
| 5. **Check `podSandboxImage` in `edgecore.yaml`** | ||||||||||||||||||
| Confirm that `podSandboxImage` points to a valid and compatible image for your node's architecture. | ||||||||||||||||||
|
|
||||||||||||||||||
| 6. **Manually run the container** | ||||||||||||||||||
| Attempt to run the container image manually on the edge node to isolate potential runtime issues: | ||||||||||||||||||
|
|
||||||||||||||||||
| ```bash | ||||||||||||||||||
| docker run <your-container-image> | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| 7. If the docker container image is not pulled from the docker hub, please check that there is enough space on the edge node. | ||||||||||||||||||
| 7. **Check disk space** | ||||||||||||||||||
| Insufficient disk space can prevent new images from being pulled. Free up space if necessary. | ||||||||||||||||||
|
|
||||||||||||||||||
| ## Where do we find cloudcore/edgecore logs | ||||||||||||||||||
| --- | ||||||||||||||||||
|
|
||||||||||||||||||
| ## Viewing logs | ||||||||||||||||||
|
|
||||||||||||||||||
| This depends on the how cloudcore/ edgecore has been executed. | ||||||||||||||||||
| ### CloudCore / EdgeCore logs | ||||||||||||||||||
|
|
||||||||||||||||||
| 1. If `systemd` was used to start the cloudcore/ edgecore? then use `journalctl --unit <name of the service probably edgecore.service>` to view the logs. | ||||||||||||||||||
| 2. If `nohup` was used to start the cloudcore/ edgecore, either a path would have been added where the log is located, Otherwise, if the log file wasn't provided, the logs would be written to stdout. | ||||||||||||||||||
| - **Using systemd**: | ||||||||||||||||||
| ```bash | ||||||||||||||||||
| journalctl --unit cloudcore.service -f | ||||||||||||||||||
| journalctl --unit edgecore.service -f | ||||||||||||||||||
| ``` | ||||||||||||||||||
| - **Using nohup or manual execution:** | ||||||||||||||||||
| Logs are usually written to the file specified during startup, or to stdout if no file was given. | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Pod logs | ||||||||||||||||||
|
|
||||||||||||||||||
| To access pod logs, first **connect to the edge node**, then use one of the following methods: | ||||||||||||||||||
|
|
||||||||||||||||||
| 1. **Check log files directly**: | ||||||||||||||||||
| The logs are located in the `/var/log/pods` directory. | ||||||||||||||||||
| 2. **Use Docker to view container logs**: | ||||||||||||||||||
| ```bash | ||||||||||||||||||
| docker logs <container-id> | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| Alternatively, you can enable the `kubectl logs` feature to access logs from the cloud side. See [Enable kubectl logs/exec to debug pods on the edge](https://kubeedge.io/docs/advanced/debug/). | ||||||||||||||||||
|
|
||||||||||||||||||
| --- | ||||||||||||||||||
|
|
||||||||||||||||||
| ## Where do we find the pod logs | ||||||||||||||||||
| ## Important Considerations | ||||||||||||||||||
|
|
||||||||||||||||||
| Connect to the edge node and then either | ||||||||||||||||||
| - Device CRDs are **v1beta1** since v1.15. Older `v1alpha1` and `v1alpha2` CRDs are deprecated. | ||||||||||||||||||
| - Always replace version numbers (`v1.21.0` in this example) with the version you plan to install. | ||||||||||||||||||
| - Official release artifacts and checksums are on the [KubeEdge v1.21.0 release page](https://github.com/kubeedge/kubeedge/releases/tag/v1.21.0). | ||||||||||||||||||
| - CRDs are located in the [build/crds folder of the KubeEdge GitHub repository](https://github.com/kubeedge/kubeedge/tree/v1.21.0/build/crds). | ||||||||||||||||||
|
|
||||||||||||||||||
| - use the log file located in `/var/log/pods` or | ||||||||||||||||||
| - use commands like `docker logs <container id>` | ||||||||||||||||||
| <!-- end list --> | ||||||||||||||||||
|
|
||||||||||||||||||
| You can also enable `kubectl logs` feature refer to this [guide](../advanced/debug.md). | ||||||||||||||||||
| ``` | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
Comment on lines
+134
to
+135
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The file starts with a blank line and a markdown code fence. This is invalid and will cause rendering issues. The YAML frontmatter must be at the very beginning of the file, and the markdown fence is causing the entire document to be treated as a code block. Please remove the blank line and the fence.