From f71f5d199bcd99a798ae0fd49dcb0fb1b4cdd3a6 Mon Sep 17 00:00:00 2001 From: KushguptaPST Date: Sat, 31 Jan 2026 19:07:41 +0530 Subject: [PATCH 1/2] fix: resolve sidebar duplicates, plugins refactor (#424), and blog build errors Squashed commit addressing: - Plugin docs refactoring - Blog frontmatter sanitization - Netlify Docusaurus config (Node 20) Signed-off-by: KushguptaPST --- .gitignore | 27 +- .../1.4-release-en.md | 38 +- {content/en/blog => blog}/ING_case-en.md | 59 +- .../en/blog => blog}/Quick-Start-Volcano.md | 38 +- .../blog => blog}/Volcano-1.10.0-release.md | 62 +- .../blog => blog}/Volcano-1.11.0-release.md | 74 +- .../blog => blog}/Volcano-1.12.0-release.md | 52 +- .../blog => blog}/Volcano-1.13.0-release.md | 56 +- .../blog => blog}/Volcano-1.7.0-release-en.md | 72 +- .../en/blog => blog}/Volcano-1.8.2-release.md | 60 +- .../en/blog => blog}/Volcano-1.9.0-release.md | 56 +- ...lcano-community-co-construction-program.md | 79 +- {content/en/blog => blog}/aiqiyi-en.md | 72 +- blog/authors.yml | 53 + ...uted-training-and-inference-performance.md | 54 +- {content/en/blog => blog}/hpc-en.md | 71 +- .../en/blog => blog}/iflytek_case_study.md | 58 +- ...-llm-inference-for-the-cloud-native-era.md | 68 +- .../en/blog => blog}/kube-batch-customers.md | 46 +- .../en/blog => blog}/kube-batch-startup.md | 46 +- {content/en/blog => blog}/leinao-en.md | 76 +- ...g-with-volcano-in-ai-big-data-scenarios.md | 41 +- {content/en/blog => blog}/paddlepaddle-en.md | 36 +- {content/en/blog => blog}/pengcheng-en.md | 53 +- {content/en/blog => blog}/ruitian-en.md | 50 +- {content/en/blog => blog}/ruitian2-en.md | 75 +- .../volcano-2025-security-audit.md | 43 +- {content/en/blog => blog}/xiaohongshu-en.md | 59 +- build_log.txt | 2639 +++++++++++++++++ build_log_2.txt | 661 +++++ build_log_3.txt | 201 ++ build_log_4.txt | 138 + build_log_5.txt | 414 +++ build_log_blog.txt | 93 + build_log_blog_2.txt | 45 + build_log_blog_3.txt | 56 + build_log_blog_4.txt | 57 + build_log_blog_5.txt | 58 + build_log_blog_6.txt | 383 +++ build_log_blog_7.txt | 109 + build_log_blog_fix_svg.txt | 151 + build_log_debug_blog_2.txt | 60 + build_log_debug_blog_3.txt | 237 ++ build_log_debug_blog_4.txt | 65 + build_log_debug_blog_5.txt | 65 + content/en/author/volcano/_index.md | 79 - content/en/author/volcano/avatar.png | Bin 5677 -> 0 bytes content/en/author/volcano/blog-avatar.png | Bin 5661 -> 0 bytes content/en/blog/_index.md | 16 - content/en/docs/.DS_Store | Bin 6148 -> 0 bytes content/en/docs/v1-10-0/architecture.md | 42 - content/en/docs/v1-10-0/flink_on_volcano.md | 279 -- content/en/docs/v1-10-0/membership.md | 135 - .../en/docs/v1-10-0/mindspore_on_volcano.md | 77 - content/en/docs/v1-10-0/mpi_on_volcano.md | 112 - content/en/docs/v1-10-0/plugins.md | 176 -- content/en/docs/v1-10-0/pp_on_volcano.md | 242 -- content/en/docs/v1-10-0/referrals.md | 88 - content/en/docs/v1-10-0/spark_on_volcano.md | 101 - content/en/docs/v1-10-0/tf_on_volcano.md | 111 - content/en/docs/v1-11-0/architecture.md | 43 - content/en/docs/v1-11-0/cli.md | 74 - content/en/docs/v1-11-0/contribution.md | 165 -- content/en/docs/v1-11-0/flink_on_volcano.md | 279 -- .../en/docs/v1-11-0/kubeflow_on_volcano.md | 246 -- content/en/docs/v1-11-0/membership.md | 136 - .../en/docs/v1-11-0/mindspore_on_volcano.md | 77 - content/en/docs/v1-11-0/mpi_on_volcano.md | 112 - content/en/docs/v1-11-0/plugins.md | 176 -- content/en/docs/v1-11-0/pp_on_volcano.md | 242 -- content/en/docs/v1-11-0/referrals.md | 88 - .../en/docs/v1-11-0/schduler_introduction.md | 103 - content/en/docs/v1-11-0/spark_on_volcano.md | 101 - content/en/docs/v1-11-0/tf_on_volcano.md | 111 - content/en/docs/v1-11-0/tutorials.md | 277 -- content/en/docs/v1-12-0/architecture.md | 43 - content/en/docs/v1-12-0/cli.md | 74 - content/en/docs/v1-12-0/flink_on_volcano.md | 279 -- .../en/docs/v1-12-0/kubeflow_on_volcano.md | 246 -- content/en/docs/v1-12-0/membership.md | 136 - .../en/docs/v1-12-0/mindspore_on_volcano.md | 77 - content/en/docs/v1-12-0/mpi_on_volcano.md | 112 - content/en/docs/v1-12-0/plugins.md | 176 -- content/en/docs/v1-12-0/pp_on_volcano.md | 242 -- content/en/docs/v1-12-0/referrals.md | 88 - .../en/docs/v1-12-0/schduler_introduction.md | 103 - content/en/docs/v1-12-0/spark_on_volcano.md | 101 - content/en/docs/v1-12-0/tf_on_volcano.md | 111 - content/en/docs/v1-7-0/cli.md | 74 - content/en/docs/v1-7-0/podgroup.md | 104 - .../en/docs/v1-7-0/schduler_introduction.md | 103 - content/en/home/about.md | 13 - content/en/home/hero.md | 36 - content/en/home/home_slider.md | 127 - content/en/home/posts.md | 66 - content/en/home/required.md | 119 - content/en/home/skills.md | 53 - content/en/home/supporter.md | 165 -- content/en/home/talks.md | 68 - content/en/publication/_index.md | 17 - content/en/talk/_index.md | 15 - .../talk/kubecon-barcelona-2019/featured.png | Bin 273373 -> 0 bytes .../en/talk/kubecon-barcelona-2019/index.md | 87 - content/zh/author/volcano/_index.md | 79 - content/zh/author/volcano/avatar.png | Bin 5677 -> 0 bytes content/zh/author/volcano/blog-avatar.png | Bin 5661 -> 0 bytes content/zh/blog/1.4 release.md | 55 - content/zh/blog/ING_case.md | 125 - content/zh/blog/Quick-Start-Volcano.md | 76 - content/zh/blog/Volcano-1.10.0-release.md | 235 -- content/zh/blog/Volcano-1.11.0-release.md | 462 --- content/zh/blog/Volcano-1.12.0-release.md | 313 -- content/zh/blog/Volcano-1.13.0-release.md | 362 --- content/zh/blog/Volcano-1.7.0-release.md | 161 - content/zh/blog/Volcano-1.8.2-release.md | 263 -- content/zh/blog/Volcano-1.9.0-release.md | 217 -- ...lcano-community-co-construction-program.md | 113 - content/zh/blog/_index.md | 16 - content/zh/blog/aiqiyi.md | 282 -- content/zh/blog/hpc.md | 291 -- content/zh/blog/iflytek_case_study.md | 59 - content/zh/blog/introducing_kthena.md | 143 - content/zh/blog/kube-batch-customers.md | 33 - content/zh/blog/kube-batch-startup.md | 194 -- content/zh/blog/leinao-cloud-os.md | 155 - content/zh/blog/paddlepaddle.md | 614 ---- content/zh/blog/pengcheng.md | 272 -- content/zh/blog/ruitian.md | 381 --- content/zh/blog/ruitian2.md | 289 -- content/zh/blog/xiaohongshu.md | 175 -- content/zh/docs/.DS_Store | Bin 6148 -> 0 bytes content/zh/docs/_index.md | 122 - content/zh/docs/actions.md | 84 - content/zh/docs/architecture.md | 42 - content/zh/docs/cli.md | 69 - content/zh/docs/colocation.md | 624 ---- content/zh/docs/contribution.md | 166 -- content/zh/docs/cron_volcanoJob.md | 143 - content/zh/docs/descheduler.md | 151 - content/zh/docs/flink_on_volcano.md | 279 -- content/zh/docs/gpu_virtualization.md | 202 -- content/zh/docs/hierarchical_queue.md | 145 - content/zh/docs/installation.md | 124 - content/zh/docs/kubeflow_on_volcano.md | 253 -- content/zh/docs/membership.md | 130 - content/zh/docs/mindspore_on_volcano.md | 77 - content/zh/docs/mpi_on_volcano.md | 112 - content/zh/docs/multi_cluster_scheduling.md | 49 - .../docs/network_topology_aware_scheduling.md | 458 --- content/zh/docs/plugins.md | 176 -- content/zh/docs/podgroup.md | 148 - content/zh/docs/pp_on_volcano.md | 247 -- content/zh/docs/queue.md | 124 - content/zh/docs/queue_resource_management.md | 241 -- content/zh/docs/ray_on_volcano.md | 219 -- content/zh/docs/referrals.md | 91 - content/zh/docs/schduler_introduction.md | 109 - content/zh/docs/spark_on_volcano.md | 101 - content/zh/docs/tf_on_volcano.md | 111 - content/zh/docs/tutorials.md | 359 --- content/zh/docs/unified_scheduling.md | 253 -- content/zh/docs/v1-10-0/_index.md | 86 - content/zh/docs/v1-10-0/actions.md | 90 - content/zh/docs/v1-10-0/architecture.md | 41 - content/zh/docs/v1-10-0/cli.md | 68 - content/zh/docs/v1-10-0/contribution.md | 165 -- content/zh/docs/v1-10-0/flink_on_volcano.md | 279 -- content/zh/docs/v1-10-0/installation.md | 124 - .../zh/docs/v1-10-0/kubeflow_on_volcano.md | 253 -- content/zh/docs/v1-10-0/membership.md | 129 - .../zh/docs/v1-10-0/mindspore_on_volcano.md | 77 - content/zh/docs/v1-10-0/mpi_on_volcano.md | 112 - content/zh/docs/v1-10-0/plugins.md | 176 -- content/zh/docs/v1-10-0/podgroup.md | 147 - content/zh/docs/v1-10-0/pp_on_volcano.md | 247 -- content/zh/docs/v1-10-0/queue.md | 134 - content/zh/docs/v1-10-0/referrals.md | 91 - .../zh/docs/v1-10-0/schduler_introduction.md | 108 - content/zh/docs/v1-10-0/spark_on_volcano.md | 101 - content/zh/docs/v1-10-0/tf_on_volcano.md | 111 - content/zh/docs/v1-10-0/tutorials.md | 276 -- content/zh/docs/v1-10-0/vcjob.md | 352 --- content/zh/docs/v1-11-0/_index.md | 122 - content/zh/docs/v1-11-0/actions.md | 84 - content/zh/docs/v1-11-0/architecture.md | 42 - content/zh/docs/v1-11-0/cli.md | 69 - content/zh/docs/v1-11-0/colocation.md | 624 ---- content/zh/docs/v1-11-0/contribution.md | 166 -- content/zh/docs/v1-11-0/descheduler.md | 151 - content/zh/docs/v1-11-0/flink_on_volcano.md | 279 -- content/zh/docs/v1-11-0/gpu_virtualization.md | 155 - content/zh/docs/v1-11-0/hierarchical_queue.md | 145 - content/zh/docs/v1-11-0/installation.md | 124 - .../zh/docs/v1-11-0/kubeflow_on_volcano.md | 253 -- content/zh/docs/v1-11-0/membership.md | 130 - .../zh/docs/v1-11-0/mindspore_on_volcano.md | 77 - content/zh/docs/v1-11-0/mpi_on_volcano.md | 112 - .../docs/v1-11-0/multi_cluster_scheduling.md | 49 - .../network_topology_aware_scheduling.md | 301 -- content/zh/docs/v1-11-0/plugins.md | 176 -- content/zh/docs/v1-11-0/podgroup.md | 148 - content/zh/docs/v1-11-0/pp_on_volcano.md | 247 -- content/zh/docs/v1-11-0/queue.md | 124 - .../docs/v1-11-0/queue_resource_management.md | 241 -- content/zh/docs/v1-11-0/referrals.md | 91 - .../zh/docs/v1-11-0/schduler_introduction.md | 109 - content/zh/docs/v1-11-0/spark_on_volcano.md | 101 - content/zh/docs/v1-11-0/tf_on_volcano.md | 111 - content/zh/docs/v1-11-0/tutorials.md | 277 -- content/zh/docs/v1-11-0/unified_scheduling.md | 176 -- content/zh/docs/v1-11-0/vcjob.md | 353 --- content/zh/docs/v1-12-0/_index.md | 117 - content/zh/docs/v1-12-0/actions.md | 84 - content/zh/docs/v1-12-0/architecture.md | 42 - content/zh/docs/v1-12-0/cli.md | 69 - content/zh/docs/v1-12-0/colocation.md | 624 ---- content/zh/docs/v1-12-0/contribution.md | 166 -- content/zh/docs/v1-12-0/descheduler.md | 151 - content/zh/docs/v1-12-0/flink_on_volcano.md | 279 -- content/zh/docs/v1-12-0/gpu_virtualization.md | 214 -- content/zh/docs/v1-12-0/hierarchical_queue.md | 145 - content/zh/docs/v1-12-0/installation.md | 124 - .../zh/docs/v1-12-0/kubeflow_on_volcano.md | 253 -- content/zh/docs/v1-12-0/membership.md | 130 - .../zh/docs/v1-12-0/mindspore_on_volcano.md | 77 - content/zh/docs/v1-12-0/mpi_on_volcano.md | 112 - .../docs/v1-12-0/multi_cluster_scheduling.md | 49 - .../network_topology_aware_scheduling.md | 446 --- content/zh/docs/v1-12-0/plugins.md | 176 -- content/zh/docs/v1-12-0/podgroup.md | 148 - content/zh/docs/v1-12-0/pp_on_volcano.md | 247 -- content/zh/docs/v1-12-0/queue.md | 124 - .../docs/v1-12-0/queue_resource_management.md | 241 -- content/zh/docs/v1-12-0/referrals.md | 91 - .../zh/docs/v1-12-0/schduler_introduction.md | 109 - content/zh/docs/v1-12-0/spark_on_volcano.md | 101 - content/zh/docs/v1-12-0/tf_on_volcano.md | 111 - content/zh/docs/v1-12-0/tutorials.md | 358 --- content/zh/docs/v1-12-0/unified_scheduling.md | 253 -- content/zh/docs/v1-12-0/vcjob.md | 353 --- content/zh/docs/v1-7-0/_index.md | 86 - content/zh/docs/v1-7-0/actions.md | 90 - content/zh/docs/v1-7-0/architecture.md | 42 - content/zh/docs/v1-7-0/cli.md | 69 - content/zh/docs/v1-7-0/contribution.md | 166 -- content/zh/docs/v1-7-0/flink_on_volcano.md | 279 -- content/zh/docs/v1-7-0/installation.md | 123 - content/zh/docs/v1-7-0/kubeflow_on_volcano.md | 253 -- content/zh/docs/v1-7-0/membership.md | 130 - .../zh/docs/v1-7-0/mindspore_on_volcano.md | 77 - content/zh/docs/v1-7-0/mpi_on_volcano.md | 112 - content/zh/docs/v1-7-0/plugins.md | 176 -- content/zh/docs/v1-7-0/podgroup.md | 148 - content/zh/docs/v1-7-0/pp_on_volcano.md | 247 -- content/zh/docs/v1-7-0/queue.md | 135 - content/zh/docs/v1-7-0/referrals.md | 91 - .../zh/docs/v1-7-0/schduler_introduction.md | 109 - content/zh/docs/v1-7-0/spark_on_volcano.md | 101 - content/zh/docs/v1-7-0/tf_on_volcano.md | 111 - content/zh/docs/v1-7-0/tutorials.md | 277 -- content/zh/docs/v1-7-0/vcjob.md | 353 --- content/zh/docs/v1-8-2/_index.md | 86 - content/zh/docs/v1-8-2/actions.md | 90 - content/zh/docs/v1-8-2/architecture.md | 42 - content/zh/docs/v1-8-2/cli.md | 69 - content/zh/docs/v1-8-2/contribution.md | 166 -- content/zh/docs/v1-8-2/flink_on_volcano.md | 279 -- content/zh/docs/v1-8-2/installation.md | 123 - content/zh/docs/v1-8-2/kubeflow_on_volcano.md | 253 -- content/zh/docs/v1-8-2/membership.md | 130 - .../zh/docs/v1-8-2/mindspore_on_volcano.md | 77 - content/zh/docs/v1-8-2/mpi_on_volcano.md | 112 - content/zh/docs/v1-8-2/plugins.md | 176 -- content/zh/docs/v1-8-2/podgroup.md | 148 - content/zh/docs/v1-8-2/pp_on_volcano.md | 247 -- content/zh/docs/v1-8-2/queue.md | 135 - content/zh/docs/v1-8-2/referrals.md | 91 - .../zh/docs/v1-8-2/schduler_introduction.md | 109 - content/zh/docs/v1-8-2/spark_on_volcano.md | 101 - content/zh/docs/v1-8-2/tf_on_volcano.md | 111 - content/zh/docs/v1-8-2/tutorials.md | 277 -- content/zh/docs/v1-8-2/vcjob.md | 353 --- content/zh/docs/v1-9-0/_index.md | 86 - content/zh/docs/v1-9-0/actions.md | 90 - content/zh/docs/v1-9-0/architecture.md | 41 - content/zh/docs/v1-9-0/cli.md | 68 - content/zh/docs/v1-9-0/contribution.md | 165 -- content/zh/docs/v1-9-0/flink_on_volcano.md | 279 -- content/zh/docs/v1-9-0/installation.md | 124 - content/zh/docs/v1-9-0/kubeflow_on_volcano.md | 253 -- content/zh/docs/v1-9-0/membership.md | 129 - .../zh/docs/v1-9-0/mindspore_on_volcano.md | 77 - content/zh/docs/v1-9-0/mpi_on_volcano.md | 112 - content/zh/docs/v1-9-0/plugins.md | 176 -- content/zh/docs/v1-9-0/podgroup.md | 147 - content/zh/docs/v1-9-0/pp_on_volcano.md | 247 -- content/zh/docs/v1-9-0/queue.md | 134 - content/zh/docs/v1-9-0/referrals.md | 91 - .../zh/docs/v1-9-0/schduler_introduction.md | 108 - content/zh/docs/v1-9-0/spark_on_volcano.md | 101 - content/zh/docs/v1-9-0/tf_on_volcano.md | 111 - content/zh/docs/v1-9-0/tutorials.md | 276 -- content/zh/docs/v1-9-0/vcjob.md | 352 --- content/zh/docs/vcjob.md | 353 --- content/zh/home/about.md | 13 - content/zh/home/hero.md | 36 - content/zh/home/home_slider.md | 136 - content/zh/home/posts.md | 66 - content/zh/home/required.md | 118 - content/zh/home/skills.md | 53 - content/zh/home/supporter.md | 165 -- content/zh/home/talks.md | 68 - content/zh/publication/_index.md | 17 - content/zh/talk/_index.md | 15 - .../talk/kubecon-barcelona-2019/featured.png | Bin 273373 -> 0 bytes .../zh/talk/kubecon-barcelona-2019/index.md | 87 - convert_blog_frontmatter.py | 79 + {content/en/docs/v1-12-0 => docs}/actions.md | 25 +- {content/en/docs => docs}/architecture.md | 28 +- {content/en/docs => docs}/cli.md | 24 +- {content/en/docs => docs}/colocation.md | 70 +- {content/en/docs => docs}/contribution.md | 30 +- {content/en/docs => docs}/cron_volcanoJob.md | 23 +- {content/en/docs => docs}/descheduler.md | 28 +- {content/en/docs => docs}/flink_on_volcano.md | 24 +- .../v1-12-0 => docs}/gpu_virtualization.md | 39 +- .../en/docs => docs}/hierarchical_queue.md | 24 +- {content/en/docs => docs}/installation.md | 24 +- .../en/docs/_index.md => docs/introduction.md | 28 +- .../en/docs => docs}/kubeflow_on_volcano.md | 26 +- {content/en/docs => docs}/membership.md | 24 +- .../en/docs => docs}/mindspore_on_volcano.md | 24 +- {content/en/docs => docs}/mpi_on_volcano.md | 26 +- .../docs => docs}/multi_cluster_scheduling.md | 28 +- .../network_topology_aware_scheduling.md | 28 +- docs/plugins/binpack.md | 15 + docs/plugins/drf.md | 15 + docs/plugins/gang.md | 17 + docs/plugins/index.md | 11 + docs/plugins/nodeorder.md | 13 + docs/plugins/numa-aware.md | 23 + docs/plugins/predicates.md | 13 + docs/plugins/priority.md | 17 + docs/plugins/proportion.md | 13 + docs/plugins/sla.md | 13 + docs/plugins/task-topology.md | 22 + docs/plugins/tdm.md | 13 + {content/en/docs => docs}/podgroup.md | 2 +- {content/en/docs => docs}/pp_on_volcano.md | 24 +- {content/en/docs => docs}/queue.md | 30 +- .../queue_resource_management.md | 32 +- {content/en/docs => docs}/ray_on_volcano.md | 24 +- {content/en/docs => docs}/referrals.md | 26 +- docs/scheduler_introduction.md | 94 + {content/en/docs => docs}/spark_on_volcano.md | 24 +- {content/en/docs => docs}/tf_on_volcano.md | 26 +- {content/en/docs => docs}/tutorials.md | 26 +- .../en/docs => docs}/unified_scheduling.md | 21 +- .../docs/v1-7-0 => docs/v1-10-0}/actions.md | 25 +- .../v1-9-0 => docs/v1-10-0}/architecture.md | 27 +- .../en/docs/v1-9-0 => docs/v1-10-0}/cli.md | 23 +- .../v1-9-0 => docs/v1-10-0}/contribution.md | 29 +- .../v1-10-0}/flink_on_volcano.md | 24 +- .../v1-8-2/_index.md => docs/v1-10-0/index.md | 22 +- .../en/docs => docs}/v1-10-0/installation.md | 25 +- docs/v1-10-0/kubeflow_on_volcano.md | 236 ++ .../v1-9-0 => docs/v1-10-0}/membership.md | 25 +- .../v1-10-0}/mindspore_on_volcano.md | 24 +- .../v1-7-0 => docs/v1-10-0}/mpi_on_volcano.md | 26 +- {content/en/docs => docs/v1-10-0}/plugins.md | 29 +- .../docs/v1-9-0 => docs/v1-10-0}/podgroup.md | 25 +- .../v1-7-0 => docs/v1-10-0}/pp_on_volcano.md | 24 +- .../en/docs/v1-9-0 => docs/v1-10-0}/queue.md | 25 +- .../docs/v1-7-0 => docs/v1-10-0}/referrals.md | 26 +- .../v1-10-0}/schduler_introduction.md | 27 +- .../v1-10-0}/spark_on_volcano.md | 24 +- .../v1-7-0 => docs/v1-10-0}/tf_on_volcano.md | 26 +- .../docs/v1-9-0 => docs/v1-10-0}/tutorials.md | 23 +- .../en/docs/v1-9-0 => docs/v1-10-0}/vcjob.md | 23 +- {content/en/docs => docs/v1-11-0}/actions.md | 25 +- .../v1-7-0 => docs/v1-11-0}/architecture.md | 28 +- .../en/docs/v1-8-2 => docs/v1-11-0}/cli.md | 24 +- .../en/docs => docs}/v1-11-0/colocation.md | 70 +- .../v1-10-0 => docs/v1-11-0}/contribution.md | 29 +- .../en/docs => docs}/v1-11-0/descheduler.md | 28 +- .../v1-11-0}/flink_on_volcano.md | 24 +- .../v1-11-0/gpu_virtualization.md | 32 +- .../v1-11-0}/hierarchical_queue.md | 24 +- .../_index.md => docs/v1-11-0/index.md | 28 +- .../v1-12-0 => docs/v1-11-0}/installation.md | 24 +- .../v1-11-0}/kubeflow_on_volcano.md | 24 +- .../v1-7-0 => docs/v1-11-0}/membership.md | 24 +- .../v1-11-0}/mindspore_on_volcano.md | 24 +- .../v1-8-2 => docs/v1-11-0}/mpi_on_volcano.md | 26 +- .../v1-11-0/multi_cluster_scheduling.md | 28 +- .../network_topology_aware_scheduling.md | 28 +- .../docs/v1-7-0 => docs/v1-11-0}/plugins.md | 29 +- {content/en/docs => docs}/v1-11-0/podgroup.md | 2 +- .../v1-8-2 => docs/v1-11-0}/pp_on_volcano.md | 24 +- {content/en/docs => docs}/v1-11-0/queue.md | 30 +- .../v1-11-0/queue_resource_management.md | 32 +- .../docs/v1-8-2 => docs/v1-11-0}/referrals.md | 26 +- .../v1-11-0}/schduler_introduction.md | 26 +- .../v1-11-0}/spark_on_volcano.md | 24 +- .../v1-8-2 => docs/v1-11-0}/tf_on_volcano.md | 26 +- .../v1-10-0 => docs/v1-11-0}/tutorials.md | 23 +- .../v1-11-0/unified_scheduling.md | 21 +- {content/en/docs => docs/v1-11-0}/vcjob.md | 24 +- .../docs/v1-11-0 => docs/v1-12-0}/actions.md | 25 +- .../v1-8-2 => docs/v1-12-0}/architecture.md | 28 +- .../en/docs/v1-10-0 => docs/v1-12-0}/cli.md | 23 +- .../en/docs => docs}/v1-12-0/colocation.md | 70 +- .../en/docs => docs}/v1-12-0/contribution.md | 30 +- .../en/docs => docs}/v1-12-0/descheduler.md | 28 +- .../v1-12-0}/flink_on_volcano.md | 24 +- .../v1-12-0}/gpu_virtualization.md | 39 +- .../v1-12-0}/hierarchical_queue.md | 24 +- .../_index.md => docs/v1-12-0/index.md | 26 +- .../v1-11-0 => docs/v1-12-0}/installation.md | 24 +- .../v1-12-0}/kubeflow_on_volcano.md | 26 +- .../v1-8-2 => docs/v1-12-0}/membership.md | 24 +- .../v1-12-0}/mindspore_on_volcano.md | 24 +- .../v1-9-0 => docs/v1-12-0}/mpi_on_volcano.md | 26 +- .../v1-12-0/multi_cluster_scheduling.md | 28 +- .../network_topology_aware_scheduling.md | 28 +- .../docs/v1-8-2 => docs/v1-12-0}/plugins.md | 29 +- {content/en/docs => docs}/v1-12-0/podgroup.md | 2 +- .../v1-9-0 => docs/v1-12-0}/pp_on_volcano.md | 24 +- {content/en/docs => docs}/v1-12-0/queue.md | 30 +- .../v1-12-0/queue_resource_management.md | 32 +- .../docs/v1-9-0 => docs/v1-12-0}/referrals.md | 26 +- .../v1-12-0}/schduler_introduction.md | 26 +- .../v1-12-0}/spark_on_volcano.md | 24 +- .../v1-9-0 => docs/v1-12-0}/tf_on_volcano.md | 26 +- .../en/docs => docs}/v1-12-0/tutorials.md | 26 +- .../v1-12-0/unified_scheduling.md | 21 +- {content/en/docs => docs}/v1-12-0/vcjob.md | 24 +- .../en/docs/v1-9-0 => docs/v1-7-0}/actions.md | 25 +- docs/v1-7-0/architecture.md | 35 + docs/v1-7-0/cli.md | 66 + .../en/docs => docs}/v1-7-0/contribution.md | 30 +- docs/v1-7-0/flink_on_volcano.md | 271 ++ .../v1-9-0/_index.md => docs/v1-7-0/index.md | 22 +- .../en/docs => docs}/v1-7-0/installation.md | 24 +- .../v1-7-0}/kubeflow_on_volcano.md | 26 +- docs/v1-7-0/membership.md | 128 + docs/v1-7-0/mindspore_on_volcano.md | 69 + docs/v1-7-0/mpi_on_volcano.md | 104 + .../en/docs/v1-9-0 => docs/v1-7-0}/plugins.md | 29 +- .../docs/v1-8-2 => docs/v1-7-0}/podgroup.md | 26 +- docs/v1-7-0/pp_on_volcano.md | 234 ++ .../en/docs/v1-10-0 => docs/v1-7-0}/queue.md | 25 +- docs/v1-7-0/referrals.md | 80 + .../v1-7-0}/schduler_introduction.md | 27 +- docs/v1-7-0/spark_on_volcano.md | 93 + docs/v1-7-0/tf_on_volcano.md | 103 + .../docs/v1-8-2 => docs/v1-7-0}/tutorials.md | 24 +- .../en/docs/v1-10-0 => docs/v1-7-0}/vcjob.md | 23 +- {content/en/docs => docs}/v1-8-2/actions.md | 25 +- docs/v1-8-2/architecture.md | 35 + docs/v1-8-2/cli.md | 66 + .../en/docs => docs}/v1-8-2/contribution.md | 30 +- docs/v1-8-2/flink_on_volcano.md | 271 ++ .../v1-7-0/_index.md => docs/v1-8-2/index.md | 22 +- .../en/docs => docs}/v1-8-2/installation.md | 24 +- .../v1-8-2}/kubeflow_on_volcano.md | 26 +- docs/v1-8-2/membership.md | 128 + docs/v1-8-2/mindspore_on_volcano.md | 69 + docs/v1-8-2/mpi_on_volcano.md | 104 + docs/v1-8-2/plugins.md | 169 ++ .../docs/v1-10-0 => docs/v1-8-2}/podgroup.md | 25 +- docs/v1-8-2/pp_on_volcano.md | 234 ++ .../en/docs/v1-7-0 => docs/v1-8-2}/queue.md | 24 +- docs/v1-8-2/referrals.md | 80 + docs/v1-8-2/schduler_introduction.md | 94 + docs/v1-8-2/spark_on_volcano.md | 93 + docs/v1-8-2/tf_on_volcano.md | 103 + .../docs/v1-7-0 => docs/v1-8-2}/tutorials.md | 24 +- {content/en/docs => docs}/v1-8-2/vcjob.md | 24 +- .../docs/v1-10-0 => docs/v1-9-0}/actions.md | 25 +- docs/v1-9-0/architecture.md | 35 + docs/v1-9-0/cli.md | 66 + docs/v1-9-0/contribution.md | 157 + docs/v1-9-0/flink_on_volcano.md | 271 ++ .../v1-10-0/_index.md => docs/v1-9-0/index.md | 22 +- .../en/docs => docs}/v1-9-0/installation.md | 25 +- docs/v1-9-0/kubeflow_on_volcano.md | 238 ++ docs/v1-9-0/membership.md | 128 + docs/v1-9-0/mindspore_on_volcano.md | 69 + docs/v1-9-0/mpi_on_volcano.md | 104 + docs/v1-9-0/plugins.md | 169 ++ docs/v1-9-0/podgroup.md | 96 + docs/v1-9-0/pp_on_volcano.md | 234 ++ .../en/docs/v1-8-2 => docs/v1-9-0}/queue.md | 24 +- docs/v1-9-0/referrals.md | 80 + docs/v1-9-0/schduler_introduction.md | 94 + docs/v1-9-0/spark_on_volcano.md | 93 + docs/v1-9-0/tf_on_volcano.md | 103 + docs/v1-9-0/tutorials.md | 269 ++ .../en/docs/v1-7-0 => docs/v1-9-0}/vcjob.md | 24 +- {content/en/docs/v1-11-0 => docs}/vcjob.md | 24 +- docusaurus.config.js | 155 + netlify.toml | 31 +- package.json | 44 + sidebars.js | 35 + src/components/HomepageFeatures/index.js | 64 + .../HomepageFeatures/styles.module.css | 11 + src/css/custom.css | 30 + src/pages/index.js | 43 + src/pages/index.module.css | 23 + src/pages/markdown-page.md | 7 + static/.nojekyll | 0 static/_redirects | 4 +- static/img/descheduler/descheduler_EN.svg | 2 +- static/img/docusaurus-social-card.jpg | Bin 0 -> 55746 bytes static/img/docusaurus.png | Bin 0 -> 5142 bytes static/img/favicon.ico | Bin 0 -> 3626 bytes static/img/logo.svg | 1 + .../multi-cluster/volcano_global_design.svg | 2 +- static/img/undraw_docusaurus_mountain.svg | 171 ++ static/img/undraw_docusaurus_react.svg | 170 ++ static/img/undraw_docusaurus_tree.svg | 40 + 522 files changed, 13702 insertions(+), 44578 deletions(-) rename content/en/blog/1.4 release-en.md => blog/1.4-release-en.md (85%) rename {content/en/blog => blog}/ING_case-en.md (87%) rename {content/en/blog => blog}/Quick-Start-Volcano.md (80%) rename {content/en/blog => blog}/Volcano-1.10.0-release.md (91%) rename {content/en/blog => blog}/Volcano-1.11.0-release.md (94%) rename {content/en/blog => blog}/Volcano-1.12.0-release.md (96%) rename {content/en/blog => blog}/Volcano-1.13.0-release.md (94%) rename {content/en/blog => blog}/Volcano-1.7.0-release-en.md (87%) rename {content/en/blog => blog}/Volcano-1.8.2-release.md (90%) rename {content/en/blog => blog}/Volcano-1.9.0-release.md (91%) rename {content/en/blog => blog}/Volcano-community-co-construction-program.md (83%) rename {content/en/blog => blog}/aiqiyi-en.md (90%) create mode 100644 blog/authors.yml rename {content/en/blog => blog}/how-volcano-boosts-distributed-training-and-inference-performance.md (82%) rename {content/en/blog => blog}/hpc-en.md (88%) rename {content/en/blog => blog}/iflytek_case_study.md (78%) rename {content/en/blog => blog}/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md (86%) rename {content/en/blog => blog}/kube-batch-customers.md (58%) rename {content/en/blog => blog}/kube-batch-startup.md (90%) rename {content/en/blog => blog}/leinao-en.md (79%) rename content/en/blog/Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios.md => blog/meet-cloud-native-batch-computing-with-volcano-in-ai-big-data-scenarios.md (83%) rename {content/en/blog => blog}/paddlepaddle-en.md (97%) rename {content/en/blog => blog}/pengcheng-en.md (91%) rename {content/en/blog => blog}/ruitian-en.md (92%) rename {content/en/blog => blog}/ruitian2-en.md (86%) rename {content/en/blog => blog}/volcano-2025-security-audit.md (91%) rename {content/en/blog => blog}/xiaohongshu-en.md (92%) create mode 100644 build_log.txt create mode 100644 build_log_2.txt create mode 100644 build_log_3.txt create mode 100644 build_log_4.txt create mode 100644 build_log_5.txt create mode 100644 build_log_blog.txt create mode 100644 build_log_blog_2.txt create mode 100644 build_log_blog_3.txt create mode 100644 build_log_blog_4.txt create mode 100644 build_log_blog_5.txt create mode 100644 build_log_blog_6.txt create mode 100644 build_log_blog_7.txt create mode 100644 build_log_blog_fix_svg.txt create mode 100644 build_log_debug_blog_2.txt create mode 100644 build_log_debug_blog_3.txt create mode 100644 build_log_debug_blog_4.txt create mode 100644 build_log_debug_blog_5.txt delete mode 100644 content/en/author/volcano/_index.md delete mode 100644 content/en/author/volcano/avatar.png delete mode 100644 content/en/author/volcano/blog-avatar.png delete mode 100644 content/en/blog/_index.md delete mode 100644 content/en/docs/.DS_Store delete mode 100644 content/en/docs/v1-10-0/architecture.md delete mode 100644 content/en/docs/v1-10-0/flink_on_volcano.md delete mode 100644 content/en/docs/v1-10-0/membership.md delete mode 100644 content/en/docs/v1-10-0/mindspore_on_volcano.md delete mode 100644 content/en/docs/v1-10-0/mpi_on_volcano.md delete mode 100644 content/en/docs/v1-10-0/plugins.md delete mode 100644 content/en/docs/v1-10-0/pp_on_volcano.md delete mode 100644 content/en/docs/v1-10-0/referrals.md delete mode 100644 content/en/docs/v1-10-0/spark_on_volcano.md delete mode 100644 content/en/docs/v1-10-0/tf_on_volcano.md delete mode 100644 content/en/docs/v1-11-0/architecture.md delete mode 100644 content/en/docs/v1-11-0/cli.md delete mode 100644 content/en/docs/v1-11-0/contribution.md delete mode 100644 content/en/docs/v1-11-0/flink_on_volcano.md delete mode 100644 content/en/docs/v1-11-0/kubeflow_on_volcano.md delete mode 100644 content/en/docs/v1-11-0/membership.md delete mode 100644 content/en/docs/v1-11-0/mindspore_on_volcano.md delete mode 100644 content/en/docs/v1-11-0/mpi_on_volcano.md delete mode 100644 content/en/docs/v1-11-0/plugins.md delete mode 100644 content/en/docs/v1-11-0/pp_on_volcano.md delete mode 100644 content/en/docs/v1-11-0/referrals.md delete mode 100644 content/en/docs/v1-11-0/schduler_introduction.md delete mode 100644 content/en/docs/v1-11-0/spark_on_volcano.md delete mode 100644 content/en/docs/v1-11-0/tf_on_volcano.md delete mode 100644 content/en/docs/v1-11-0/tutorials.md delete mode 100644 content/en/docs/v1-12-0/architecture.md delete mode 100644 content/en/docs/v1-12-0/cli.md delete mode 100644 content/en/docs/v1-12-0/flink_on_volcano.md delete mode 100644 content/en/docs/v1-12-0/kubeflow_on_volcano.md delete mode 100644 content/en/docs/v1-12-0/membership.md delete mode 100644 content/en/docs/v1-12-0/mindspore_on_volcano.md delete mode 100644 content/en/docs/v1-12-0/mpi_on_volcano.md delete mode 100644 content/en/docs/v1-12-0/plugins.md delete mode 100644 content/en/docs/v1-12-0/pp_on_volcano.md delete mode 100644 content/en/docs/v1-12-0/referrals.md delete mode 100644 content/en/docs/v1-12-0/schduler_introduction.md delete mode 100644 content/en/docs/v1-12-0/spark_on_volcano.md delete mode 100644 content/en/docs/v1-12-0/tf_on_volcano.md delete mode 100644 content/en/docs/v1-7-0/cli.md delete mode 100644 content/en/docs/v1-7-0/podgroup.md delete mode 100644 content/en/docs/v1-7-0/schduler_introduction.md delete mode 100644 content/en/home/about.md delete mode 100644 content/en/home/hero.md delete mode 100644 content/en/home/home_slider.md delete mode 100644 content/en/home/posts.md delete mode 100644 content/en/home/required.md delete mode 100644 content/en/home/skills.md delete mode 100644 content/en/home/supporter.md delete mode 100644 content/en/home/talks.md delete mode 100644 content/en/publication/_index.md delete mode 100644 content/en/talk/_index.md delete mode 100644 content/en/talk/kubecon-barcelona-2019/featured.png delete mode 100644 content/en/talk/kubecon-barcelona-2019/index.md delete mode 100644 content/zh/author/volcano/_index.md delete mode 100644 content/zh/author/volcano/avatar.png delete mode 100644 content/zh/author/volcano/blog-avatar.png delete mode 100644 content/zh/blog/1.4 release.md delete mode 100644 content/zh/blog/ING_case.md delete mode 100644 content/zh/blog/Quick-Start-Volcano.md delete mode 100644 content/zh/blog/Volcano-1.10.0-release.md delete mode 100644 content/zh/blog/Volcano-1.11.0-release.md delete mode 100644 content/zh/blog/Volcano-1.12.0-release.md delete mode 100644 content/zh/blog/Volcano-1.13.0-release.md delete mode 100644 content/zh/blog/Volcano-1.7.0-release.md delete mode 100644 content/zh/blog/Volcano-1.8.2-release.md delete mode 100644 content/zh/blog/Volcano-1.9.0-release.md delete mode 100644 content/zh/blog/Volcano-community-co-construction-program.md delete mode 100644 content/zh/blog/_index.md delete mode 100644 content/zh/blog/aiqiyi.md delete mode 100644 content/zh/blog/hpc.md delete mode 100644 content/zh/blog/iflytek_case_study.md delete mode 100644 content/zh/blog/introducing_kthena.md delete mode 100644 content/zh/blog/kube-batch-customers.md delete mode 100644 content/zh/blog/kube-batch-startup.md delete mode 100644 content/zh/blog/leinao-cloud-os.md delete mode 100644 content/zh/blog/paddlepaddle.md delete mode 100644 content/zh/blog/pengcheng.md delete mode 100644 content/zh/blog/ruitian.md delete mode 100644 content/zh/blog/ruitian2.md delete mode 100644 content/zh/blog/xiaohongshu.md delete mode 100644 content/zh/docs/.DS_Store delete mode 100644 content/zh/docs/_index.md delete mode 100644 content/zh/docs/actions.md delete mode 100644 content/zh/docs/architecture.md delete mode 100644 content/zh/docs/cli.md delete mode 100644 content/zh/docs/colocation.md delete mode 100644 content/zh/docs/contribution.md delete mode 100644 content/zh/docs/cron_volcanoJob.md delete mode 100644 content/zh/docs/descheduler.md delete mode 100644 content/zh/docs/flink_on_volcano.md delete mode 100644 content/zh/docs/gpu_virtualization.md delete mode 100644 content/zh/docs/hierarchical_queue.md delete mode 100644 content/zh/docs/installation.md delete mode 100644 content/zh/docs/kubeflow_on_volcano.md delete mode 100644 content/zh/docs/membership.md delete mode 100644 content/zh/docs/mindspore_on_volcano.md delete mode 100644 content/zh/docs/mpi_on_volcano.md delete mode 100644 content/zh/docs/multi_cluster_scheduling.md delete mode 100644 content/zh/docs/network_topology_aware_scheduling.md delete mode 100644 content/zh/docs/plugins.md delete mode 100644 content/zh/docs/podgroup.md delete mode 100644 content/zh/docs/pp_on_volcano.md delete mode 100644 content/zh/docs/queue.md delete mode 100644 content/zh/docs/queue_resource_management.md delete mode 100644 content/zh/docs/ray_on_volcano.md delete mode 100644 content/zh/docs/referrals.md delete mode 100644 content/zh/docs/schduler_introduction.md delete mode 100644 content/zh/docs/spark_on_volcano.md delete mode 100644 content/zh/docs/tf_on_volcano.md delete mode 100644 content/zh/docs/tutorials.md delete mode 100644 content/zh/docs/unified_scheduling.md delete mode 100644 content/zh/docs/v1-10-0/_index.md delete mode 100644 content/zh/docs/v1-10-0/actions.md delete mode 100644 content/zh/docs/v1-10-0/architecture.md delete mode 100644 content/zh/docs/v1-10-0/cli.md delete mode 100644 content/zh/docs/v1-10-0/contribution.md delete mode 100644 content/zh/docs/v1-10-0/flink_on_volcano.md delete mode 100644 content/zh/docs/v1-10-0/installation.md delete mode 100644 content/zh/docs/v1-10-0/kubeflow_on_volcano.md delete mode 100644 content/zh/docs/v1-10-0/membership.md delete mode 100644 content/zh/docs/v1-10-0/mindspore_on_volcano.md delete mode 100644 content/zh/docs/v1-10-0/mpi_on_volcano.md delete mode 100644 content/zh/docs/v1-10-0/plugins.md delete mode 100644 content/zh/docs/v1-10-0/podgroup.md delete mode 100644 content/zh/docs/v1-10-0/pp_on_volcano.md delete mode 100644 content/zh/docs/v1-10-0/queue.md delete mode 100644 content/zh/docs/v1-10-0/referrals.md delete mode 100644 content/zh/docs/v1-10-0/schduler_introduction.md delete mode 100644 content/zh/docs/v1-10-0/spark_on_volcano.md delete mode 100644 content/zh/docs/v1-10-0/tf_on_volcano.md delete mode 100644 content/zh/docs/v1-10-0/tutorials.md delete mode 100644 content/zh/docs/v1-10-0/vcjob.md delete mode 100644 content/zh/docs/v1-11-0/_index.md delete mode 100644 content/zh/docs/v1-11-0/actions.md delete mode 100644 content/zh/docs/v1-11-0/architecture.md delete mode 100644 content/zh/docs/v1-11-0/cli.md delete mode 100644 content/zh/docs/v1-11-0/colocation.md delete mode 100644 content/zh/docs/v1-11-0/contribution.md delete mode 100644 content/zh/docs/v1-11-0/descheduler.md delete mode 100644 content/zh/docs/v1-11-0/flink_on_volcano.md delete mode 100644 content/zh/docs/v1-11-0/gpu_virtualization.md delete mode 100644 content/zh/docs/v1-11-0/hierarchical_queue.md delete mode 100644 content/zh/docs/v1-11-0/installation.md delete mode 100644 content/zh/docs/v1-11-0/kubeflow_on_volcano.md delete mode 100644 content/zh/docs/v1-11-0/membership.md delete mode 100644 content/zh/docs/v1-11-0/mindspore_on_volcano.md delete mode 100644 content/zh/docs/v1-11-0/mpi_on_volcano.md delete mode 100644 content/zh/docs/v1-11-0/multi_cluster_scheduling.md delete mode 100644 content/zh/docs/v1-11-0/network_topology_aware_scheduling.md delete mode 100644 content/zh/docs/v1-11-0/plugins.md delete mode 100644 content/zh/docs/v1-11-0/podgroup.md delete mode 100644 content/zh/docs/v1-11-0/pp_on_volcano.md delete mode 100644 content/zh/docs/v1-11-0/queue.md delete mode 100644 content/zh/docs/v1-11-0/queue_resource_management.md delete mode 100644 content/zh/docs/v1-11-0/referrals.md delete mode 100644 content/zh/docs/v1-11-0/schduler_introduction.md delete mode 100644 content/zh/docs/v1-11-0/spark_on_volcano.md delete mode 100644 content/zh/docs/v1-11-0/tf_on_volcano.md delete mode 100644 content/zh/docs/v1-11-0/tutorials.md delete mode 100644 content/zh/docs/v1-11-0/unified_scheduling.md delete mode 100644 content/zh/docs/v1-11-0/vcjob.md delete mode 100644 content/zh/docs/v1-12-0/_index.md delete mode 100644 content/zh/docs/v1-12-0/actions.md delete mode 100644 content/zh/docs/v1-12-0/architecture.md delete mode 100644 content/zh/docs/v1-12-0/cli.md delete mode 100644 content/zh/docs/v1-12-0/colocation.md delete mode 100644 content/zh/docs/v1-12-0/contribution.md delete mode 100644 content/zh/docs/v1-12-0/descheduler.md delete mode 100644 content/zh/docs/v1-12-0/flink_on_volcano.md delete mode 100644 content/zh/docs/v1-12-0/gpu_virtualization.md delete mode 100644 content/zh/docs/v1-12-0/hierarchical_queue.md delete mode 100644 content/zh/docs/v1-12-0/installation.md delete mode 100644 content/zh/docs/v1-12-0/kubeflow_on_volcano.md delete mode 100644 content/zh/docs/v1-12-0/membership.md delete mode 100644 content/zh/docs/v1-12-0/mindspore_on_volcano.md delete mode 100644 content/zh/docs/v1-12-0/mpi_on_volcano.md delete mode 100644 content/zh/docs/v1-12-0/multi_cluster_scheduling.md delete mode 100644 content/zh/docs/v1-12-0/network_topology_aware_scheduling.md delete mode 100644 content/zh/docs/v1-12-0/plugins.md delete mode 100644 content/zh/docs/v1-12-0/podgroup.md delete mode 100644 content/zh/docs/v1-12-0/pp_on_volcano.md delete mode 100644 content/zh/docs/v1-12-0/queue.md delete mode 100644 content/zh/docs/v1-12-0/queue_resource_management.md delete mode 100644 content/zh/docs/v1-12-0/referrals.md delete mode 100644 content/zh/docs/v1-12-0/schduler_introduction.md delete mode 100644 content/zh/docs/v1-12-0/spark_on_volcano.md delete mode 100644 content/zh/docs/v1-12-0/tf_on_volcano.md delete mode 100644 content/zh/docs/v1-12-0/tutorials.md delete mode 100644 content/zh/docs/v1-12-0/unified_scheduling.md delete mode 100644 content/zh/docs/v1-12-0/vcjob.md delete mode 100644 content/zh/docs/v1-7-0/_index.md delete mode 100644 content/zh/docs/v1-7-0/actions.md delete mode 100644 content/zh/docs/v1-7-0/architecture.md delete mode 100644 content/zh/docs/v1-7-0/cli.md delete mode 100644 content/zh/docs/v1-7-0/contribution.md delete mode 100644 content/zh/docs/v1-7-0/flink_on_volcano.md delete mode 100644 content/zh/docs/v1-7-0/installation.md delete mode 100644 content/zh/docs/v1-7-0/kubeflow_on_volcano.md delete mode 100644 content/zh/docs/v1-7-0/membership.md delete mode 100644 content/zh/docs/v1-7-0/mindspore_on_volcano.md delete mode 100644 content/zh/docs/v1-7-0/mpi_on_volcano.md delete mode 100644 content/zh/docs/v1-7-0/plugins.md delete mode 100644 content/zh/docs/v1-7-0/podgroup.md delete mode 100644 content/zh/docs/v1-7-0/pp_on_volcano.md delete mode 100644 content/zh/docs/v1-7-0/queue.md delete mode 100644 content/zh/docs/v1-7-0/referrals.md delete mode 100644 content/zh/docs/v1-7-0/schduler_introduction.md delete mode 100644 content/zh/docs/v1-7-0/spark_on_volcano.md delete mode 100644 content/zh/docs/v1-7-0/tf_on_volcano.md delete mode 100644 content/zh/docs/v1-7-0/tutorials.md delete mode 100644 content/zh/docs/v1-7-0/vcjob.md delete mode 100644 content/zh/docs/v1-8-2/_index.md delete mode 100644 content/zh/docs/v1-8-2/actions.md delete mode 100644 content/zh/docs/v1-8-2/architecture.md delete mode 100644 content/zh/docs/v1-8-2/cli.md delete mode 100644 content/zh/docs/v1-8-2/contribution.md delete mode 100644 content/zh/docs/v1-8-2/flink_on_volcano.md delete mode 100644 content/zh/docs/v1-8-2/installation.md delete mode 100644 content/zh/docs/v1-8-2/kubeflow_on_volcano.md delete mode 100644 content/zh/docs/v1-8-2/membership.md delete mode 100644 content/zh/docs/v1-8-2/mindspore_on_volcano.md delete mode 100644 content/zh/docs/v1-8-2/mpi_on_volcano.md delete mode 100644 content/zh/docs/v1-8-2/plugins.md delete mode 100644 content/zh/docs/v1-8-2/podgroup.md delete mode 100644 content/zh/docs/v1-8-2/pp_on_volcano.md delete mode 100644 content/zh/docs/v1-8-2/queue.md delete mode 100644 content/zh/docs/v1-8-2/referrals.md delete mode 100644 content/zh/docs/v1-8-2/schduler_introduction.md delete mode 100644 content/zh/docs/v1-8-2/spark_on_volcano.md delete mode 100644 content/zh/docs/v1-8-2/tf_on_volcano.md delete mode 100644 content/zh/docs/v1-8-2/tutorials.md delete mode 100644 content/zh/docs/v1-8-2/vcjob.md delete mode 100644 content/zh/docs/v1-9-0/_index.md delete mode 100644 content/zh/docs/v1-9-0/actions.md delete mode 100644 content/zh/docs/v1-9-0/architecture.md delete mode 100644 content/zh/docs/v1-9-0/cli.md delete mode 100644 content/zh/docs/v1-9-0/contribution.md delete mode 100644 content/zh/docs/v1-9-0/flink_on_volcano.md delete mode 100644 content/zh/docs/v1-9-0/installation.md delete mode 100644 content/zh/docs/v1-9-0/kubeflow_on_volcano.md delete mode 100644 content/zh/docs/v1-9-0/membership.md delete mode 100644 content/zh/docs/v1-9-0/mindspore_on_volcano.md delete mode 100644 content/zh/docs/v1-9-0/mpi_on_volcano.md delete mode 100644 content/zh/docs/v1-9-0/plugins.md delete mode 100644 content/zh/docs/v1-9-0/podgroup.md delete mode 100644 content/zh/docs/v1-9-0/pp_on_volcano.md delete mode 100644 content/zh/docs/v1-9-0/queue.md delete mode 100644 content/zh/docs/v1-9-0/referrals.md delete mode 100644 content/zh/docs/v1-9-0/schduler_introduction.md delete mode 100644 content/zh/docs/v1-9-0/spark_on_volcano.md delete mode 100644 content/zh/docs/v1-9-0/tf_on_volcano.md delete mode 100644 content/zh/docs/v1-9-0/tutorials.md delete mode 100644 content/zh/docs/v1-9-0/vcjob.md delete mode 100644 content/zh/docs/vcjob.md delete mode 100644 content/zh/home/about.md delete mode 100644 content/zh/home/hero.md delete mode 100644 content/zh/home/home_slider.md delete mode 100644 content/zh/home/posts.md delete mode 100644 content/zh/home/required.md delete mode 100644 content/zh/home/skills.md delete mode 100644 content/zh/home/supporter.md delete mode 100644 content/zh/home/talks.md delete mode 100644 content/zh/publication/_index.md delete mode 100644 content/zh/talk/_index.md delete mode 100644 content/zh/talk/kubecon-barcelona-2019/featured.png delete mode 100644 content/zh/talk/kubecon-barcelona-2019/index.md create mode 100644 convert_blog_frontmatter.py rename {content/en/docs/v1-12-0 => docs}/actions.md (90%) rename {content/en/docs => docs}/architecture.md (62%) rename {content/en/docs => docs}/cli.md (83%) rename {content/en/docs => docs}/colocation.md (91%) rename {content/en/docs => docs}/contribution.md (92%) rename {content/en/docs => docs}/cron_volcanoJob.md (94%) rename {content/en/docs => docs}/descheduler.md (96%) rename {content/en/docs => docs}/flink_on_volcano.md (96%) rename {content/en/docs/v1-12-0 => docs}/gpu_virtualization.md (95%) rename {content/en/docs => docs}/hierarchical_queue.md (94%) rename {content/en/docs => docs}/installation.md (91%) rename content/en/docs/_index.md => docs/introduction.md (93%) rename {content/en/docs => docs}/kubeflow_on_volcano.md (95%) rename {content/en/docs => docs}/membership.md (94%) rename {content/en/docs => docs}/mindspore_on_volcano.md (81%) rename {content/en/docs => docs}/mpi_on_volcano.md (87%) rename {content/en/docs => docs}/multi_cluster_scheduling.md (86%) rename {content/en/docs => docs}/network_topology_aware_scheduling.md (98%) create mode 100644 docs/plugins/binpack.md create mode 100644 docs/plugins/drf.md create mode 100644 docs/plugins/gang.md create mode 100644 docs/plugins/index.md create mode 100644 docs/plugins/nodeorder.md create mode 100644 docs/plugins/numa-aware.md create mode 100644 docs/plugins/predicates.md create mode 100644 docs/plugins/priority.md create mode 100644 docs/plugins/proportion.md create mode 100644 docs/plugins/sla.md create mode 100644 docs/plugins/task-topology.md create mode 100644 docs/plugins/tdm.md rename {content/en/docs => docs}/podgroup.md (98%) rename {content/en/docs => docs}/pp_on_volcano.md (95%) rename {content/en/docs => docs}/queue.md (84%) rename {content/en/docs => docs}/queue_resource_management.md (90%) rename {content/en/docs => docs}/ray_on_volcano.md (96%) rename {content/en/docs => docs}/referrals.md (95%) create mode 100644 docs/scheduler_introduction.md rename {content/en/docs => docs}/spark_on_volcano.md (90%) rename {content/en/docs => docs}/tf_on_volcano.md (88%) rename {content/en/docs => docs}/tutorials.md (96%) rename {content/en/docs => docs}/unified_scheduling.md (98%) rename {content/en/docs/v1-7-0 => docs/v1-10-0}/actions.md (90%) rename {content/en/docs/v1-9-0 => docs/v1-10-0}/architecture.md (62%) rename {content/en/docs/v1-9-0 => docs/v1-10-0}/cli.md (83%) rename {content/en/docs/v1-9-0 => docs/v1-10-0}/contribution.md (92%) rename {content/en/docs/v1-7-0 => docs/v1-10-0}/flink_on_volcano.md (96%) rename content/en/docs/v1-8-2/_index.md => docs/v1-10-0/index.md (90%) rename {content/en/docs => docs}/v1-10-0/installation.md (90%) create mode 100644 docs/v1-10-0/kubeflow_on_volcano.md rename {content/en/docs/v1-9-0 => docs/v1-10-0}/membership.md (94%) rename {content/en/docs/v1-7-0 => docs/v1-10-0}/mindspore_on_volcano.md (81%) rename {content/en/docs/v1-7-0 => docs/v1-10-0}/mpi_on_volcano.md (87%) rename {content/en/docs => docs/v1-10-0}/plugins.md (96%) rename {content/en/docs/v1-9-0 => docs/v1-10-0}/podgroup.md (92%) rename {content/en/docs/v1-7-0 => docs/v1-10-0}/pp_on_volcano.md (95%) rename {content/en/docs/v1-9-0 => docs/v1-10-0}/queue.md (94%) rename {content/en/docs/v1-7-0 => docs/v1-10-0}/referrals.md (94%) rename {content/en/docs => docs/v1-10-0}/schduler_introduction.md (93%) rename {content/en/docs/v1-7-0 => docs/v1-10-0}/spark_on_volcano.md (90%) rename {content/en/docs/v1-7-0 => docs/v1-10-0}/tf_on_volcano.md (88%) rename {content/en/docs/v1-9-0 => docs/v1-10-0}/tutorials.md (95%) rename {content/en/docs/v1-9-0 => docs/v1-10-0}/vcjob.md (97%) rename {content/en/docs => docs/v1-11-0}/actions.md (90%) rename {content/en/docs/v1-7-0 => docs/v1-11-0}/architecture.md (62%) rename {content/en/docs/v1-8-2 => docs/v1-11-0}/cli.md (83%) rename {content/en/docs => docs}/v1-11-0/colocation.md (91%) rename {content/en/docs/v1-10-0 => docs/v1-11-0}/contribution.md (92%) rename {content/en/docs => docs}/v1-11-0/descheduler.md (96%) rename {content/en/docs/v1-8-2 => docs/v1-11-0}/flink_on_volcano.md (96%) rename {content/en/docs => docs}/v1-11-0/gpu_virtualization.md (88%) rename {content/en/docs/v1-12-0 => docs/v1-11-0}/hierarchical_queue.md (94%) rename content/en/docs/v1-11-0/_index.md => docs/v1-11-0/index.md (92%) rename {content/en/docs/v1-12-0 => docs/v1-11-0}/installation.md (91%) rename {content/en/docs/v1-10-0 => docs/v1-11-0}/kubeflow_on_volcano.md (95%) rename {content/en/docs/v1-7-0 => docs/v1-11-0}/membership.md (94%) rename {content/en/docs/v1-8-2 => docs/v1-11-0}/mindspore_on_volcano.md (81%) rename {content/en/docs/v1-8-2 => docs/v1-11-0}/mpi_on_volcano.md (87%) rename {content/en/docs => docs}/v1-11-0/multi_cluster_scheduling.md (86%) rename {content/en/docs => docs}/v1-11-0/network_topology_aware_scheduling.md (96%) rename {content/en/docs/v1-7-0 => docs/v1-11-0}/plugins.md (96%) rename {content/en/docs => docs}/v1-11-0/podgroup.md (98%) rename {content/en/docs/v1-8-2 => docs/v1-11-0}/pp_on_volcano.md (95%) rename {content/en/docs => docs}/v1-11-0/queue.md (84%) rename {content/en/docs => docs}/v1-11-0/queue_resource_management.md (90%) rename {content/en/docs/v1-8-2 => docs/v1-11-0}/referrals.md (94%) rename {content/en/docs/v1-9-0 => docs/v1-11-0}/schduler_introduction.md (93%) rename {content/en/docs/v1-8-2 => docs/v1-11-0}/spark_on_volcano.md (90%) rename {content/en/docs/v1-8-2 => docs/v1-11-0}/tf_on_volcano.md (88%) rename {content/en/docs/v1-10-0 => docs/v1-11-0}/tutorials.md (95%) rename {content/en/docs => docs}/v1-11-0/unified_scheduling.md (97%) rename {content/en/docs => docs/v1-11-0}/vcjob.md (97%) rename {content/en/docs/v1-11-0 => docs/v1-12-0}/actions.md (90%) rename {content/en/docs/v1-8-2 => docs/v1-12-0}/architecture.md (62%) rename {content/en/docs/v1-10-0 => docs/v1-12-0}/cli.md (83%) rename {content/en/docs => docs}/v1-12-0/colocation.md (91%) rename {content/en/docs => docs}/v1-12-0/contribution.md (92%) rename {content/en/docs => docs}/v1-12-0/descheduler.md (96%) rename {content/en/docs/v1-9-0 => docs/v1-12-0}/flink_on_volcano.md (96%) rename {content/en/docs => docs/v1-12-0}/gpu_virtualization.md (95%) rename {content/en/docs/v1-11-0 => docs/v1-12-0}/hierarchical_queue.md (94%) rename content/en/docs/v1-12-0/_index.md => docs/v1-12-0/index.md (94%) rename {content/en/docs/v1-11-0 => docs/v1-12-0}/installation.md (91%) rename {content/en/docs/v1-8-2 => docs/v1-12-0}/kubeflow_on_volcano.md (95%) rename {content/en/docs/v1-8-2 => docs/v1-12-0}/membership.md (94%) rename {content/en/docs/v1-9-0 => docs/v1-12-0}/mindspore_on_volcano.md (81%) rename {content/en/docs/v1-9-0 => docs/v1-12-0}/mpi_on_volcano.md (87%) rename {content/en/docs => docs}/v1-12-0/multi_cluster_scheduling.md (86%) rename {content/en/docs => docs}/v1-12-0/network_topology_aware_scheduling.md (97%) rename {content/en/docs/v1-8-2 => docs/v1-12-0}/plugins.md (96%) rename {content/en/docs => docs}/v1-12-0/podgroup.md (98%) rename {content/en/docs/v1-9-0 => docs/v1-12-0}/pp_on_volcano.md (95%) rename {content/en/docs => docs}/v1-12-0/queue.md (84%) rename {content/en/docs => docs}/v1-12-0/queue_resource_management.md (90%) rename {content/en/docs/v1-9-0 => docs/v1-12-0}/referrals.md (94%) rename {content/en/docs/v1-10-0 => docs/v1-12-0}/schduler_introduction.md (93%) rename {content/en/docs/v1-9-0 => docs/v1-12-0}/spark_on_volcano.md (90%) rename {content/en/docs/v1-9-0 => docs/v1-12-0}/tf_on_volcano.md (88%) rename {content/en/docs => docs}/v1-12-0/tutorials.md (96%) rename {content/en/docs => docs}/v1-12-0/unified_scheduling.md (98%) rename {content/en/docs => docs}/v1-12-0/vcjob.md (97%) rename {content/en/docs/v1-9-0 => docs/v1-7-0}/actions.md (90%) create mode 100644 docs/v1-7-0/architecture.md create mode 100644 docs/v1-7-0/cli.md rename {content/en/docs => docs}/v1-7-0/contribution.md (92%) create mode 100644 docs/v1-7-0/flink_on_volcano.md rename content/en/docs/v1-9-0/_index.md => docs/v1-7-0/index.md (90%) rename {content/en/docs => docs}/v1-7-0/installation.md (90%) rename {content/en/docs/v1-9-0 => docs/v1-7-0}/kubeflow_on_volcano.md (95%) create mode 100644 docs/v1-7-0/membership.md create mode 100644 docs/v1-7-0/mindspore_on_volcano.md create mode 100644 docs/v1-7-0/mpi_on_volcano.md rename {content/en/docs/v1-9-0 => docs/v1-7-0}/plugins.md (96%) rename {content/en/docs/v1-8-2 => docs/v1-7-0}/podgroup.md (92%) create mode 100644 docs/v1-7-0/pp_on_volcano.md rename {content/en/docs/v1-10-0 => docs/v1-7-0}/queue.md (94%) create mode 100644 docs/v1-7-0/referrals.md rename {content/en/docs/v1-8-2 => docs/v1-7-0}/schduler_introduction.md (93%) create mode 100644 docs/v1-7-0/spark_on_volcano.md create mode 100644 docs/v1-7-0/tf_on_volcano.md rename {content/en/docs/v1-8-2 => docs/v1-7-0}/tutorials.md (95%) rename {content/en/docs/v1-10-0 => docs/v1-7-0}/vcjob.md (97%) rename {content/en/docs => docs}/v1-8-2/actions.md (90%) create mode 100644 docs/v1-8-2/architecture.md create mode 100644 docs/v1-8-2/cli.md rename {content/en/docs => docs}/v1-8-2/contribution.md (92%) create mode 100644 docs/v1-8-2/flink_on_volcano.md rename content/en/docs/v1-7-0/_index.md => docs/v1-8-2/index.md (90%) rename {content/en/docs => docs}/v1-8-2/installation.md (90%) rename {content/en/docs/v1-7-0 => docs/v1-8-2}/kubeflow_on_volcano.md (95%) create mode 100644 docs/v1-8-2/membership.md create mode 100644 docs/v1-8-2/mindspore_on_volcano.md create mode 100644 docs/v1-8-2/mpi_on_volcano.md create mode 100644 docs/v1-8-2/plugins.md rename {content/en/docs/v1-10-0 => docs/v1-8-2}/podgroup.md (92%) create mode 100644 docs/v1-8-2/pp_on_volcano.md rename {content/en/docs/v1-7-0 => docs/v1-8-2}/queue.md (94%) create mode 100644 docs/v1-8-2/referrals.md create mode 100644 docs/v1-8-2/schduler_introduction.md create mode 100644 docs/v1-8-2/spark_on_volcano.md create mode 100644 docs/v1-8-2/tf_on_volcano.md rename {content/en/docs/v1-7-0 => docs/v1-8-2}/tutorials.md (95%) rename {content/en/docs => docs}/v1-8-2/vcjob.md (97%) rename {content/en/docs/v1-10-0 => docs/v1-9-0}/actions.md (90%) create mode 100644 docs/v1-9-0/architecture.md create mode 100644 docs/v1-9-0/cli.md create mode 100644 docs/v1-9-0/contribution.md create mode 100644 docs/v1-9-0/flink_on_volcano.md rename content/en/docs/v1-10-0/_index.md => docs/v1-9-0/index.md (90%) rename {content/en/docs => docs}/v1-9-0/installation.md (90%) create mode 100644 docs/v1-9-0/kubeflow_on_volcano.md create mode 100644 docs/v1-9-0/membership.md create mode 100644 docs/v1-9-0/mindspore_on_volcano.md create mode 100644 docs/v1-9-0/mpi_on_volcano.md create mode 100644 docs/v1-9-0/plugins.md create mode 100644 docs/v1-9-0/podgroup.md create mode 100644 docs/v1-9-0/pp_on_volcano.md rename {content/en/docs/v1-8-2 => docs/v1-9-0}/queue.md (94%) create mode 100644 docs/v1-9-0/referrals.md create mode 100644 docs/v1-9-0/schduler_introduction.md create mode 100644 docs/v1-9-0/spark_on_volcano.md create mode 100644 docs/v1-9-0/tf_on_volcano.md create mode 100644 docs/v1-9-0/tutorials.md rename {content/en/docs/v1-7-0 => docs/v1-9-0}/vcjob.md (97%) rename {content/en/docs/v1-11-0 => docs}/vcjob.md (97%) create mode 100644 docusaurus.config.js create mode 100644 package.json create mode 100644 sidebars.js create mode 100644 src/components/HomepageFeatures/index.js create mode 100644 src/components/HomepageFeatures/styles.module.css create mode 100644 src/css/custom.css create mode 100644 src/pages/index.js create mode 100644 src/pages/index.module.css create mode 100644 src/pages/markdown-page.md create mode 100644 static/.nojekyll create mode 100644 static/img/docusaurus-social-card.jpg create mode 100644 static/img/docusaurus.png create mode 100644 static/img/favicon.ico create mode 100644 static/img/logo.svg create mode 100644 static/img/undraw_docusaurus_mountain.svg create mode 100644 static/img/undraw_docusaurus_react.svg create mode 100644 static/img/undraw_docusaurus_tree.svg diff --git a/.gitignore b/.gitignore index d2c22fd1..33190b11 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,23 @@ -.idea/* -resources* -public -.vscode/* +# Dependencies +/node_modules + +# Production +/build + +# Generated files +.docusaurus +.cache-loader + +# Misc .DS_Store -.hugo_build.lock \ No newline at end of file +.env.local +.env.development.local +.env.test.local +.env.production.local + +npm-debug.log* +yarn-debug.log* +yarn-error.log* + +# Legacy backups +hugo_backup/ diff --git a/content/en/blog/1.4 release-en.md b/blog/1.4-release-en.md similarity index 85% rename from content/en/blog/1.4 release-en.md rename to blog/1.4-release-en.md index e8cd5383..93c4e11b 100644 --- a/content/en/blog/1.4 release-en.md +++ b/blog/1.4-release-en.md @@ -1,28 +1,16 @@ -+++ -title = "Volcano v1.4 (Beta) Release Note" -description = "Volcano v1.4 (Beta) Release Includes New Features Such as NUMA-Aware" -subtitle = "" - -date = 2021-08-31 -lastmod = 2021-09-13 -datemonth = "Sep" -dateyear = "2021" -dateday = 13 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Thor-wl"] - -tags = ["Tutorials"] -summary = "Volcano v1.4 (Beta) Release Includes New Features Such as NUMA-Aware" - -# Add menu entry to sidebar. -linktitle = "Volcano v1.4 (Beta) Release Note" -[menu.posts] -parent = "tutorials" -weight = 12 -+++ +--- +title: "Volcano v1.4 (Beta) Release Note" +description: "Volcano v1.4 (Beta) Release Includes New Features Such as NUMA-Aware" +subtitle: "" +date: 2021-08-31 +draft: false +authors: + - Thor-wl +tags: + - Tutorials +summary: "Volcano v1.4 (Beta) Release Includes New Features Such as NUMA-Aware" +--- +--- >This article was firstly released at `Container Cube` on September 6th, 2021, refer to[Volcano v1.4.0-Beta发布，支持NUMA-Aware等多个重要特性](https://mp.weixin.qq.com/s/S5JAQI0uLoTEx0lvYDXM4Q) diff --git a/content/en/blog/ING_case-en.md b/blog/ING_case-en.md similarity index 87% rename from content/en/blog/ING_case-en.md rename to blog/ING_case-en.md index 2daa1f14..26e4206e 100644 --- a/content/en/blog/ING_case-en.md +++ b/blog/ING_case-en.md @@ -1,28 +1,15 @@ -+++ -title = "ING Bank: How Volcano Empowers Their Big Data Analytics Platform" -description = "ING Bank: How Volcano Empowers Their Big Data Analytics Platform" -subtitle = "" - -date = 2022-12-28 -lastmod = 2022-12-28 -datemonth = "Dec" -dateyear = "2022" -dateday = 28 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "ING Bank: How Volcano Empowers Their Big Data Analytics Platform" - -# Add menu entry to sidebar. -linktitle = "ING Bank: How Volcano Empowers Their Big Data Analytics Platform" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ +--- +title: "ING Bank: How Volcano Empowers Their Big Data Analytics Platform" +description: "ING Bank: How Volcano Empowers Their Big Data Analytics Platform" +subtitle: "" +date: 2022-12-28 +draft: false +authors: + - volcano +tags: + - Practice +summary: "ING Bank: How Volcano Empowers Their Big Data Analytics Platform" +--- >2On October 26, 2022, Krzysztof Adamski and Tinco Boekestijn from ING Group delivered a keynote speech "Efficient Scheduling Of High Performance Batch Computing For Analytics Workloads With Volcano" at KubeCon North America. The speech focused on how Volcano, a cloud native batch computing project, supports high-performance scheduling for big data analytics jobs on ING's data management platform. More details: [KubeCon + CloudNativeCon North America](https://events.linuxfoundation.org/archive/2022/kubecon-cloudnativecon-north-america/program/schedule/) @@ -38,7 +25,7 @@ ING provides services in more than 40 countries around the world. Core businesse Regulations and restrictions on banking vary depending on the country/region. Data silos, data security, and compliance requirements can be really challenging. It is not easy to introduce new technologies. Therefore, ING builds their Data Analytics Platform (DAP) to provide secure, self-service functionality for employees to manage services throughout the entire process. -{{
}} +![](/img/ing-1.png) In 2013, they conceptualized data platform. In 2018, ING introduced cloud native technologies to upgrade their infrastructure platform. Since then, more and more employees and departments turn to the platform, and by now, there are more than 400 projects on the data index platform. @@ -51,7 +38,7 @@ They aim to meet all analytics needs in a highly secure, self-service platform t ## Challenges and Solutions -{{
}} +![](/img/ing-2.png) ING is shifting from Hadoop to Kubernetes. They met some challenges in job management and multi-framework support. For example: @@ -70,20 +57,20 @@ ING is shifting from Hadoop to Kubernetes. They met some challenges in job manag Managing applications (stateless and even stateful ones) with Kubernetes would be a perfect choice, if Kubernetes is as user-friendly as Yarn in the scheduling and management of batch computing jobs. Yarn also provides limited support, for example, on TensorFlow and PyTorch. Therefore, ING looked for better solutions. __Kubernetes + Hadoop__ -{{
}} +![](/img/ing-3.png) When managing clusters, ING once separated Hadoop and Kubernetes. They ran almost all Spark jobs in Hadoop clusters, and other tasks and algorithms in Kubernetes clusters. They want to run all the jobs in Kubernetes clusters to simplify management. -{{
}} +![](/img/ing-4.png) When Kubernetes and Yarn work together, Kubernetes and Hadoop resources are statically divided. During office hours, Hadoop applications and Kubernetes use their own resources. Spark tasks, when heavily pressured, cannot be allocated extra resources. At night, there are only batch processing tasks in clusters. All Kubernetes resources are idle but cannot be allocated to Hadoop. In this case, resources are not fully used. __Kubernetes with Volcano__ -{{
}} +![](/img/ing-5.png) When managing clusters with Kubernetes and scheduling Spark tasks with Volcano, resources do not need to be statically divided. Cluster resources can be dynamically re-allocated based on the priorities and resource pressure of pods, batch tasks, and interactive tasks, which greatly improves the overall utilization of cluster resources. For example, during office hours, idle resources of common service applications can be used by batch and interactive applications temporarily. In holidays or nights, batch applications can use all cluster resources for data computing. -{{
}} +![](/img/ing-6.png) Volcano is a batch scheduling engine developed for Kubernetes with the following capabilities: - Job queues with weighted priority @@ -96,10 +83,10 @@ Volcano supplements Kubernetes in batch scheduling. Since Apache Spark 3.3, Volc ## Highlighted Features __Redundancy and Local Affinity__ -{{
}} +![](/img/ing-7.png) Volcano retains the affinity and anti-affinity policies for pods in Kubernetes, and adds those for tasks. -{{
}} +![](/img/ing-8.png) The idea of DRF is that in a multi-resource environment, resource allocation should be determined by the dominant share of an entity (user or queue). The volcano-scheduler observes the dominant resource requested by each job and uses it as a measure of cluster resource usage. Based on this dominant resource, the volcano-scheduler calculates the share of the job. The job with a lower share has a higher scheduling priority. For example, a cluster has 18 CPUs and 72 GB memory in total. User1 and User2 are each allocated one queue. Any submitted job will get its scheduling priority based on the dominant resource. @@ -112,15 +99,15 @@ Under a DRF policy, the job with a lower share will be first scheduled, that is, Queue resources in a cluster can be divided by configuring weights. However, overcommitted tasks in a queue can use the idle resources in other queues. In this example, after using up the CPUs of its own queue, User2 can use the idle CPUs of User1. When User1 commits a new task, it triggers resource preemption and reclaims the resources occupied by other queues. __Resource Reservation__ -{{
}} +![](/img/ing-9.png) Batch computing tasks and other services may preempt resources and cause conflicts. Assume there are two available nodes in a cluster and we need to deploy a unified service layer in the cluster to provide services externally, such as Presto or cache services like Alluxio, batch computing tasks may have already taken all resources and we can't deploy or upgrade that service layer. Therefore, ING's platform now allows users to reserve some resources for other services. __DRF Dashboard__ -{{
}} +![](/img/ing-10.png) ING built a DRF scheduling dashboard based on the monitoring data from Volcano to obtain scheduling data at different layers. In the service cluster, ING stores the tasks of interactive users in one queue, and the computing tasks of all key projects running on the data platform in another queue. ING can take certain resources from other queues to the key project queue, but that won't do any good to the tasks of interactive users. ING is considering displaying the peak hours of cluster use to provide users with more information. With this, users can decide when to start their tasks based on the cluster resource readiness, improving computing performance without complex configurations in the background. -{{
}} +![](/img/ing-11.png) ## Summary Volcano abstracts batch task scheduling, allowing Kubernetes to better serve ING in task scheduling. ING will contribute their developed functions to the community, such as the DRF dashboard, idle resource reservation on each node, auto queue management, new Prometheus monitoring metrics, Grafana dashboard updates, kube-state-metrics update, and cluster role restrictions. diff --git a/content/en/blog/Quick-Start-Volcano.md b/blog/Quick-Start-Volcano.md similarity index 80% rename from content/en/blog/Quick-Start-Volcano.md rename to blog/Quick-Start-Volcano.md index 2392a6f0..726c30ec 100644 --- a/content/en/blog/Quick-Start-Volcano.md +++ b/blog/Quick-Start-Volcano.md @@ -1,29 +1,15 @@ -+++ -title = "Quick Start Guide for Volcano" -description = "Bring up the Volcano in any K8s Cluster within few mins" -subtitle ="" - -date = 2019-03-28 -lastmod = 2019-03-29 -datemonth = "Mar" -dateyear = "2019" -dateday = 28 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Volcano"] -authors_img = "/img/icon_user.svg" - -tags = ["Tutorials"] -summary = "Quick Start Guide for Volcano" - -# Add menu entry to sidebar. -linktitle = "Quick Start Guide for Volcano" -[menu.posts] - parent = "tutorials" - weight = 1 -+++ +--- +title: "Quick Start Guide for Volcano" +description: "Bring up the Volcano in any K8s Cluster within few mins" +subtitle: "" +date: 2019-03-28 +draft: false +authors: + - Volcano +tags: + - Tutorials +summary: "Quick Start Guide for Volcano" +--- # Quick Start Guide The easiest way to deploy Volcano is using Helm charts. ### Preparation diff --git a/content/en/blog/Volcano-1.10.0-release.md b/blog/Volcano-1.10.0-release.md similarity index 91% rename from content/en/blog/Volcano-1.10.0-release.md rename to blog/Volcano-1.10.0-release.md index 87f558b2..df0566a7 100644 --- a/content/en/blog/Volcano-1.10.0-release.md +++ b/blog/Volcano-1.10.0-release.md @@ -1,28 +1,34 @@ -+++ -title = "Volcano v1.10.0 Available Now" -description = "New features: Support Queue Priority Scheduling Strategy, Enable Fine-Grained GPU Resource Sharing and Reclaim, Introduce Pod Scheduling Readiness Support, Add Sidecar Container Scheduling Capabilities, Enhance Vcctl Command Line Tool, Ensure Compatibility with Kubernetes v1.30, Strengthen Volcano Security Measures, Optimize Volcano for Large-Scale Performance, Improve GPU Monitoring Function, Optimize Helm Chart Installation And Upgrade Processes, etc." -subtitle = "" - -date = 2024-09-29 -lastmod = 2024-09-29 -datemonth = "Sep" -dateyear = "2024" -dateday = 29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "New features: Support Queue Priority Scheduling Strategy, Enable Fine-Grained GPU Resource Sharing and Reclaim, Introduce Pod Scheduling Readiness Support, Add Sidecar Container Scheduling Capabilities, Enhance Vcctl Command Line Tool, Ensure Compatibility with Kubernetes v1.30, Strengthen Volcano Security Measures, Optimize Volcano for Large-Scale Performance, Improve GPU Monitoring Function, Optimize Helm Chart Installation And Upgrade Processes, etc." - -# Add menu entry to sidebar. -linktitle = "Volcano v1.10.0 Available Now" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ +--- +title: Volcano v1.10.0 Available Now +description: 'New features: Support Queue Priority Scheduling Strategy, Enable Fine-Grained + GPU Resource Sharing and Reclaim, Introduce Pod Scheduling Readiness Support, Add + Sidecar Container Scheduling Capabilities, Enhance Vcctl Command Line Tool, Ensure + Compatibility with Kubernetes v1.30, Strengthen Volcano Security Measures, Optimize + Volcano for Large-Scale Performance, Improve GPU Monitoring Function, Optimize Helm + Chart Installation And Upgrade Processes, etc.' +subtitle: '' +date: 2024-09-29 +lastmod: '2024-09-29' +datemonth: Sep +dateyear: '2024' +dateday: '29' +draft: false +toc: true +type: posts +authors: +- volcano +tags: +- Practice +summary: 'New features: Support Queue Priority Scheduling Strategy, Enable Fine-Grained + GPU Resource Sharing and Reclaim, Introduce Pod Scheduling Readiness Support, Add + Sidecar Container Scheduling Capabilities, Enhance Vcctl Command Line Tool, Ensure + Compatibility with Kubernetes v1.30, Strengthen Volcano Security Measures, Optimize + Volcano for Large-Scale Performance, Improve GPU Monitoring Function, Optimize Helm + Chart Installation And Upgrade Processes, etc.' +linktitle: Volcano v1.10.0 Available Now +parent: tutorials +weight: '6' +--- On Sep 19, 2024, UTC+8, Volcano version v1.10.0 was officially released. This version introduced the following new features: @@ -47,7 +53,7 @@ On Sep 19, 2024, UTC+8, Volcano version v1.10.0 was officially released. This ve - **Optimize Helm Chart Installation And Upgrade Processes** -{{
}} +![](/img/volcano_logo.png) Volcano is the industry-first cloud native batch computing project. Open-sourced at KubeCon Shanghai in June 2019, it became an official CNCF project in April 2020. In April 2022, Volcano was promoted to a CNCF incubating project. By now, more than 600 global developers have committed code to the project. The community is seeing growing popularity among developers, partners, and users. ## Key Features @@ -117,9 +123,9 @@ For more details on the proportion plugin, please visit: [Proportion Plugin](htt Once a Pod is created, it is considered ready for scheduling. In Kube-scheduler, it will try its best to find a suitable node to place all pending Pods. However, in reality, some Pods may be in a "lack of necessary resources" state for a long time. These Pods actually interfere with the decision-making and operation of the scheduler (and downstream components such as Cluster AutoScaler) in an unnecessary way, causing problems such as resource waste. Pod Scheduling Readiness is a new feature of Kube-sheduler. In Kubernetes v.1.30 GA, it has become a stable feature. It controls the scheduling timing of Pods by setting the schedulingGates field of the Pod. -
{{
}} + ![](/img/v1.10.0/podSchedulingGates.svg) Pod SchedulingGates -
+ In previous versions, Volcano has integrated all algorithms of the K8s default scheduler, fully covering the native scheduling functions of Kube-scheduler. Therefore, Volcano can completely replace Kube-scheduler as a unified scheduler under the cloud native platform, supporting unified scheduling of microservices and AI/big data workloads. In the latest version v1.10, Volcano has introduced Pod Scheduling Readiness scheduling capability to further meet users' scheduling needs in diverse scenarios. diff --git a/content/en/blog/Volcano-1.11.0-release.md b/blog/Volcano-1.11.0-release.md similarity index 94% rename from content/en/blog/Volcano-1.11.0-release.md rename to blog/Volcano-1.11.0-release.md index 6afa798c..b3f413ef 100644 --- a/content/en/blog/Volcano-1.11.0-release.md +++ b/blog/Volcano-1.11.0-release.md @@ -1,28 +1,34 @@ -+++ -title = "Volcano v1.11.0 Available Now" -description = "New Features: Network topology-aware scheduling, Elastic hierarchical queues, Multi-cluster AI job scheduling, online/offline workloads colocation with dynamic resource oversubscription, Load-aware descheduling, Fine-grained job fault recovery policies, Volcano Dashboard for resource visualization, Supports for Kubernetes v1.31, Volcano Job supports for Preemption Policy, and Performance optimizations for large-scale scenarios" -subtitle = "" - -date = 2025-02-07 -lastmod = 2025-02-07 -datemonth = "Feb" -dateyear = "2025" -dateday = 07 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "New Features: Network topology-aware scheduling, Elastic hierarchical queues, Multi-cluster AI job scheduling, online/offline workloads colocation with dynamic resource oversubscription, Load-aware descheduling, Fine-grained job fault recovery policies, Volcano Dashboard for resource visualization, Supports for Kubernetes v1.31, Volcano Job supports for Preemption Policy, and Performance optimizations for large-scale scenarios" - -# Add menu entry to sidebar. -linktitle = "Volcano v1.11.0 Available Now" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ +--- +title: Volcano v1.11.0 Available Now +description: 'New Features: Network topology-aware scheduling, Elastic hierarchical + queues, Multi-cluster AI job scheduling, online/offline workloads colocation with + dynamic resource oversubscription, Load-aware descheduling, Fine-grained job fault + recovery policies, Volcano Dashboard for resource visualization, Supports for Kubernetes + v1.31, Volcano Job supports for Preemption Policy, and Performance optimizations + for large-scale scenarios' +subtitle: '' +date: 2025-02-07 +lastmod: '2025-02-07' +datemonth: Feb +dateyear: '2025' +dateday: '07' +draft: false +toc: true +type: posts +authors: +- volcano +tags: +- Practice +summary: 'New Features: Network topology-aware scheduling, Elastic hierarchical queues, + Multi-cluster AI job scheduling, online/offline workloads colocation with dynamic + resource oversubscription, Load-aware descheduling, Fine-grained job fault recovery + policies, Volcano Dashboard for resource visualization, Supports for Kubernetes + v1.31, Volcano Job supports for Preemption Policy, and Performance optimizations + for large-scale scenarios' +linktitle: Volcano v1.11.0 Available Now +parent: tutorials +weight: '6' +--- # Volcano v1.11 released: A New Era of Cloud-Native Scheduling for AI and Big Data @@ -72,8 +78,8 @@ To abstract away the differences in data center network types, Volcano defines a A HyperNode represents a network topology performance domain, typically mapped to a switch. Multiple HyperNodes connect hierarchically to form a tree structure. For example: -
{{
}} -
+ ![](/img/network-topology/hypernode-example.png) + - **Leaf HyperNodes** (s0, s1, s2, s3): Represent actual cluster nodes. - **Non-Leaf HyperNodes** (s4, s5, s6): Represent other HyperNodes. @@ -184,7 +190,7 @@ Volcano’s elastic hierarchical queues offer the following key features to meet A simple hierarchical queue structure might look like this: -{{
}} +![](/img/hierarchical-queue-example.png) - **Root Queue:** Manages global resource allocation. - **Department Queues:** Represent resource pools for different departments or teams. @@ -217,8 +223,8 @@ Volcano Global enhances Karmada with the following features to meet the complex 3. **Job Priority Scheduling and Queuing:** Enables job-level priority scheduling and queuing across clusters, ensuring critical tasks are executed promptly. 4. **Multi-Tenant Fair Scheduling:** Provides fair resource allocation across tenants, preventing resource contention. -
{{
}} -
+ ![](/img/multi-cluster/volcano_global_design.svg) + For detailed deployment and user guide, please refer to: **[Multi-Cluster AI Job Scheduling | Volcano](https://volcano.sh/en/docs/multi_cluster_scheduling/)**. @@ -251,9 +257,9 @@ Volcano’s cloud-native colocation solution provides end-to-end resource isolat **Enhanced OS:** Volcano implements fine-grained QoS guarantees at the kernel level, using cgroups to set resource limits for online and offline workloads, ensuring online workloads receive sufficient resources even under high load. -
{{
}} + ![](/img/colocation/architecture.png) Architecture -
+ #### **Core Capabilities: Balancing Resource Utilization and Stability** @@ -283,8 +289,8 @@ In Kubernetes clusters, dynamic workload changes often lead to uneven node resou - **Uneven Node Resource Utilization:** Balances node load when some nodes are overutilized while others are underutilized. - **Hotspot Node Management:** Migrates Pods from overloaded nodes to ensure stability. -
{{
}} -
+ ![](/img/descheduler/descheduler_EN.svg) + #### **Technical Highlights:** diff --git a/content/en/blog/Volcano-1.12.0-release.md b/blog/Volcano-1.12.0-release.md similarity index 96% rename from content/en/blog/Volcano-1.12.0-release.md rename to blog/Volcano-1.12.0-release.md index cde626ed..fbf7acc2 100644 --- a/content/en/blog/Volcano-1.12.0-release.md +++ b/blog/Volcano-1.12.0-release.md @@ -1,28 +1,30 @@ -+++ -title = "Volcano v1.12.0 Available Now" -description = "New features: Network Topology-Aware Scheduling (Alpha), Dynamic MIG Partitioning for GPU Virtualization, DRA Support, Queue Capacity Management in Volcano Global, Security Enhancements, Performance Optimizations, Gang Scheduling for Generic Workloads, Job Flow Improvements, and Kubernetes v1.32 Support." -subtitle = "" - -date = 2025-06-12 -lastmod = 2025-06-12 -datemonth = "June" -dateyear = "2025" -dateday = 12 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "New features: Network Topology-Aware Scheduling (Alpha), Dynamic MIG Partitioning for GPU Virtualization, DRA Support, Queue Capacity Management in Volcano Global, Security Enhancements, Performance Optimizations, Gang Scheduling for Generic Workloads, Job Flow Improvements, and Kubernetes v1.32 Support." - -# Add menu entry to sidebar. -linktitle = "Volcano v1.12.0 Available Now" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ +--- +title: Volcano v1.12.0 Available Now +description: 'New features: Network Topology-Aware Scheduling (Alpha), Dynamic MIG + Partitioning for GPU Virtualization, DRA Support, Queue Capacity Management in Volcano + Global, Security Enhancements, Performance Optimizations, Gang Scheduling for Generic + Workloads, Job Flow Improvements, and Kubernetes v1.32 Support.' +subtitle: '' +date: 2025-06-12 +lastmod: '2025-06-12' +datemonth: June +dateyear: '2025' +dateday: '12' +draft: false +toc: true +type: posts +authors: +- volcano +tags: +- Practice +summary: 'New features: Network Topology-Aware Scheduling (Alpha), Dynamic MIG Partitioning + for GPU Virtualization, DRA Support, Queue Capacity Management in Volcano Global, + Security Enhancements, Performance Optimizations, Gang Scheduling for Generic Workloads, + Job Flow Improvements, and Kubernetes v1.32 Support.' +linktitle: Volcano v1.12.0 Available Now +parent: tutorials +weight: '6' +--- ## Volcano v1.12 released: Advancing Cloud-Native AI and Batch Computing diff --git a/content/en/blog/Volcano-1.13.0-release.md b/blog/Volcano-1.13.0-release.md similarity index 94% rename from content/en/blog/Volcano-1.13.0-release.md rename to blog/Volcano-1.13.0-release.md index b07a3d24..fe7c264b 100644 --- a/content/en/blog/Volcano-1.13.0-release.md +++ b/blog/Volcano-1.13.0-release.md @@ -1,28 +1,34 @@ -+++ -title = "Volcano v1.13 Released: Comprehensive Enhancement of Scheduling Capabilities for LLM Training and Inference" -description = "New Features: LeaderWorkerSet support for large model inference, Cron VolcanoJob, Label-based HyperNode auto-discovery, Native Ray framework support, HCCL plugin support, Enhanced NodeGroup functionality, ResourceStrategyFit plugin, Colocation decoupled from OS, Custom oversubscription resource names, Kubernetes v1.33 support, and more" -subtitle = "" - -date = 2025-09-29 -lastmod = 2025-09-29 -datemonth = "Sep" -dateyear = "2025" -dateday = 29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "New Features: LeaderWorkerSet support for large model inference, Cron VolcanoJob, Label-based HyperNode auto-discovery, Native Ray framework support, HCCL plugin support, Enhanced NodeGroup functionality, ResourceStrategyFit plugin, Colocation decoupled from OS, Custom oversubscription resource names, Kubernetes v1.33 support, and more" - -# Add menu entry to sidebar. -linktitle = "Volcano v1.13 Released: Comprehensive Enhancement of Scheduling Capabilities for LLM Training and Inference" -[menu.posts] -parent = "tutorials" -weight = 5 -+++ +--- +title: 'Volcano v1.13 Released: Comprehensive Enhancement of Scheduling Capabilities + for LLM Training and Inference' +description: 'New Features: LeaderWorkerSet support for large model inference, Cron + VolcanoJob, Label-based HyperNode auto-discovery, Native Ray framework support, + HCCL plugin support, Enhanced NodeGroup functionality, ResourceStrategyFit plugin, + Colocation decoupled from OS, Custom oversubscription resource names, Kubernetes + v1.33 support, and more' +subtitle: '' +date: 2025-09-29 +lastmod: '2025-09-29' +datemonth: Sep +dateyear: '2025' +dateday: '29' +draft: false +toc: true +type: posts +authors: +- volcano +tags: +- Practice +summary: 'New Features: LeaderWorkerSet support for large model inference, Cron VolcanoJob, + Label-based HyperNode auto-discovery, Native Ray framework support, HCCL plugin + support, Enhanced NodeGroup functionality, ResourceStrategyFit plugin, Colocation + decoupled from OS, Custom oversubscription resource names, Kubernetes v1.33 support, + and more' +linktitle: 'Volcano v1.13 Released: Comprehensive Enhancement of Scheduling Capabilities + for LLM Training and Inference' +parent: tutorials +weight: '5' +--- # Volcano v1.13 Released: Comprehensive Enhancement of Scheduling Capabilities for LLM Training and Inference diff --git a/content/en/blog/Volcano-1.7.0-release-en.md b/blog/Volcano-1.7.0-release-en.md similarity index 87% rename from content/en/blog/Volcano-1.7.0-release-en.md rename to blog/Volcano-1.7.0-release-en.md index 881dd6cd..ce47b2e9 100644 --- a/content/en/blog/Volcano-1.7.0-release-en.md +++ b/blog/Volcano-1.7.0-release-en.md @@ -1,28 +1,28 @@ -+++ -title = "Volcano 1.7.0 Available Now" -description = "New features: enhanced plugin for PyTorch Jobs, Ray on Volcano, enhanced scheduling for general Kubernetes services, multi-architecture images of Volcano, and optimized queue status info" -subtitle = "" - -date = 2023-01-12 -lastmod = 2023-01-12 -datemonth = "Jan" -dateyear = "2023" -dateday = 12 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "New features: enhanced plugin for PyTorch Jobs, Ray on Volcano, enhanced scheduling for general Kubernetes services, multi-architecture images of Volcano, and optimized queue status info" - -# Add menu entry to sidebar. -linktitle = "Volcano 1.7.0 Available Now" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ +--- +title: Volcano 1.7.0 Available Now +description: 'New features: enhanced plugin for PyTorch Jobs, Ray on Volcano, enhanced + scheduling for general Kubernetes services, multi-architecture images of Volcano, + and optimized queue status info' +subtitle: '' +date: 2023-01-12 +lastmod: '2023-01-12' +datemonth: Jan +dateyear: '2023' +dateday: '12' +draft: false +toc: true +type: posts +authors: +- volcano +tags: +- Practice +summary: 'New features: enhanced plugin for PyTorch Jobs, Ray on Volcano, enhanced + scheduling for general Kubernetes services, multi-architecture images of Volcano, + and optimized queue status info' +linktitle: Volcano 1.7.0 Available Now +parent: tutorials +weight: '6' +---

Volcano 1.7.0 is now available with the following new features: @@ -33,7 +33,7 @@ Volcano 1.7.0 is now available with the following new features: - **multi-architecture images of Volcano** - **optimized queue status info** -{{
}} +![](/img/volcano_logo.png) Volcano is the industry-first cloud native batch computing project. Open-sourced at KubeCon Shanghai in June 2019, it became an official CNCF project in April 2020. In April 2022, Volcano was promoted to a CNCF incubating project. By now, more than 490 global developers have committed code to the project. The community is seeing growing popularity among developers, partners, and users. ### Key Features @@ -47,9 +47,9 @@ Other enhanced plugins include those for TensorFlow, MPI, and PyTorch Jobs. They Volcano also provides an extended development framework for you to tailor Job plugins to your needs. -Design Documentation: [Pytorch-plugin](https://github.com/volcano-sh/volcano/blob/master/docs/design/distributed-framework-plugins.md#pytorch-plugin)
-User Guide: [Pytorch-plugin-user-guide](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_pytorch_plugin.md#pytorch-plugin-user-guide)
-Issue：[#2292](https://github.com/volcano-sh/volcano/issues/2292)
+Design Documentation: [Pytorch-plugin](https://github.com/volcano-sh/volcano/blob/master/docs/design/distributed-framework-plugins.md#pytorch-plugin)
+User Guide: [Pytorch-plugin-user-guide](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_pytorch_plugin.md#pytorch-plugin-user-guide)
+Issue：[#2292](https://github.com/volcano-sh/volcano/issues/2292)
#### 2. Ray on Volcano @@ -59,8 +59,8 @@ As machine learning workloads are hosting computing jobs at a density higher tha For users running multiple types of Jobs, Volcano partners with Ray to provide high-performance batch scheduling. Ray on Volcano has been released in [KubeRay v0.4](https://github.com/ray-project/kuberay/releases/tag/v0.4.0). -User Guide: [KubeRay-integration-with-Volcano](https://ray-project.github.io/kuberay/guidance/volcano-integration/#kuberay-integration-with-volcano)
-Issue: [#2429](https://github.com/volcano-sh/volcano/issues/2429)， [#213](https://github.com/ray-project/kuberay/issues/213)
+User Guide: [KubeRay-integration-with-Volcano](https://ray-project.github.io/kuberay/guidance/volcano-integration/#kuberay-integration-with-volcano)
+Issue: [#2429](https://github.com/volcano-sh/volcano/issues/2429)， [#213](https://github.com/ray-project/kuberay/issues/213)
#### 3. Enhance Scheduling for General Kubernetes Services Schedulers have their own advantages according to the use case. For example, in batch computing, Volcano provides more scheduling policies and capabilities. In general scheduling, the Kubernetes default scheduler is more balanced. However, it's often the case that a user runs multiple types of tasks in the same cluster. When there are both batch computing and general tasks, scheduling can be a challenge. @@ -83,8 +83,8 @@ Issue: [#2394](https://github.com/volcano-sh/volcano/issues/2394)，[#2510](http #### 4. Multi-architecture Images You can now compile multi-architecture Volcano images by a few clicks through cross compilation. For example, you can compile the base images of the amd64 and arm64 architectures on an amd64 host and push the images to the image repository. During installation and deployment, the system automatically selects a proper image based on the host architecture for you, more user-friendly than before. -User Guide: [building-docker-images](https://github.com/volcano-sh/volcano/blob/master/docs/development/development.md#building-docker-images)
-Issue: [#2435](https://github.com/volcano-sh/volcano/pull/2435)
+User Guide: [building-docker-images](https://github.com/volcano-sh/volcano/blob/master/docs/development/development.md#building-docker-images)
+Issue: [#2435](https://github.com/volcano-sh/volcano/pull/2435)
#### 5. Optimized Queue Status Info Volcano can now collect statistics on allocated resources in real time to the queue status info, which eases dynamic resource adjustment and puts cluster resources into good use. @@ -98,7 +98,7 @@ Issue: [#2571](https://github.com/volcano-sh/volcano/issues/2571) ### Contributors Volcano 1.7.0 is brought into being from hundreds of code commits from 29 contributors. Thanks for your contributions. -**Contributors on GitHub:**
+**Contributors on GitHub:**
@@ -153,8 +153,8 @@ Volcano 1.7.0 is brought into being from hundreds of code commits from 29 contri
@xiaoxubeii
#### Links -Release note：[v1.7.0](https://github.com/volcano-sh/volcano/releases/tag/v1.7.0)
-Branch：[release-1.7](https://github.com/volcano-sh/volcano/tree/release-1.7)
+Release note：[v1.7.0](https://github.com/volcano-sh/volcano/releases/tag/v1.7.0)
+Branch：[release-1.7](https://github.com/volcano-sh/volcano/tree/release-1.7)
### About Volcano Volcano is designed for high-performance computing applications such as AI, big data, gene sequencing, and rendering, and supports mainstream general computing frameworks. More than 26,000 global developers joined us, among whom the in-house ones come from companies such as Huawei, AWS, Baidu, Tencent, JD, and Xiaohongshu. There are 2,800 Stars and 670 Forks for the project. Volcano has been proven feasible for mass data computing and analytics, such as AI, big data, and gene sequencing. Supported frameworks include Spark, Flink, TensorFlow, PyTorch, Argo, MindSpore, Paddlepaddle, Kubeflow, MPI, Horovod, MXNet, KubeGene, and Ray. The ecosystem is thriving with more developers and use cases coming up. \ No newline at end of file diff --git a/content/en/blog/Volcano-1.8.2-release.md b/blog/Volcano-1.8.2-release.md similarity index 90% rename from content/en/blog/Volcano-1.8.2-release.md rename to blog/Volcano-1.8.2-release.md index 3cd7f34b..6afb4606 100644 --- a/content/en/blog/Volcano-1.8.2-release.md +++ b/blog/Volcano-1.8.2-release.md @@ -1,28 +1,32 @@ -+++ -title = "Volcano v1.8.2 Available Now" -description = "New features: Support for vGPU scheduling and isolation, support for vGPU and user-defined resource preemption capabilities, addition of JobFlow workflow scheduling engine, node load-aware scheduling and rescheduling support for diverse monitoring systems, optimization of Volcano's ability to schedule microservices, optimization of Volcano charts packages for publishing and archiving, etc." -subtitle = "" - -date = 2024-01-31 -lastmod = 2024-01-31 -datemonth = "Jan" -dateyear = "2024" -dateday = 31 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "New features: Support for vGPU scheduling and isolation, support for vGPU and user-defined resource preemption capabilities, addition of JobFlow workflow scheduling engine, node load-aware scheduling and rescheduling support for diverse monitoring systems, optimization of Volcano's ability to schedule microservices, optimization of Volcano charts packages for publishing and archiving, etc." - -# Add menu entry to sidebar. -linktitle = "Volcano v1.8.2 Available Now" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ +--- +title: Volcano v1.8.2 Available Now +description: 'New features: Support for vGPU scheduling and isolation, support for + vGPU and user-defined resource preemption capabilities, addition of JobFlow workflow + scheduling engine, node load-aware scheduling and rescheduling support for diverse + monitoring systems, optimization of Volcano''s ability to schedule microservices, + optimization of Volcano charts packages for publishing and archiving, etc.' +subtitle: '' +date: 2024-01-31 +lastmod: '2024-01-31' +datemonth: Jan +dateyear: '2024' +dateday: '31' +draft: false +toc: true +type: posts +authors: +- volcano +tags: +- Practice +summary: 'New features: Support for vGPU scheduling and isolation, support for vGPU + and user-defined resource preemption capabilities, addition of JobFlow workflow + scheduling engine, node load-aware scheduling and rescheduling support for diverse + monitoring systems, optimization of Volcano''s ability to schedule microservices, + optimization of Volcano charts packages for publishing and archiving, etc.' +linktitle: Volcano v1.8.2 Available Now +parent: tutorials +weight: '6' +--- On January 9, 2024, UTC+8, Volcano version v1.8.2 was officially released. This version added the following new features: @@ -39,7 +43,7 @@ On January 9, 2024, UTC+8, Volcano version v1.8.2 was officially released. This - **Optimization of Volcano charts packages for publishing and archiving** -{{
}} +![](/img/volcano_logo.png) Volcano is the industry-first cloud native batch computing project. Open-sourced at KubeCon Shanghai in June 2019, it became an official CNCF project in April 2020. In April 2022, Volcano was promoted to a CNCF incubating project. By now, more than 600 global developers have committed code to the project. The community is seeing growing popularity among developers, partners, and users. ### Key Features @@ -85,7 +89,7 @@ JobFlow is a lightweight task flow orchestration engine that focuses on Volcano' A demonstration of a JobFlow task running is shown below: -
{{
}}
+ ![](/img/v1.8.2/jobflow.gif) For more information about JobFlow, please refer to: @@ -180,7 +184,7 @@ https://github.com/volcano-sh/helm-charts Volcano 1.8.2 is brought into being from hundreds of code commits from 33 contributors. Thanks for your contributions. -**Contributors on GitHub:**
+**Contributors on GitHub:**
diff --git a/content/en/blog/Volcano-1.9.0-release.md b/blog/Volcano-1.9.0-release.md similarity index 91% rename from content/en/blog/Volcano-1.9.0-release.md rename to blog/Volcano-1.9.0-release.md index 75a76f3c..c9e6f637 100644 --- a/content/en/blog/Volcano-1.9.0-release.md +++ b/blog/Volcano-1.9.0-release.md @@ -1,28 +1,30 @@ -+++ -title = "Volcano v1.9.0 Available Now" -description = "New features: Support elastic queue capacity scheduling, Supports affinity scheduling between queues and nodes, GPU sharing feature supports node scoring scheduling, Volcano Support for Kubernetes v1.29, Enhance scheduler metrics, Add license compliance check, Improve scheduling stability, etc." -subtitle = "" - -date = 2024-05-21 -lastmod = 2024-05-21 -datemonth = "May" -dateyear = "2024" -dateday = 21 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "New features: Support elastic queue capacity scheduling, Supports affinity scheduling between queues and nodes, GPU sharing feature supports node scoring scheduling, Volcano Support for Kubernetes v1.29, Enhance scheduler metrics, Add license compliance check, Improve scheduling stability, etc." - -# Add menu entry to sidebar. -linktitle = "Volcano v1.9.0 Available Now" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ +--- +title: Volcano v1.9.0 Available Now +description: 'New features: Support elastic queue capacity scheduling, Supports affinity + scheduling between queues and nodes, GPU sharing feature supports node scoring scheduling, + Volcano Support for Kubernetes v1.29, Enhance scheduler metrics, Add license compliance + check, Improve scheduling stability, etc.' +subtitle: '' +date: 2024-05-21 +lastmod: '2024-05-21' +datemonth: May +dateyear: '2024' +dateday: '21' +draft: false +toc: true +type: posts +authors: +- volcano +tags: +- Practice +summary: 'New features: Support elastic queue capacity scheduling, Supports affinity + scheduling between queues and nodes, GPU sharing feature supports node scoring scheduling, + Volcano Support for Kubernetes v1.29, Enhance scheduler metrics, Add license compliance + check, Improve scheduling stability, etc.' +linktitle: Volcano v1.9.0 Available Now +parent: tutorials +weight: '6' +--- On May 21, 2024, UTC+8, Volcano version v1.9.0 was officially released. This version added the following new features: @@ -40,7 +42,7 @@ On May 21, 2024, UTC+8, Volcano version v1.9.0 was officially released. This ver - **Improve scheduling stability** -{{
}} +![](/img/volcano_logo.png) Volcano is the industry-first cloud native batch computing project. Open-sourced at KubeCon Shanghai in June 2019, it became an official CNCF project in April 2020. In April 2022, Volcano was promoted to a CNCF incubating project. By now, more than 600 global developers have committed code to the project. The community is seeing growing popularity among developers, partners, and users. ### Key Features @@ -162,7 +164,7 @@ Volcano v1.9.0 has done more optimization in preemption, retry for scheduling fa Volcano 1.9.0 is brought into being from hundreds of code commits from many contributors. Thanks for your contributions. -**Contributors on GitHub:**
+**Contributors on GitHub:**
@shaobo76
diff --git a/content/en/blog/Volcano-community-co-construction-program.md b/blog/Volcano-community-co-construction-program.md similarity index 83% rename from content/en/blog/Volcano-community-co-construction-program.md rename to blog/Volcano-community-co-construction-program.md index 940497aa..615094ac 100644 --- a/content/en/blog/Volcano-community-co-construction-program.md +++ b/blog/Volcano-community-co-construction-program.md @@ -1,50 +1,49 @@ -+++ -title = "Volcano Community Co-construction Program" -description = "Huawei Cloud Joins Hands with 11 Partners to Launch Volcano Community Co-construction Program" -subtitle = "" +--- +title: Volcano Community Co-construction Program +description: Huawei Cloud Joins Hands with 11 Partners to Launch Volcano Community + Co-construction Program +subtitle: '' +date: 2023-08-11 +lastmod: '2023-08-11' +datemonth: Aug +dateyear: '2023' +dateday: '11' +draft: false +toc: true +type: posts +authors: +- volcano +tags: +- Tutorials +summary: Huawei Cloud Joins Hands with 11 Partners to Launch Volcano Community Co-construction + Program +linktitle: Huawei Cloud Joins Hands with 11 Partners to Launch Volcano Community Co-construction + Program +parent: tutorials +weight: '6' +--- + + + +![](/img/volcano_logo.png) -date = 2023-08-11 -lastmod = 2023-08-11 -datemonth = "Aug" -dateyear = "2023" -dateday = 11 -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] -tags = ["Tutorials"] -summary = "Huawei Cloud Joins Hands with 11 Partners to Launch Volcano Community Co-construction Program" -# Add menu entry to sidebar. -linktitle = "Huawei Cloud Joins Hands with 11 Partners to Launch Volcano Community Co-construction Program" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ - - As artificial intelligence (AI) technologies advance and large language models (LLMs) grow more popular, the demand for AI compute has been booming. This has generated huge demand for high-performance scheduling for the AI and for hardware like AI chips. - -{{
}} - - - - Volcano is the first cloud native batch computing project in the industry. In 2019, it was donated by Huawei Cloud to the Cloud Native Computing Foundation (CNCF) and became CNCF's first and only batch computing incubator project. Volcano provides unified high-performance job management for AI, big data, and high-performance computing (HPC) and supports a variety of high-level scheduling policies, including online and offline scheduling, AI elastic training scheduling, service level agreement (SLA), topology-based scheduling, fairness, load aware scheduling, rescheduling, preemption, and reclamation. It offers unified lifecycle management, job dependency management, and task dependency management for workloads like Spark, Flink, PyTorch, MPI, and TensorFlow. In terms of fine-grained resource management, Volcano supports min-max queue resource management, queue resource reservation, and dynamic resource sharing for multi-tenant resource leasing or preemption. Additionally, Volcano schedules heterogeneous resources including x86, Arm, GPUs, and Ascend, and provides refined scheduling of CPUs and GPUs. Users can allocate resources based on their requirements and significantly improve cost-effectiveness using Volcano. The Volcano community has attracted **more than 58,000 developers worldwide** and won more than **3,200 Stars and over 730 Forks** on GitHub. The contributors include Huawei, AWS, IBM, Baidu, Tencent, JD, Xiaohongshu, 4Paradigm, BoCloud, DaoCloud, Ruitian Capital, Qiniu Cloud, Yinqing Technology, ByteDance, Kuaishou, Unisound, Infosys, Visa, NetEase, Red Hat, Kingsoft Cloud, Inspur, ZTE, Oracle, and iQIYI. -**More than 50 cases** related to Volcano have been implemented. These cases are widely distributed in industries such as Internet, advanced manufacturing, finance, life sciences, scientific research, autonomous driving, and medicine. They cover massive data computing and analysis scenarios like AI, big data, genomic sequencing, and rendering. The main users are Tencent, Amazon, ING Bank, Baidu, Xiaohongshu, DiDi, 360, iQIYI, Leinao, Pengcheng Laboratory, Cruise, Li Auto, Unisound, Ximalaya, Vipshop, GrandOmics, BOSS Zhipin, and so on. **With the expansion of the Volcano ecosystems, more and more users are highly willing to join the community.** **Huawei Cloud has worked with 11 partners to launch the Volcano community co-construction program and cultivate a more prosperous Volcano ecosystem.** +**More than 50 cases** related to Volcano have been implemented. These cases are widely distributed in industries such as Internet, advanced manufacturing, finance, life sciences, scientific research, autonomous driving, and medicine. They cover massive data computing and analysis scenarios like AI, big data, genomic sequencing, and rendering. The main users are Tencent, Amazon, ING Bank, Baidu, Xiaohongshu, DiDi, 360, iQIYI, Leinao, Pengcheng Laboratory, Cruise, Li Auto, Unisound, Ximalaya, Vipshop, GrandOmics, BOSS Zhipin, and so on. **With the expansion of the Volcano ecosystems, more and more users are highly willing to join the community.** + - According to **Deng Mingkun, General Manager of Huawei Cloud Open Source Services**, "The cloud native batch computing project, Volcano, has been widely adopted in domains such as AI, big data, genomic sequencing, rendering, transcoding, multimedia, and finance, since June 2019. A group of industry users not only actively promote the implementation of Volcano in production environments, but also contribute a lot to the Volcano community based on their own experience. Huawei Cloud intends to work with partners to launch the Volcano community co-construction program to create a more prosperous Volcano ecosystem and help more enterprises accelerate their cloud native progress." **The first batch of members to join the program are Baidu, BoCloud, 4Paradigm, Vipshop, Ruitian Capital, Leinao, Pinlan, 360, NetEase Shufan, Ximalaya, and BOSS Zhipin.** -
{{
}}
+ ![](/img/Volcano community co-construction program/co-construction-1.jpg) According to **Zhou Ti, the tech lead of Baidu's PaddlePaddle open source ecosystem**, "PaddlePaddle and Volcano jointly released the PaddlePaddle on Volcano solution to improve PaddlePaddle's computing efficiency. As a platform for high-performance computing, Volcano makes up for Kubernetes' lack of basic capabilities in machine learning, deep learning, HPC, and big data computing. Additionally, Volcano enhances the batch creation and lifecycle management of computing tasks, fair-share scheduling and other aspects on the basis of the native Kubernetes capability. These features meet PaddlePaddle's basic requirements." @@ -78,29 +77,29 @@ According to **Zhou Ti, the tech lead of Baidu's PaddlePaddle open source ecosys **The owner of BOSS Zhipin AI fundamental platform team** said, "BOSS Zhipin builds infrastructures based on Volcano in AI and big data computing scenarios. Volcano's powerful batch processing and robust scheduling policies are very convenient for us. They help support complex service scenarios and greatly improve BOSS Zhipin's cluster resource utilization and stability. With the support of its robust ecosystem and the community, Volcano has greatly helped our technological and business development." -We look forward to working with more organizations to build a more inclusive Volcano community. +We look forward to working with more organizations to build a more inclusive Volcano community. + + + + -
- **Introduction to the Volcano Community Co-construction Program** -
- The Volcano community launched the co-construction program to more quickly include users into the Volcano community, to accelerate cloud native progress, and to ensure a diverse Volcano ecosystem. Through this program, you will have opportunities for technological guidance, promotion, as well as online and offline technological sharing. If your company or organization recognizes the value that Volcano has to offer, wants help using Volcano, or wants to exert their technological influence, consider joining the program. For details about the requirements and benefits, see https://github.com/volcano-sh/community/blob/master/community-building-program.md. - + ## Application to the program - Scan the QR code or click to read the full text and fill in the application form. -
{{
}}
+ ![](/img/Volcano community co-construction program/co-construction-2.jpg) - The result will be sent by email. Please wait. - **If you have any questions, please contact the Volcano community at wang.platform@gmail.com.** + diff --git a/content/en/blog/aiqiyi-en.md b/blog/aiqiyi-en.md similarity index 90% rename from content/en/blog/aiqiyi-en.md rename to blog/aiqiyi-en.md index 6e3916ff..2a451754 100644 --- a/content/en/blog/aiqiyi-en.md +++ b/blog/aiqiyi-en.md @@ -1,28 +1,24 @@ -+++ -title = "iQIYI:Volcano-based Cloud Native Migration Practices" -description = "Volcano use case in deep learning and service migration" -subtitle = "" - -date = 2020-09-30 -lastmod = 2021-08-25 -datemonth = "Aug" -dateyear = "2021" -dateday = 25 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Hao Li/Senior R&D Engineer in iQIYI"] - -tags = ["Practice"] -summary = "Volcano use case in deep learning and service migration" - -# Add menu entry to sidebar. -linktitle = "iQIYI:Volcano-based Cloud Native Migration Practices" -[menu.posts] -parent = "tutorials" -weight = 9 -+++ +--- +title: iQIYI:Volcano-based Cloud Native Migration Practices +description: Volcano use case in deep learning and service migration +subtitle: '' +date: 2020-09-30 +lastmod: '2021-08-25' +datemonth: Aug +dateyear: '2021' +dateday: '25' +draft: false +toc: true +type: posts +authors: +- Hao Li/Senior R&D Engineer in iQIYI +tags: +- Practice +summary: Volcano use case in deep learning and service migration +linktitle: iQIYI:Volcano-based Cloud Native Migration Practices +parent: tutorials +weight: '9' +--- >This article was firstly released at `Container Cube` on September 30th, 2020, refer to[揭秘爱奇艺深度学习平台云原生迁移实践](https://mp.weixin.qq.com/s/YtP-ZURRBr5-ba1eWfKS2A) @@ -38,13 +34,13 @@ The platform can be used in advertising, search, recommendation, NLP, and other -{{
}} +![](/img/aiqiyi-en1.png) __One-stop Platform Service__ Four small platforms are used to provide the service. The first is the data preprocessing platform. It analyzes the training data in a visualized manner, helps users adjust parameters, and detects abnormal data in a timely manner. -{{
}} +![](/img/aiqiyi-en2.png) The second is the training code compilation platform. You can use the RunOnce or notebook training to obtain an environment that is the same as the training environment. You then can compile the training code, and commit the code to GitLab. @@ -65,7 +61,7 @@ __Training Platform Architecture Before Volcano Is Used__ The following figure shows the training platform architecture before Volcano is used. -{{
}} +![](/img/aiqiyi-4.png) The process is as follows: @@ -105,7 +101,7 @@ Queues can be used to manage quotas. PodGroup is a group of pods and can be used for advanced upper-layer scheduling. -{{
}} +![](/img/aiqiyi-5.png) So far: @@ -126,7 +122,7 @@ __Gang Scheduling__ Gang scheduled pods run simultaneously or none of them run. This is important for AI training, especially distributed training in most scenarios. The feature of distributed training is that a large number of pods, for example, 40 or 50 pods, are started at a time. If some pods of a task are scheduled and some pods are not scheduled, the task cannot run properly. This will also cause resource waste, or even deadlocks. -{{
}} +![](/img/aiqiyi-en6.png) For example, there are only four GPUs in a resource pool and tasks A and B. Each task has four pods, and each pod requires one GPU. When tasks A and B are created at the same time, without gang scheduling, each task may obtain only two GPUs. In this case, neither of the tasks can be completed, resulting in a deadlock. Unless resources are added to the pool, the deadlock cannot be resolved. @@ -138,13 +134,13 @@ Volcano schedules jobs in the unit of PodGroup to implement gang scheduling, avo Take TensorFlow distributed training as an example. It has the following roles: Parameter Server (PS), master, and worker. PS is used to store parameters. Master and worker are used to calculate gradients. In each iteration, master and worker obtain parameters from PS and update the calculated gradients to PS. PS aggregates the gradients returned from master and worker, updates the parameters, and broadcasts the updated parameters to master and worker. -{{
}} +![](/img/aiqiyi-7.png) Let's focus on one of its network structures. If master or worker needs to communicate with PS, problems will occur. When creating a pod, a user may not know the IP address of the pod. Multiple pods created in a deployment may not know the IP address or domain name of each other. Without Volcano, solutions to these problems are complicated. Each role must know the IP address or domain name of the other roles, the role it plays, and the number of indexes. A TF_CONFIG configuration file is required to include the IP addresses or domain names of master, worker, and PS. These are difficult to implement in Kubernetes. However, with Volcano, the solutions become simple. -{{
}} +![](/img/aiqiyi-8.png) Volcano can easily build TF_CONFIG through file injection to support TensorFlow distributed training. Volcano injects a folder (etc/volcano) to multiple pods under a vcjob. The folder includes all domain names of master, volcano, and PS. In this way, each pod knows the peers in the entire cluster, and the TensorFlow distributed training can be performed. @@ -155,7 +151,7 @@ Currently, TensorFlow provides some high-level APIs, such as TF estimator. The s Volcano supports Horovod, which is similar to TensorFlow. They are both used for distributed training but differ in the way of updating parameters. -{{
}} +![](/img/aiqiyi-9.png) Horovod uses the ring allreduce method to update parameters, and what does that mean for us when we want to build a basic environment for upper-layer applications to use? What does the ring allreduce architecture require? @@ -167,7 +163,7 @@ Volcano uses queues (CRD objects) to schedule jobs. Let's assume that we have tw The teams in our platform are similar to the Volcano queues. How? Each team has a quota, and quotas are independent between teams. When the resource usage reaches the quota of a team, the jobs in the team have to wait in queue. When resources are available, the queued jobs will be executed based on the priority, which means the jobs with a higher priority will be run first. Considering this similarity, the interconnection between Volcano and iQIYI’s platform can be fairly easy. -{{
}} +![](/img/aiqiyi-en10.png) @@ -175,7 +171,7 @@ The teams in our platform are similar to the Volcano queues. How? Each team has iQIYI has added the volcano_plugin, which encapsulates the RESTful APIs of vcjob, queue, and PodGroup. It converts the gRPC requests into YAML configurations that comply with the Kubernetes API specifications, and calls the Kubernetes API to create containers. -{{
}} +![](/img/aiqiyi-11.png) Jarvis Core determines which backend to use based on the passed cluster information. @@ -183,13 +179,13 @@ Jarvis Core determines which backend to use based on the passed cluster informat Issue 1 -Symptom:During Volcano upgrade, the image in https://github.com/volcano_x0002_sh/volcano/blob/master/installer/volcano-development.yaml was directly modified, and kubectl apply -f was executed. The existing queues and vcjobs all disappeared. +Symptom:During Volcano upgrade, the image in https://github.com/volcano_x0002_sh/volcano/blob/master/installer/volcano-development.yaml was directly modified, and kubectl apply -f `yaml file` was executed. The existing queues and vcjobs all disappeared. Cause:volcano-admission-init in the YAML file was executed repeatedly. As a result, Volcano was reset. Solution: Upgrade only the necessary components. -{{
}} +![](/img/aiqiyi-12.png) Issue 2 @@ -197,7 +193,7 @@ Symptom: When list_and_watch was used to monitor vcjob status, the watch connect Cause: The default http timeout for CRD objects in Kubernetes is time.Duration(float64(minRequestTimeout) * (rand.Float64() + 1.0)), where miniRequestTimeout is set to 1 minute. You can specify timeoutSecond on the client to avoid this issue. -{{
}} +![](/img/aiqiyi-13.png) Issue 3 diff --git a/blog/authors.yml b/blog/authors.yml new file mode 100644 index 00000000..04ccd84d --- /dev/null +++ b/blog/authors.yml @@ -0,0 +1,53 @@ +volcano: + name: Volcano + url: https://volcano.sh + image_url: /img/volcano_logo.png +Volcano: + name: Volcano + url: https://volcano.sh + image_url: /img/volcano_logo.png +"Adam Korczynski": + name: Adam Korczynski +"Hao Li/Senior R&D Engineer in iQIYI": + name: Hao Li + title: Senior R&D Engineer in iQIYI +"Zhe Jin/Senior Engineer in HUAWEI CLOUD": + name: Zhe Jin + title: Senior Engineer in HUAWEI CLOUD +"Junmao Li/System Architect in Pengcheng Laboratory": + name: Junmao Li + title: System Architect in Pengcheng Laboratory +Thor-wl: + name: Thor-wl + url: https://github.com/Thor-wl +"Yunzhi Cheng/Ruitian Tech-Leader": + name: Yunzhi Cheng + title: Ruitian Tech-Leader +"Yunzhi Cheng/Tech-Leader in Ruitian": + name: Yunzhi Cheng + title: Tech-Leader in Ruitian +"Jian Zhu/Senior Architect in Leinao OS": + name: Jian Zhu + title: Senior Architect in Leinao OS +"Yi Guo/Tech-Leader": + name: Yi Guo + title: Tech-Leader +"Zhengyu Xu/R&D in Ruitian": + name: Zhengyu Xu + title: R&D in Ruitian +"Ada Logics": + name: Ada Logics + url: https://adalogics.com +"Xavier Chang": + name: Xavier Chang +"Huawei and Volcano maintainer": + name: Huawei and Volcano maintainer +"Volcano Engine Team": + name: Volcano Engine Team + url: https://volcano.sh +"PaddlePaddle Team": + name: PaddlePaddle Team + url: https://www.paddlepaddle.org.cn/ +"Volcano Team": + name: Volcano Team + url: https://volcano.sh diff --git a/content/en/blog/how-volcano-boosts-distributed-training-and-inference-performance.md b/blog/how-volcano-boosts-distributed-training-and-inference-performance.md similarity index 82% rename from content/en/blog/how-volcano-boosts-distributed-training-and-inference-performance.md rename to blog/how-volcano-boosts-distributed-training-and-inference-performance.md index 77699dcf..322499b2 100644 --- a/content/en/blog/how-volcano-boosts-distributed-training-and-inference-performance.md +++ b/blog/how-volcano-boosts-distributed-training-and-inference-performance.md @@ -1,28 +1,24 @@ -+++ -title = "How volcano boosts distributed training and inference performance" -description = "Join Volcano at KubeCon + CloudNativeCon Europe, 1-4 April 2025 in London!" -subtitle = "" - -date = 2025-04-01 -lastmod = 2025-04-01 -datemonth = "Apr" -dateyear = "2025" -dateday = 01 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["News"] -summary = "Join Volcano at KubeCon + CloudNativeCon Europe, 1-4 April 2025 in London!" - -# Add menu entry to sidebar. -linktitle = "How volcano boosts distributed training and inference performanc" -[menu.posts] -parent = "news" -weight = 6 -+++ +--- +title: How volcano boosts distributed training and inference performance +description: Join Volcano at KubeCon + CloudNativeCon Europe, 1-4 April 2025 in London! +subtitle: '' +date: 2025-04-01 +lastmod: '2025-04-01' +datemonth: Apr +dateyear: '2025' +dateday: '01' +draft: false +toc: true +type: posts +authors: +- volcano +tags: +- News +summary: Join Volcano at KubeCon + CloudNativeCon Europe, 1-4 April 2025 in London! +linktitle: How volcano boosts distributed training and inference performanc +parent: news +weight: '6' +--- ## The Growing Demand for LLM Workloads and Associated Challenges @@ -33,8 +29,8 @@ The increasing adoption of large language models (LLMs) has led to heightened de In LLM training, model parallelism distributes workloads across multiple nodes, requiring frequent data exchanges. Network communication can become a bottleneck, particularly in heterogeneous environments with InfiniBand (IB), RoCE, or NVSwitch configurations. Communication efficiency depends on network topology—fewer switches between nodes typically result in lower latency and higher throughput. One approach to mitigating this challenge is Network Topology-Aware Scheduling, which optimizes workload placement to minimize cross-switch communication. A key component of this strategy is the HyperNode, an abstraction for representing network topology via Custom Resource Definitions (CRDs). Unlike label-based methods, HyperNode provides a hierarchical structure that reflects actual network layouts, improving management and optimization. Nodes within the same HyperNode communicate more efficiently than those spanning multiple layers. -
{{
}} -
+ ![](/img/network-topology/hypernode-example.png) + Topology constraints can also be specified for jobs through the networkTopology field, with options for strict (Hard Mode) or flexible (Soft Mode) enforcement. This granular control helps ensure workloads are deployed in optimal network environments, reducing latency and improving throughput. @@ -43,8 +39,8 @@ Topology constraints can also be specified for jobs through the networkTopology As AI workloads expand, single Kubernetes clusters may no longer suffice for large-scale training and inference. While multiple clusters can address this limitation, managing them efficiently presents challenges. The Volcano Global subproject extends scheduling capabilities to multi-cluster environments, integrating with Karmada to enable cross-cluster scheduling for distributed workloads. Features such as Queue Priority Scheduling, Job Priority Scheduling, and Multi-Tenant Fair Scheduling help optimize resource allocation and ensure equitable access across tenants. This approach simplifies multi-cluster management while supporting scalable AI workloads. -
{{
}} -
+ ![](/img/multi-cluster/volcano_global_design.svg) + ## Improving Stability with Fine-Grained Fault Recovery diff --git a/content/en/blog/hpc-en.md b/blog/hpc-en.md similarity index 88% rename from content/en/blog/hpc-en.md rename to blog/hpc-en.md index b5995658..1f2cbebc 100644 --- a/content/en/blog/hpc-en.md +++ b/blog/hpc-en.md @@ -1,28 +1,29 @@ -+++ -title = "HPC on Volcano: How Containers Support HPC Applications in the Meteorological Industry" -description = "This article uses a traditional HPC application, the Weather Research and Forecasting (WRF) model, as an example to describe how Volcano works for HPC applications." -subtitle = "" - -date = 2020-10-27 -lastmod = 2021-08-24 -datemonth = "Aug" -dateyear = "2021" -dateday = 24 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Zhe Jin/Senior Engineer in HUAWEI CLOUD"] - -tags = ["Practice"] -summary = "This article uses a traditional HPC application, the Weather Research and Forecasting (WRF) model, as an example to describe how Volcano works for HPC applications." - -# Add menu entry to sidebar. -linktitle = "HPC on Volcano: How Containers Support HPC Applications in the Meteorological Industry" -[menu.posts] -parent = "tutorials" -weight = 7 -+++ +--- +title: 'HPC on Volcano: How Containers Support HPC Applications in the Meteorological + Industry' +description: This article uses a traditional HPC application, the Weather Research + and Forecasting (WRF) model, as an example to describe how Volcano works for HPC + applications. +subtitle: '' +date: 2020-10-27 +lastmod: '2021-08-24' +datemonth: Aug +dateyear: '2021' +dateday: '24' +draft: false +toc: true +type: posts +authors: +- Zhe Jin/Senior Engineer in HUAWEI CLOUD +tags: +- Practice +summary: This article uses a traditional HPC application, the Weather Research and + Forecasting (WRF) model, as an example to describe how Volcano works for HPC applications. +linktitle: 'HPC on Volcano: How Containers Support HPC Applications in the Meteorological + Industry' +parent: tutorials +weight: '7' +--- >This article was firstly released at `Container Cube` on October 27th, 2020, refer to [HPC on Volcano：容器在气象行业HPC高性能计算场景的应用](https://mp.weixin.qq.com/s/wLIoJeUSey9tzOCV6GZRig) @@ -37,7 +38,7 @@ HPC and HPCC are two common terms in the area of computing jobs. HPCC is short f An HPCC consists of three parts: -{{
}} +![](/img/hpc-1.png) - Portable Batch System (PBS): A resource manager that manages all node resources in a cluster. Other common resource management systems include Slurm and Platform Load Sharing Facility (or simply LSF). @@ -52,7 +53,7 @@ PBS and Maui are imperceptible to users. Users only need to submit jobs in the m The following uses __mpirun -np 4 ./mpi_hello_world__ as an example to illustrate how an MPI job runs. -{{
}} +![](/img/hpc-2.png) - Invoke Open MPI or other MPI libraries to compile the source code. In this example, Hello World! is printed. @@ -69,7 +70,7 @@ The Weather Research and Forecasting (WRF) model is a common HPC application. WR WRF consists of multiple modules with different processing flows. The following illustrates a WRF process. -{{
}} +![](/img/hpc-en3.png) As shown in the figure above, this WRF process has four parts: - External data sources @@ -112,7 +113,7 @@ As the core module of the WRF model, WRF performs simulation and prediction base real.exe and wrf.exe can run as MPI parallel jobs to improve the computing speed. -{{
}} +![](/img/hpc-en4.png) As shown in the preceding figure, wrfinput_d0X and wrfbdy_d0X are the calculation results generated by real.exe. wrf.exe performs meteorological simulation based on these results to generate the final result wrfout_dxx_yyyy-mm-dd_hh:mm:ss, which is verified and displayed by the post-processing system. @@ -120,20 +121,20 @@ __Post-Processing System__ The post-processing system verifies and displays the calculation results generated by WRF. It consists of various third-party images and verification tools. The following figure shows the simulation and prediction results of the relative humidity in each area in CONUS 2.5km case. -{{
}} +![](/img/hpc-en5.png) CONUS 2.5km refers to the 2.5 km resolution case covering the Continental U.S. (CONUS) domain. (In this case, the entire domain is divided into multiple cubes of 2.5 km x 2.5 km x 2.5 km. The meteorological information in each cube is considered consistent.) ## HPC on Volcano -{{
}} +![](/img/hpc-en6.png) As mentioned above, an HPCC consists of a resource manager, scheduler, and MPI parallel computing library. In the container context, Kubernetes functions as the resource manager and Volcano functions as the scheduler. To run HPC applications in the Kubernetes+Volcano environment is to run HPC jobs in containers, as shown in the following figure. -{{
}} +![](/img/hpc-en7.png) Two types of containers are involved: master and worker. The master container starts the mpirun and mpiexec commands, and the worker containers run computing jobs. @@ -247,16 +248,16 @@ spec: -{{
}} +![](/img/hpc-8.png) The job is executed. -{{
}} +![](/img/hpc-9.png) 3. Check the execution result of the master pod. -{{
}} +![](/img/hpc-10.png) The preceding execution result shows that Volcano clears only the worker pods and retains the master pod after the job completes. In this way, you can run the kubectl command to obtain the execution result. diff --git a/content/en/blog/iflytek_case_study.md b/blog/iflytek_case_study.md similarity index 78% rename from content/en/blog/iflytek_case_study.md rename to blog/iflytek_case_study.md index 8662a223..173fa15a 100644 --- a/content/en/blog/iflytek_case_study.md +++ b/blog/iflytek_case_study.md @@ -1,31 +1,33 @@ -+++ -title = "iFlytek Enhances AI Infrastructure with Volcano, Wins CNCF End-User Case Study Award" -description = "iFlytek was awarded for its innovative use of Volcano in the CNCF End-User Case Study Competition and shared its success in large-scale AI model training at KubeCon + CloudNativeCon China 2025." -subtitle = "" - -date = 2025-06-13 -lastmod = 2025-06-13 -datemonth = "June" -dateyear = "2025" -dateday = 13 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["News"] -summary = "iFlytek was awarded for its innovative use of Volcano in the CNCF End-User Case Study Competition and shared its success in large-scale AI model training at KubeCon + CloudNativeCon China 2025." - -# Add menu entry to sidebar. -linktitle = "iFlytek Enhances AI Infrastructure with Volcano, Wins CNCF End-User Case Study Award" -[menu.posts] -parent = "news" -weight = 6 -+++ - -
{{
}} -
+--- +title: iFlytek Enhances AI Infrastructure with Volcano, Wins CNCF End-User Case Study + Award +description: iFlytek was awarded for its innovative use of Volcano in the CNCF End-User + Case Study Competition and shared its success in large-scale AI model training at + KubeCon + CloudNativeCon China 2025. +subtitle: '' +date: 2025-06-13 +lastmod: '2025-06-13' +datemonth: June +dateyear: '2025' +dateday: '13' +draft: false +toc: true +type: posts +authors: +- volcano +tags: +- News +summary: iFlytek was awarded for its innovative use of Volcano in the CNCF End-User + Case Study Competition and shared its success in large-scale AI model training at + KubeCon + CloudNativeCon China 2025. +linktitle: iFlytek Enhances AI Infrastructure with Volcano, Wins CNCF End-User Case + Study Award +parent: news +weight: '6' +--- + + ![](/img/kubecon/iflytek.jpeg) + [HONG KONG, CHINA — June 10, 2025] — The Cloud Native Computing Foundation (CNCF) today announced that iFlytek has won the CNCF End-User Case Study Competition. The CNCF, which is committed to building a sustainable ecosystem for cloud native software, recognized iFlytek for its innovative use of Volcano. The company shared its success in large-scale AI model training at the KubeCon + CloudNativeCon China conference, held in Hong Kong from June 10-11. diff --git a/content/en/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md b/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md similarity index 86% rename from content/en/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md rename to blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md index 45c45dad..3c59a1aa 100644 --- a/content/en/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md +++ b/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md @@ -1,28 +1,26 @@ -+++ -title = "Introducing Kthena: Redefining LLM Inference for the Cloud-Native Era" -description = "Kthena is a Kubernetes-native, high-performance LLM inference routing and orchestration system. It improves GPU/NPU utilization and reduces latency with topology-aware scheduling, KV Cache-aware routing, and Prefill-Decode disaggregation." -subtitle = "" - -date = 2026-01-06 -lastmod = 2026-01-06 -datemonth = "Jan" -dateyear = "2026" -dateday = 6 - -draft = false -toc = true -type = "posts" -authors = ["volcano"] - -tags = ["Practice"] -summary = "Kthena is a Kubernetes-native LLM inference routing and orchestration layer for production-scale serving." - -linktitle = "Introducing Kthena: Redefining LLM Inference for the Cloud-Native Era" -[menu.posts] -parent = "tutorials" -weight = 5 -+++ - +--- +title: "Introducing Kthena: Redefining LLM Inference for the Cloud-Native Era" +description: "Volcano Kthena: Redefining LLM Inference for the Cloud-Native Era" +subtitle: "" +date: 2024-08-21 +lastmod: 2026-01-06 +datemonth: "Jan" +dateyear: "2026" +dateday: 6 +draft: false +toc: true +type: "posts" +authors: + - "Volcano Engine Team" +tags: + - Announcement +summary: "Kthena is a Kubernetes-native LLM inference routing and orchestration layer for production-scale serving." +linktitle: "Introducing Kthena: Redefining LLM Inference for the Cloud-Native Era" +menu: + posts: + parent: "tutorials" + weight: 5 +--- Today, the [Volcano](https://volcano.sh/) community is proud to announce the launch of [Kthena](https://github.com/volcano-sh/kthena), a new sub-project designed for global developers and MLOps engineers. Kthena is a cloud-native, high-performance system for LLM inference routing, orchestration, and scheduling, tailored specifically for Kubernetes. Engineered to address the complexity of serving LLMs at production scale, Kthena delivers granular control and enhanced flexibility. Through features like topology-aware scheduling, KV Cache-aware routing, and Prefill-Decode (PD) disaggregation, it significantly improves GPU/NPU utilization and throughput while minimizing latency. @@ -42,8 +40,8 @@ While Large Language Models (LLMs) are reshaping industries, deploying them effi Kthena was built to conquer these challenges. Rather than replacing existing inference engines (like vLLM or SGLang), Kthena acts as an intelligent orchestration layer atop them, deeply integrated into Kubernetes. -
{{
}} -
+ ![](/img/kthena/kthena-arch.svg) + Kthena consists of two core components: @@ -57,8 +55,8 @@ Kthena consists of two core components: ### 1. Production-Grade Inference Orchestration (ModelServing) -
{{
}} -
+ ![](/img/kthena/model-serving.svg) + Kthena introduces a Hierarchical Workload Architecture (ModelServing -> ServingGroup -> Role). @@ -121,13 +119,13 @@ This release marks not only a significant milestone in Volcano's technical evolu Through Kthena, we look forward to collaborating with global developers to build an open, thriving ecosystem that lays a robust foundation for the intelligent transformation of industries worldwide." -
—— Xiaobo Qi, Director of General Computing Services, Huawei Cloud
+—— Xiaobo Qi, Director of General Computing Services, Huawei Cloud "Kthena further solidifies Volcano's leadership in intelligent workload scheduling. By leveraging Volcano's unified scheduling and resource pooling capabilities, our platform addresses diverse compute requirements—spanning general-purpose computing, AI training, and inference—within a single, unified framework. This enables dynamic resource allocation across different scenarios, effectively eliminating resource silos. Looking ahead, we are excited to combine Kthena with Volcano’s elastic scaling and Volcano Global’s cross-cluster scheduling to drive resource utilization to new heights." -
—— Lei Yang, PaaS R&D Director, China Telecom AI
+—— Lei Yang, PaaS R&D Director, China Telecom AI "Since its inception, Volcano has evolved in lockstep with the community to address diverse AI scenarios, establishing a comprehensive ecosystem for AI batch processing. @@ -135,13 +133,13 @@ The launch of Kthena marks a major milestone, extending Volcano's capabilities i By leveraging the existing Kubernetes and Volcano ecosystems, teams can achieve smarter scheduling decisions and higher compute efficiency at a lower cost. For DaoCloud, Kthena not only solves tangible inference challenges but also embodies the future of Cloud Native AI—an open, intelligent ecosystem worthy of our long-term investment and deep engagement." -
—— Paco Xu, Open Source Team Lead at DaoCloud, Member of Kubernetes Steering Committee
+—— Paco Xu, Open Source Team Lead at DaoCloud, Member of Kubernetes Steering Committee "Deploying and managing self-hosted LLM inference services at production scale is a complex systems engineering challenge. It encompasses the entire lifecycle—deployment, operations, elasticity, and recovery—alongside critical requirements like GPU stability, scheduling efficiency, and AI observability. Kthena is engineered specifically to address these complexities. During Kthena’s planning phase, the Xiaohongshu Cloud Native team engaged deeply with contributors to co-design various intelligent traffic scheduling strategies. Moving forward, we will continue our collaboration on the AI Gateway front. By leveraging Xiaohongshu’s production insights, we aim to provide the community with production-ready capabilities, including granular traffic scheduling, model API management, and MCP protocol support." -
—— Kong Gu (Huachang Chen), Cloud Native Business Gateway Lead, Xiaohongshu
+—— Kong Gu (Huachang Chen), Cloud Native Business Gateway Lead, Xiaohongshu "After an in-depth evaluation of Kthena, China Unicom Cloud is impressed by its forward-looking design. We are particularly excited about its joint scheduling capabilities with Volcano. @@ -149,7 +147,7 @@ Features like topology awareness and Gang Scheduling directly address the critic We believe Kthena’s superior low latency, high throughput, and intelligent routing will provide the open-source community with a truly production-ready solution, empowering developers to build and manage cloud-native AI applications with greater efficiency." -
—— Zhaoxu Lu, Team Lead, Intelligent Computing Center, China Unicom Cloud
+—— Zhaoxu Lu, Team Lead, Intelligent Computing Center, China Unicom Cloud "Openness and collaboration fuel innovation. Within the CNCF ecosystem, we are dedicated to driving infrastructure towards an 'AI Native' future. @@ -157,7 +155,7 @@ By launching the Kthena sub-project, the Volcano community applies its proven ex We invite developers worldwide to join us in refining this critical infrastructure and accelerating the AI Native era." -
—— Kevin Wang, Volcano Maintainer, CNCF TOC Vice Chair
+—— Kevin Wang, Volcano Maintainer, CNCF TOC Vice Chair ## Start Exploring Kthena Today diff --git a/content/en/blog/kube-batch-customers.md b/blog/kube-batch-customers.md similarity index 58% rename from content/en/blog/kube-batch-customers.md rename to blog/kube-batch-customers.md index a0f576c3..80a591c4 100644 --- a/content/en/blog/kube-batch-customers.md +++ b/blog/kube-batch-customers.md @@ -1,28 +1,24 @@ -+++ -title = "Customers for Kube-Batch" -description = "A Batch Scheduler for Kubernetes" -subtitle ="" - -date = 2019-01-28 -lastmod = 2019-01-29 -datemonth = "Jan" -dateyear = "2019" -dateday = 29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Volcano"] - -tags = ["Tutorials"] -summary = "A wide range of customer base for Kube-Batch" - -# Add menu entry to sidebar. -linktitle = "Customers of Kube-Batch" -[menu.posts] - parent = "customers" - weight = 1 -+++ +--- +title: Customers for Kube-Batch +description: A Batch Scheduler for Kubernetes +subtitle: '' +date: 2019-01-28 +lastmod: '2019-01-29' +datemonth: Jan +dateyear: '2019' +dateday: '29' +draft: false +toc: true +type: posts +authors: +- Volcano +tags: +- Tutorials +summary: A wide range of customer base for Kube-Batch +linktitle: Customers of Kube-Batch +parent: customers +weight: '1' +--- ## Who is Using kube-batch? | Organization | Contact (Github UserName) | Environment | Description| diff --git a/content/en/blog/kube-batch-startup.md b/blog/kube-batch-startup.md similarity index 90% rename from content/en/blog/kube-batch-startup.md rename to blog/kube-batch-startup.md index 73c67e24..c187bd7c 100644 --- a/content/en/blog/kube-batch-startup.md +++ b/blog/kube-batch-startup.md @@ -1,28 +1,24 @@ -+++ -title = "Bringing Up Kube-Batch" -description = "A Batch Scheduler for Kubernetes" -subtitle ="" - -date = 2019-01-28 -lastmod = 2019-01-29 -datemonth = "Jan" -dateyear = "2019" -dateday = 29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Volcano"] - -tags = ["Tutorials"] -summary = "Bring up the Batch Scheduler for scheduling batch workloads" - -# Add menu entry to sidebar. -linktitle = "Bringing Up Kube-Batch" -[menu.posts] - parent = "tutorials" - weight = 2 -+++ +--- +title: Bringing Up Kube-Batch +description: A Batch Scheduler for Kubernetes +subtitle: '' +date: 2019-01-28 +lastmod: '2019-01-29' +datemonth: Jan +dateyear: '2019' +dateday: '29' +draft: false +toc: true +type: posts +authors: +- Volcano +tags: +- Tutorials +summary: Bring up the Batch Scheduler for scheduling batch workloads +linktitle: Bringing Up Kube-Batch +parent: tutorials +weight: '2' +--- # Tutorial of kube-batch diff --git a/content/en/blog/leinao-en.md b/blog/leinao-en.md similarity index 79% rename from content/en/blog/leinao-en.md rename to blog/leinao-en.md index 032e5c51..46dbf780 100644 --- a/content/en/blog/leinao-en.md +++ b/blog/leinao-en.md @@ -1,28 +1,26 @@ -+++ -title = "Integrating Volcano into the Leinao Cloud OS" -description = "Deep introduction about the challenges and solutions faced by Volcano integration with Leinao Cloud OS" -subtitle = "" - -date = 2020-12-24 -lastmod = 2021-08-23 -datemonth = "Dec" -dateyear = "2020" -dateday = 23 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Jian Zhu/Senior Architect in Leinao OS"] - -tags = ["Practice"] -summary = "Deep introduction about the challenges and solutions faced by Volcano integration with Leinao Cloud OS" - -# Add menu entry to sidebar. -linktitle = "Integrating Volcano into the Leinao Cloud OS" -[menu.posts] -parent = "tutorials" -weight = 3 -+++ +--- +title: Integrating Volcano into the Leinao Cloud OS +description: Deep introduction about the challenges and solutions faced by Volcano + integration with Leinao Cloud OS +subtitle: '' +date: 2020-12-24 +lastmod: '2021-08-23' +datemonth: Dec +dateyear: '2020' +dateday: '23' +draft: false +toc: true +type: posts +authors: +- Jian Zhu/Senior Architect in Leinao OS +tags: +- Practice +summary: Deep introduction about the challenges and solutions faced by Volcano integration + with Leinao Cloud OS +linktitle: Integrating Volcano into the Leinao Cloud OS +parent: tutorials +weight: '3' +--- >This article was firstly released at `Container Cube` on December 24th, 2020, refer to [Volcano在中科类脑云OS中的落地实践](https://mp.weixin.qq.com/s/HS6RzzqztBJsHQX7P5T5ww) @@ -43,7 +41,7 @@ The Leinao cloud AI platform includes an AI development platform, public service ## The architecture of Leinao cloud OS -{{
}} +![](/img/leinao-en1.png) @@ -64,22 +62,22 @@ More importantly, Volcano allows you to configure retry policies for distributed Volcano provides enhanced job APIs. -{{
}} +![](/img/leinao-en2.png) Volcano improves many aspects of default-scheduler. -{{
}} +![](/img/leinao-en3.png) Default-scheduler and Volcano work differently in a couple of other ways as well. -{{
}} +![](/img/leinao-en4.png) We encountered some obstacles when we connected the OS to Volcano. For example, OS development was already complete when we tried to connect it to Volcano, and connecting them directly required a huge change to the OS's computing and application layers. Moreover, Volcano did not support debugging jobs and tool sets yet. That was when we decided to introduce job-server, for API adaptation and integrated development of debugging tool sets. -{{
}} +![](/img/leinao-en5.png) Another problem we faced was how to deal with task monitoring. Upper-layer services need detailed information on current and recent task status, and historical records, but Volcano only supports job monitoring. Should we customize Volcano or further develop the OS to support task monitoring? If we customize Volcano, it would get complicated later on, when we want to upgrade Volcano. The Volcano community iterates in a fast speed. We did not want to miss its latest features provided with every iteration. Therefore, we went with the latter choice, that is, to further develop the OS. @@ -88,7 +86,7 @@ Another problem we faced was how to deal with task monitoring. Upper-layer servi The following figure shows the monitoring mechanism. It uses an API server to watch jobs and pods. -{{
}} +![](/img/leinao-en6.png) @@ -104,9 +102,9 @@ __Scenario requirements:__ The workflow is as follows: -{{
}} +![](/img/leinao-en7.png) -{{
}} +![](/img/leinao-en8.png) @@ -119,13 +117,13 @@ __Scenario requirements:__ **When the job finishes, Volcano automatically deletes it.** -{{
}} +![](/img/leinao-en9.png) **Related resources (pods, services, and ingresses) are deleted.** -{{
}} +![](/img/leinao-en10.png) -{{
}} +![](/img/leinao-en11.png) @@ -134,9 +132,9 @@ __Scenario requirements:__ In OS 1.0, retry policies are set simply based on the number of retries. Volcano allows you to set retry policies based on events. It is much more flexible and better suits our scenarios. We gave up on our original solution and adopted the retry mechanism of Volcano. -{{
}} +![](/img/leinao-en12.png) -{{
}} +![](/img/leinao-en13.png) - Policies defined in taskRole have a higher priority than the retry policies defined in jobs. A retry policy consists of an event and action. @@ -149,7 +147,7 @@ In OS 1.0, retry policies are set simply based on the number of retries. Volcano When we were developing the OS to support task monitoring, we received great support from the Volcano community. For example, we once found that RestartTask became invalid. The problem was solved the same day it was reported to the community. Their response was very fast. -{{
}} +![](/img/leinao-en14.png) ## Next Up diff --git a/content/en/blog/Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios.md b/blog/meet-cloud-native-batch-computing-with-volcano-in-ai-big-data-scenarios.md similarity index 83% rename from content/en/blog/Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios.md rename to blog/meet-cloud-native-batch-computing-with-volcano-in-ai-big-data-scenarios.md index 1dbacf7c..f124a4af 100644 --- a/content/en/blog/Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios.md +++ b/blog/meet-cloud-native-batch-computing-with-volcano-in-ai-big-data-scenarios.md @@ -1,32 +1,19 @@ -+++ -title = "Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios" -description = "Join Volcano at KubeCon + CloudNativeCon Europe, 19-22 March in Paris!" -subtitle = "" - -date = 2024-03-08 -lastmod = 2024-03-08 -datemonth = "Mar" -dateyear = "2024" -dateday = 08 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["News"] -summary = "Join Volcano at KubeCon + CloudNativeCon Europe, 19-22 March in Paris!" - -# Add menu entry to sidebar. -linktitle = "Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios" -[menu.posts] -parent = "news" -weight = 6 -+++ +--- +title: "Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios" +description: "Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios" +subtitle: "" +date: 2021-08-11 +draft: false +authors: + - Volcano +tags: + - Practice +summary: "Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios" +--- Cloud native batch computing engine Volcano is designed for high-performance computing applications such as AI, big data, gene sequencing, and rendering, and supports mainstream general computing frameworks. More than 58,000 global developers joined us, among whom the in-house ones come from companies such as Huawei, AWS, Baidu, Tencent, JD, and Xiaohongshu. There are 3.7k+ Stars and 800+ Forks for the project. Volcano has been proven feasible for mass data computing and analytics, such as AI, big data, and gene sequencing. Supported frameworks include Spark, Flink, TensorFlow, PyTorch, Argo, MindSpore, Paddlepaddle, Kubeflow, MPI, Horovod, MXNet, KubeGene, and Ray. The ecosystem is thriving with more developers and use cases coming up. -{{
}} +![](/img/volcano_logo.png) As the industry-first cloud native batch computing project，Volcano was Open-sourced at KubeCon Shanghai in June 2019, it became an official CNCF project in April 2020. In April 2022, Volcano was promoted to a CNCF incubating project. By now, more than 600 global developers have committed code to the project. The community is seeing growing popularity among developers, partners, and users. ### Try new features in Volcano v1.8.2 @@ -58,7 +45,7 @@ For details about the requirements and benefits, see https://github.com/volcano- ### Join Volcano at KubeCon + CloudNativeCon Europe, 19-22 March in Paris! -
{{
}}
+ ![](/img/kubecon/2024-paris.png) Volcano will participate in several activities, including: - Speech Schedule diff --git a/content/en/blog/paddlepaddle-en.md b/blog/paddlepaddle-en.md similarity index 97% rename from content/en/blog/paddlepaddle-en.md rename to blog/paddlepaddle-en.md index b8572131..74861ed8 100644 --- a/content/en/blog/paddlepaddle-en.md +++ b/blog/paddlepaddle-en.md @@ -1,28 +1,14 @@ -+++ -title = "PaddlePaddle Distributed Training on Volcano" -description = "Best practice about PaddlePaddle distributed training on Volcano" -subtitle = "" - -date = 2019-11-06 -lastmod = 2021-08-23 -datemonth = "Dec" -dateyear = "2020" -dateday = 23 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["PaddlePaddle Team", "Volcano Team"] - -tags = ["Practice"] -summary = "Best practice about PaddlePaddle distributed training on Volcano" - -# Add menu entry to sidebar. -linktitle = "PaddlePaddle Distributed Training on Volcano" -[menu.posts] -parent = "tutorials" -weight = 4 -+++ +--- +title: PaddlePaddle Distributed Training on Volcano +description: Best practice about PaddlePaddle distributed training on Volcano +subtitle: "" +date: 2019-11-06 +authors: + - PaddlePaddle Team + - Volcano Team +tags: + - Practice +--- >This article was firstly released at `Container Cube` on November 6th, 2019, refer to [百度飞桨(PaddlePaddle)分布式训练在Volcano系统上的实践](https://mp.weixin.qq.com/s/SnUUEEy9OfNghzoel7FtUg) diff --git a/content/en/blog/pengcheng-en.md b/blog/pengcheng-en.md similarity index 91% rename from content/en/blog/pengcheng-en.md rename to blog/pengcheng-en.md index 3fa7b289..7d865e44 100644 --- a/content/en/blog/pengcheng-en.md +++ b/blog/pengcheng-en.md @@ -1,28 +1,15 @@ -+++ -title = "OpenI-Octopus: How to Avoid Resource Preemption in Kubernetes Clusters" -description = "Volcano use case in scientific research" -subtitle = "" - -date = 2021-06-01 -lastmod = 2021-08-26 -datemonth = "Aug" -dateyear = "2021" -dateday = 26 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Junmao Li/System Architect in Pengcheng Laboratory"] - -tags = ["Practice"] -summary = "Volcano use case in scientific research" - -# Add menu entry to sidebar. -linktitle = "OpenI-Octopus: How to Avoid Resource Preemption in Kubernetes Clusters" -[menu.posts] -parent = "tutorials" -weight = 10 -+++ +--- +title: "OpenI-Octopus: How to Avoid Resource Preemption in Kubernetes Clusters" +description: "Volcano use case in scientific research" +subtitle: "" +date: 2021-06-01 +draft: false +authors: + - "Junmao Li/System Architect in Pengcheng Laboratory" +tags: + - Practice +summary: "Volcano use case in scientific research" +--- >This article was firstly released at `Container Cube` on September 30th, 2020, refer to[鹏城实验室启智章鱼教你彻底摆脱Kubernetes集群资源抢占难题](https://mp.weixin.qq.com/s/h4T7KbAiQZTKepYcTcgdlA) @@ -44,7 +31,7 @@ OpenI-Octopus is a cluster management and resource scheduling system developed a - A microservice-based architecture is used. -{{
}} +![](/img/pengcheng-en1.png) The service architecture of OpenI-Octopus is illustrated above. The bottom layer is hardware. OpenI-Octopus supports various types of heterogeneous hardware, including CPUs, GPUs, NPUs, and FPGAs. Different hardware types are adapted so that the upper-layer Kubernetes services can identify and manage them. @@ -65,7 +52,7 @@ The top layer is the services provided by the system, such as data engine, model ## Business Scenarios and Challenges -{{
}} +![](/img/pengcheng-en2.png) OpenI-Octopus is built for research teams and laboratories. They develop and train models in fields such as transportation, healthcare, and finance, model training, and perform model inference. These models are used for vehicle tracking, medical image recognition, auxiliary diagnosis, financial quantization, and many other applications. Some deep learning algorithms are used in these courses, which require strong compute resources. @@ -117,14 +104,14 @@ OpenI-Octopus performed secondary development on Volcano and added some new capa The first capability is to collect statistics on resources and manage resource status. These resources include both cluster compute resources and resources such as jobs, tasks, and pods generated by Kubernetes after a user submits a job. -{{
}} +![](/img/pengcheng-en3.png) OpenI-Octopus manages to do so. It also allows users to customize the conditions and callback events of resource status transition and subscribe to the customized events and corresponding policies at the service logic layer. Assume that there is a training job that uses an ensemble learning algorithm. Generally, a distributed training manner is used. It has a combination module and several individual learners, all of which can be regarded as tasks. Each individual learner is trained using one type of algorithm, and the combination module combines the results of each individual learner to output the final result. Once the final result is obtained, the entire training job is complete. In this job-task implementation based on Kubernetes, user can create one or more pods for a task. If you want the entire job to exit as long as the combination module runs to completion, instead of waiting until all tasks are successfully executed, You can customize a job exit policy in the scheduler and use the policy at the service layer. Different scenarios may require different policies, and that's why secondary development is needed. -{{
}} +![](/img/pengcheng-en4.png) This flowchart shows job state information is transferred among OpenI-Octopus, Kubernetes, and Volcano. @@ -169,7 +156,7 @@ Requirements: - Delayed preemption -{{
}} +![](/img/pengcheng-en5.png) This flowchart shows how the delayed preemption plugin works. On the left is the running logic of the plugin in the scheduler. Kubernetes services lie in the middle, and the right part is the core OpenI-Octopus modules. @@ -179,7 +166,7 @@ Specifically, the plugin finds the jobs that need to be preempted in Volcano. Th The following chart shows the service logic. -{{
}} +![](/img/pengcheng-en6.png) 1) A Boolean attribute called Preempt is added to each job, indicating whether the job is a preempted job. @@ -199,7 +186,7 @@ The following chart shows the service logic. - If the state of a preempting or preempted job changes, the state of the other party changes accordingly. -{{
}} +![](/img/pengcheng-7.png) ## Benefits @@ -222,7 +209,7 @@ __Performance tuning__ - System stability is enhanced, cluster node resources are used in balance, and O&M workloads are reduced. -{{
}} +![](/img/pengcheng-8.png) - With 120+ nodes managed and 1100+ GPU cards in total, the GPU utilization can reach 90% or higher when the system is overloaded. diff --git a/content/en/blog/ruitian-en.md b/blog/ruitian-en.md similarity index 92% rename from content/en/blog/ruitian-en.md rename to blog/ruitian-en.md index 16c55496..1e4ef7f1 100644 --- a/content/en/blog/ruitian-en.md +++ b/blog/ruitian-en.md @@ -1,30 +1,18 @@ -+++ -title = "How Ruitian Used Volcano to Run Large-Scale Offline HPC Jobs" -description = "Deep introduction about the application practice cases of Volcano in the financial field" -subtitle = "" - -date = 2021-01-05 -lastmod = 2021-08-23 -datemonth = "Dec" -dateyear = "2020" -dateday = 23 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Yunzhi Cheng/Tech-Leader in Ruitian", "Zhengyu Xu/R&D in Ruitian"] - -tags = ["Practice"] -summary = "Deep introduction about the application practice cases of Volcano in the financial field" - -# Add menu entry to sidebar. -linktitle = "How Ruitian Used Volcano to Run Large-Scale Offline HPC Jobs" -[menu.posts] -parent = "tutorials" -weight = 5 -+++ - ->This article was firstly released at `Container Cube` on January 5th, 2021, refer to [基于Volcano的锐天离线高性能计算最佳实践](https://mp.weixin.qq.com/s/FDYExtj93lCrXmiFRozBPA) +--- +title: "How Ruitian Used Volcano to Run Large-Scale Offline HPC Jobs" +description: "Deep introduction about the application practice cases of Volcano in the financial field" +subtitle: "" +date: 2021-01-05 +draft: false +authors: + - "Yunzhi Cheng/Tech-Leader in Ruitian" + - "Zhengyu Xu/R&D in Ruitian" +tags: + - Practice +summary: "Deep introduction about the application practice cases of Volcano in the financial field" +--- + +>This article was firstly released at `Container Cube` on January 5th, 2021, refer to [基于Volcano的锐天离线高性能计算最佳实践](https://mp.weixin.qq.com/s/FDYExtj9lCrXmiFRozBPA) ## Why Volcano @@ -157,12 +145,12 @@ spec: - bash - -c - |- - python -u call_module_func.py --pickle-file /data/simulation/.pickle module.submodule magic_function + python -u call_module_func.py --pickle-file /data/simulation/`work-id`.pickle module.submodule magic_function ``` call_module_func.py is a boot script, which is mounted to the container using a ConfigMap. It is responsible for: -- Converting to the replica ID, which is obtained from the host name in the container. For example, if the host name is awesome-job-awesome-job-1, the replica ID is 1. +- Converting `work-id` to the replica ID, which is obtained from the host name in the container. For example, if the host name is awesome-job-awesome-job-1, the replica ID is 1. - Loading pickle parameters (mounted to the container using a PVC) and transferring them to the magic_function function in module.submodule. @@ -284,9 +272,9 @@ Currently, Volcano mainly monitors scheduling performance metrics, but these met With the additional metrics, Grafana can display monitoring information for the cluster, queue, and node resources as well as the job progress. In this way, you can track cluster resource usage in real time, which facilitates troubleshooting. For example, when a job state is Pending, you can view the monitoring information to check whether the queue or cluster resources are used up or if the remaining resources on each node are sufficient for a single task. -{{
}} +![](/img/ruitian2.png) -{{
}} +![](/img/ruitian3.png) diff --git a/content/en/blog/ruitian2-en.md b/blog/ruitian2-en.md similarity index 86% rename from content/en/blog/ruitian2-en.md rename to blog/ruitian2-en.md index e22d87b7..313e3fcd 100644 --- a/content/en/blog/ruitian2-en.md +++ b/blog/ruitian2-en.md @@ -1,28 +1,15 @@ -+++ -title = "Using Volcano in Large-Scale, Distributed Offline Computing" -description = "Volcano use case in the financial sector" -subtitle = "" - -date = 2021-06-15 -lastmod = 2021-08-25 -datemonth = "Aug" -dateyear = "2021" -dateday = 25 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Yunzhi Cheng/Ruitian Tech-Leader"] - -tags = ["Practice"] -summary = "Use case description: Ruitian Capital, an investment company, uses Volcano to build their offline computing platform." - -# Add menu entry to sidebar. -linktitle = "Using Volcano in Large-Scale, Distributed Offline Computing" -[menu.posts] -parent = "tutorials" -weight = 8 -+++ +--- +title: "Using Volcano in Large-Scale, Distributed Offline Computing" +description: "Volcano use case in the financial sector" +subtitle: "" +date: 2021-06-15 +draft: false +authors: + - "Yunzhi Cheng/Ruitian Tech-Leader" +tags: + - Practice +summary: "Ruitian Capital, an investment company, uses Volcano to build their offline computing platform." +--- >This article was firstly released at `Container Cube` on December 24th, 2020, refer to[锐天投资基于Volcano的大规模分布式离线计算平台的应用实践](https://mp.weixin.qq.com/s/dC4IDNG7FMGLigNJaj_Qug) @@ -78,7 +65,7 @@ Another option was kube-batch, a batch processing scheduler of the community. Ho -{{
}} +![](/img/ruitian2-1.png) __Why is Volcano__ @@ -99,7 +86,7 @@ Such a scheduling platform can satisfy Ruitian. __Service Architecture__ -{{
}} +![](/img/ruitian2-en2.png) - Ceph-based high-performance storage @@ -113,7 +100,7 @@ __Service Architecture__ __Multi-tenancy__ -{{
}} +![](/img/ruitian2-3.png) When a user submits a job, multi-tenancy can be a problem. For example, when a user adds a pod to a cluster, the cluster needs to know the running user and the UID. By default, the UID of a running user is that of the image builder, which means the UIDs of the pods submitted by all users can be the same. This is not allowed because the data obtained and generated by a user should not be accessible to other users. @@ -124,7 +111,7 @@ With PSPs, users can be isolated when accessing data, which is all stored in Cep __Workflows__ -{{
}} +![](/img/ruitian2-4.png) What comes next is basic workflows. The local configurations are rendered into a job YAML and then submitted. All dependency data of the user is synchronized to CephFS, and the pod is mounted with a PVC. Each user has the PVC permissions of their own directory in their own namespace. The permissions are managed and controlled through IBS. In this way, jobs are submitted to the cluster to run. @@ -135,23 +122,23 @@ In the basic submission framework, Ruitian provides libraries for users and is d With Jobctl, Kubernetes complexities can be shielded for users. In addition, command line submission and Python Lib integration are supported, and the most basic parallel execution by replicas and by day is provided. -{{
}} +![](/img/ruitian2-5.png) __OOM Auto Scale Up__ -{{
}} +![](/img/ruitian2-6.png) The first customization is to scale up resources of the entire job during OOM. Users may not be able to configure the exact memory required, and need to submit the job again for verification after the OOM. Therefore, Ruitian customized OOMKill Auto Scale-Up to modify the Volcano Controller to automatically scale up the resources requested by the OOMKill pods. After the scale-up, the jobs are automatically submitted again. The user will be informed upon the successful submission. This function guarantees reasonable memory requests without manual intervention, combining the Volcano policy event mechanism mentioned above. __MinSuccess__ -{{
}} +![](/img/ruitian2-7.png) - If the number of pods that run to completion reaches minAvailable, the job is complete. - Non-Gang jobs cannot be flexibly scheduled. -{{
}} +![](/img/ruitian2-8.png) - If the number of pods that run to completion reaches minSuccess, the job is complete. @@ -160,7 +147,7 @@ __MinSuccess__ __NodeZone__ -{{
}} +![](/img/ruitian2-9.png) - One Volcano instance manages all nodes. @@ -168,7 +155,7 @@ __NodeZone__ - Resources cannot be reserved for emergency. -{{
}} +![](/img/ruitian2-10.png) - Multiple Volcano instances manage multiple zones. @@ -176,17 +163,17 @@ __NodeZone__ __Volcano Namespace Quota__ -{{
}} +![](/img/ruitian2-11.png) The default Kubernetes quotas cannot satisfy Ruitian's system. When the native namespace quota is triggered, pods directly fail. Therefore, Ruitian re-designed the quota in Volcano. -{{
}} +![](/img/ruitian2-12.png) - When the Volcano namespace quota is triggered, pod creation in a queue will be suspended. __Volcano Monitoring and Alarming__ -{{
}} +![](/img/ruitian2-13.png) Volcano Exporter @@ -209,14 +196,14 @@ WatchDog __Job dashboard__ -{{
}} +![](/img/ruitian2-14.png) The upper panel covers the information about all jobs and provides a state table to display the job completion status. The panels below display the CPU, memory, and network resource usage. The negative axes refer to wasted cluster resources, which are allocated to pods (jobs) but not actually used during job running. These time series tables can provide resource insights to users in real time. __Cluster resource dashboard__ -{{
}} +![](/img/ruitian2-15.png) Graphs show the usage of overall queue resources, including CPU and memory. For jobs that consume a large amount of resources, for example, 300 or 500 GiB of memory, users need to know whether there is any node that can run such jobs. Therefore, we need to display the resource usage of each node available. @@ -228,7 +215,7 @@ In Ruitian, the number of compute nodes in a single cluster has reached 200 and __Challenge 1: Too Large Jobs__ -{{
}} +![](/img/ruitian2-16.png) Issues: - The CPU usage exceeds Max Request Size (1.5 MB) of etcd when there are a large number of pods. @@ -243,7 +230,7 @@ Solution: __Challenge 2: Out of CPU/Memory__ -{{
}} +![](/img/ruitian2-17.png) Issues: @@ -251,7 +238,7 @@ Issues: - Kubelet PLEG is under great pressure, and the pod binding takes too long. -{{
}} +![](/img/ruitian2-18.png) Issues: @@ -271,6 +258,6 @@ Solution: - When a snapshot is being created for a session, the nodes whose binding task number is smaller than 0 are skipped. -{{
}} +![](/img/ruitian2-19.png) diff --git a/content/en/blog/volcano-2025-security-audit.md b/blog/volcano-2025-security-audit.md similarity index 91% rename from content/en/blog/volcano-2025-security-audit.md rename to blog/volcano-2025-security-audit.md index cb3a51b4..791fb872 100644 --- a/content/en/blog/volcano-2025-security-audit.md +++ b/blog/volcano-2025-security-audit.md @@ -1,28 +1,21 @@ -+++ -title = "Volcano completes security audit" -description = "Volcano completes 2025 security audit" -subtitle = "" - -date = 2025-05-30 -lastmod = 2025-05-30 -datemonth = "May" -dateyear = "2025" -dateday = 30 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Adam Korczynski, Ada Logics", "Xavier Chang, Huawei and Volcano maintainer"] - -tags = ["News"] -summary = "Volcano completes 2025 security audit" - -# Add menu entry to sidebar. -linktitle = "volcano 2025 security audit" -[menu.posts] -parent = "news" -weight = 6 -+++ +--- +title: "Volcano completes security audit" +description: "Volcano completes 2025 security audit" +subtitle: "" +date: 2025-05-30 +draft: false +authors: + - "Adam Korczynski" + - "Ada Logics" + - "Xavier Chang" + - "Huawei and Volcano maintainer" +tags: + - News +summary: "Volcano completes 2025 security audit" +linktitle: volcano 2025 security audit +parent: news +weight: '6' +--- Volcano is excited to announce the completion of our CNCF-funded security audit carried out by [Ada Logics](https://adalogics.com/) and facilitated by [OSTIF](https://ostif.org/) in collaboration with the Volcano maintainers. The audit was scoped to cover the Volcano source code, supply-chain risks and fuzzing. The auditing team identified 10 security issues which the Volcano security team has fixed with the completion of the audit. diff --git a/content/en/blog/xiaohongshu-en.md b/blog/xiaohongshu-en.md similarity index 92% rename from content/en/blog/xiaohongshu-en.md rename to blog/xiaohongshu-en.md index ea8f0d86..6a4211e4 100644 --- a/content/en/blog/xiaohongshu-en.md +++ b/blog/xiaohongshu-en.md @@ -1,28 +1,15 @@ -+++ -title = "How Does Volcano Empower a Content Recommendation Engine in Xiaohongshu" -description = "Best practice of Volcano at Xiaohongshu" -subtitle = "" - -date = 2021-05-27 -lastmod = 2021-08-24 -datemonth = "Aug" -dateyear = "2021" -dateday = 24 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Yi Guo/Tech-Leader"] - -tags = ["Practice"] -summary = "Best practice of Volcano at Xiaohongshu" - -# Add menu entry to sidebar. -linktitle = "How Does Volcano Empower a Content Recommendation Engine in Xiaohongshu" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ +--- +title: "How Does Volcano Empower a Content Recommendation Engine in Xiaohongshu" +description: "Best practice of Volcano at Xiaohongshu" +subtitle: "" +date: 2021-05-27 +draft: false +authors: + - "Yi Guo/Tech-Leader" +tags: + - Practice +summary: "Best practice of Volcano at Xiaohongshu" +--- >This article was firstly released at `Container Cube` on May 27th, 2021, refer to [小红书基于Volcano的大规模离线与在线推荐模型训练实践](https://mp.weixin.qq.com/s/nZXZx78EQoHzRj1-LqMPQQ) @@ -42,7 +29,7 @@ Since 2016, we started to explore personalized recommendation for different user ## Big Data Architecture in Xiaohongshu Search, Recommendation, and Ad Scenarios -{{
}} +![](/img/xiaohongshu-en1.png) @@ -57,7 +44,7 @@ The upper right corner shows online and offline training. Online training trains __Behavior Attributions and Labels__ -{{
}} +![](/img/xiaohongshu-en2.png) This task for user behaviors can be divided into two parts: attribution and label. Attribution is to associate each behavior captured in data tracking with the past behaviors of the user. For example, you browse 4 notes in the app, one found on the Discovery page, one on the search results page, and two on a blogger's homepage. You click Like for the final piece. Your browsing and clicking Like are tracked. The tracking data does not tell us what happened before the like is given. @@ -70,7 +57,7 @@ Opposite to attribution, label calculation summarizes the actions performed by a __Real-Time Big Data Products for Search, Recommendation, and Ad__ -{{
}} +![](/img/xiaohongshu-en3.png) After the label data is generated, the above three big data products are provided for the service. @@ -84,7 +71,7 @@ Both the ad hoc data analysis and offline warehouse perform analysis based on th __Online and Offline Model Training__ -{{
}} +![](/img/xiaohongshu-en4.png) Training data generated after the combination of label data and feature data is used for both offline and online training. @@ -96,13 +83,13 @@ Though the same data source is used, there are differences between online and of __Offline Batch Only__ -{{
}} +![](/img/xiaohongshu-en5.png) The preceding figure shows the earliest offline batch label calculation process. Click behaviors of users are collected and recorded in the ODS table, that is, the original log table. The attribution and label calculation are performed by the Spark task, which is an offline batch task. The processed data forms a data warehouse table to generate daily reports and experiment reports as big data products. In the batch environment, the report is generated on a T+1 cycle. Generally, we cannot obtain the complete reports until the second or third day after each experiment. __Offline Batch + Online Streaming__ -{{
}} +![](/img/xiaohongshu-en6.png) Increase in the number of developers poses higher requirements on service implementation. Therefore, we introduced the real-time link, which is completely based on the Flink framework. The real-time link inputs data through streaming Kafka, outputs the data to Kafka, and sends the data to the OLAP database and real-time experiment analysis. However, the challenge is that Spark and Flink are two different programming frameworks. For example, the logic for determining whether a click on an ad is valid is complex, because an interaction behavior or a stay of at least three seconds after the click is required before the click can be called a valid click. @@ -120,20 +107,20 @@ We actually need to solve a technical problem: How to convert the real-time Flin __Offline Training__ -{{
}} +![](/img/xiaohongshu-en7.png) The preceding figure shows the training process of the machine learning model. At the earliest, there was only one batch data calculation task. Feature data and user behavior data are stored in offline tables. The Spark task is used to combine the data to generate a training task, and then release the learning model. The entire process may be performed once a day. __Online + Offline Training__ -{{
}} +![](/img/xiaohongshu-en8.png) To capture users' real-time interests and make judgment on newly released notes more quickly, more real-time update is required for our models. Therefore, Flink is used for real-time model training. After Flink generates data, the Volcano scheduling system is used to update models in real time and in batches. ## Optimization and Multi-cloud Management of Offline Training -{{
}} +![](/img/xiaohongshu-en9.png) The preceding figure shows the technology stack of Xiaohongshu in machine learning and big data. Xiaohongshu is a relatively new company without on-premises equipment rooms. All of our services are deployed on clouds provided by cloud vendors, and most of the services are managed through Kubernetes. @@ -155,7 +142,7 @@ Based on these, we researched products and found Volcano. Its predecessor is Kub __Enhanced Volcano Scheduling: binpack -> task-topology__ -{{
}} +![](/img/xiaohongshu-en10.png) The scheduling algorithm supported by the native Volcano is binpack. Machine learning training tasks are classified into two types: worker for performing forward and reverse computing, which is a computing task; ps with the main task of storing parameters, which is a memory-type service. If the native open-source Volcano is used, its default scheduling algorithm optimizes the resources to reduce fragments. Therefore, it will schedule as many tasks as possible to the same node, and then schedule all worker and ps tasks of those tasks to the same node. When that node does not have capacity for one of the ps tasks (taking ps1 as an example here), it can only be put on another node. @@ -167,7 +154,7 @@ With task-topology algorithm, tasks are scheduled to nodes in a balanced manner, __Data Transfer Between Multiple Clouds__ -{{
}} +![](/img/xiaohongshu-en11.png) In the online mode, users are distributed to different AZs. The feature cache of the recommendation service is stored in the local AZ. After user data tracking, users are distributed to different clusters based on their requests, and the label system performs computing for each user. Finally, all label system computing is transferred to the cloud vendors that provide offline training and services for us to combine and generate data training, and perform distributed model training. The trained models are distributed to different AZs for online services. diff --git a/build_log.txt b/build_log.txt new file mode 100644 index 00000000..fa13f427 --- /dev/null +++ b/build_log.txt @@ -0,0 +1,2639 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[webpackbar] ℹ Compiling Client +[webpackbar] ℹ Compiling Server +[webpackbar] ✔ Server: Compiled with some errors in 1.53s + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/referrals.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/plugins.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/plugins/drf.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/referrals.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/mpi_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/mpi_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/network_topology_aware_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/referrals.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/podgroup.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/architecture.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/kubeflow_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/kubeflow_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/mpi_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/plugins/priority.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/hierarchical_queue.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/multi_cluster_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/architecture.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/network_topology_aware_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/referrals.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/schduler_introduction.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/tf_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/referrals.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/plugins.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/referrals.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/hierarchical_queue.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/mpi_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/queue_resource_management.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/architecture.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/queue_resource_management.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/tf_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/plugins.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/tf_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/tf_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/descheduler.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/colocation.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/kubeflow_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/network_topology_aware_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/descheduler.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/schduler_introduction.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/architecture.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/podgroup.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/mpi_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/tf_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/podgroup.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/mpi_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/schduler_introduction.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/multi_cluster_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/architecture.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/kubeflow_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/plugins/gang.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/plugins.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/kubeflow_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/scheduler_introduction.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/gpu_virtualization.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/kubeflow_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/podgroup.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/descheduler.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/plugins.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/hierarchical_queue.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/plugins.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/architecture.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/schduler_introduction.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/multi_cluster_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/referrals.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/kubeflow_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/queue_resource_management.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/colocation.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/podgroup.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/mpi_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/architecture.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/schduler_introduction.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/tf_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/schduler_introduction.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/colocation.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/podgroup.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/podgroup.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/tf_on_volcano.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage +[webpackbar] ✔ Client: Compiled with some errors in 3.66s +[ERROR] Client bundle compiled with errors therefore further build is impossible. +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 30, + "name": "30:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 30, + "column": 28, + "offset": 1739 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/architecture.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 204, + "loc": { + "line": 14, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 14, + "name": "14:3", + "place": { + "line": 14, + "column": 3, + "offset": 204 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/colocation.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 5053, + "loc": { + "line": 48, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 48, + "name": "48:37", + "place": { + "line": 48, + "column": 37, + "offset": 5053 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/descheduler.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3519, + "loc": { + "line": 38, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 38, + "name": "38:37", + "place": { + "line": 38, + "column": 37, + "offset": 3519 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/hierarchical_queue.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3255, + "loc": { + "line": 63, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 63, + "name": "63:3", + "place": { + "line": 63, + "column": 3, + "offset": 3255 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/kubeflow_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1314, + "loc": { + "line": 19, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 19, + "name": "19:3", + "place": { + "line": 19, + "column": 3, + "offset": 1314 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/mpi_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1122, + "loc": { + "line": 31, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 31, + "name": "31:3", + "place": { + "line": 31, + "column": 3, + "offset": 1122 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/multi_cluster_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2147, + "loc": { + "line": 37, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 37, + "name": "37:37", + "place": { + "line": 37, + "column": 37, + "offset": 2147 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/network_topology_aware_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2588, + "loc": { + "line": 30, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 30, + "name": "30:37", + "place": { + "line": 30, + "column": 37, + "offset": 2588 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/plugins/drf.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 45, + "loc": { + "line": 7, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 7, + "name": "7:3", + "place": { + "line": 7, + "column": 3, + "offset": 45 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/plugins/gang.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 46, + "loc": { + "line": 7, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 7, + "name": "7:3", + "place": { + "line": 7, + "column": 3, + "offset": 46 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/plugins/priority.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 50, + "loc": { + "line": 7, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 7, + "name": "7:3", + "place": { + "line": 7, + "column": 3, + "offset": 50 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/podgroup.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3083, + "loc": { + "line": 85, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 85, + "name": "85:3", + "place": { + "line": 85, + "column": 3, + "offset": 3083 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/queue_resource_management.md" +Cause: Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_` +Details: +{ + "column": 71, + "file": "", + "message": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "line": 28, + "name": "28:71", + "place": { + "_bufferIndex": 59, + "_index": 0, + "line": 28, + "column": 71, + "offset": 1360 + }, + "reason": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/referrals.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 236, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 236 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/scheduler_introduction.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 576, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 576 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/tf_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 655, + "loc": { + "line": 21, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 21, + "name": "21:3", + "place": { + "line": 21, + "column": 3, + "offset": 655 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 29, + "name": "29:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 29, + "column": 28, + "offset": 1215 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/architecture.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 204, + "loc": { + "line": 14, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 14, + "name": "14:3", + "place": { + "line": 14, + "column": 3, + "offset": 204 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/kubeflow_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1314, + "loc": { + "line": 19, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 19, + "name": "19:3", + "place": { + "line": 19, + "column": 3, + "offset": 1314 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/mpi_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1122, + "loc": { + "line": 31, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 31, + "name": "31:3", + "place": { + "line": 31, + "column": 3, + "offset": 1122 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/plugins.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 190, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 190 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/podgroup.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2962, + "loc": { + "line": 76, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 76, + "name": "76:3", + "place": { + "line": 76, + "column": 3, + "offset": 2962 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/referrals.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 236, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 236 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/schduler_introduction.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 576, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 576 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/tf_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 655, + "loc": { + "line": 21, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 21, + "name": "21:3", + "place": { + "line": 21, + "column": 3, + "offset": 655 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 29, + "name": "29:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 29, + "column": 28, + "offset": 1215 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/architecture.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 204, + "loc": { + "line": 14, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 14, + "name": "14:3", + "place": { + "line": 14, + "column": 3, + "offset": 204 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/kubeflow_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1314, + "loc": { + "line": 19, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 19, + "name": "19:3", + "place": { + "line": 19, + "column": 3, + "offset": 1314 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/mpi_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1122, + "loc": { + "line": 31, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 31, + "name": "31:3", + "place": { + "line": 31, + "column": 3, + "offset": 1122 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/plugins.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 190, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 190 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/podgroup.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2962, + "loc": { + "line": 76, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 76, + "name": "76:3", + "place": { + "line": 76, + "column": 3, + "offset": 2962 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/referrals.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 236, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 236 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/schduler_introduction.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 576, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 576 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/tf_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 655, + "loc": { + "line": 21, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 21, + "name": "21:3", + "place": { + "line": 21, + "column": 3, + "offset": 655 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 29, + "name": "29:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 29, + "column": 28, + "offset": 1215 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/architecture.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 204, + "loc": { + "line": 14, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 14, + "name": "14:3", + "place": { + "line": 14, + "column": 3, + "offset": 204 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/kubeflow_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1314, + "loc": { + "line": 19, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 19, + "name": "19:3", + "place": { + "line": 19, + "column": 3, + "offset": 1314 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/mpi_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1122, + "loc": { + "line": 31, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 31, + "name": "31:3", + "place": { + "line": 31, + "column": 3, + "offset": 1122 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/plugins.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 190, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 190 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/podgroup.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2962, + "loc": { + "line": 76, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 76, + "name": "76:3", + "place": { + "line": 76, + "column": 3, + "offset": 2962 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/referrals.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 236, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 236 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/schduler_introduction.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 576, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 576 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/tf_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 655, + "loc": { + "line": 21, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 21, + "name": "21:3", + "place": { + "line": 21, + "column": 3, + "offset": 655 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 29, + "name": "29:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 29, + "column": 28, + "offset": 1215 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/architecture.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 204, + "loc": { + "line": 14, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 14, + "name": "14:3", + "place": { + "line": 14, + "column": 3, + "offset": 204 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/kubeflow_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1312, + "loc": { + "line": 17, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 17, + "name": "17:3", + "place": { + "line": 17, + "column": 3, + "offset": 1312 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/mpi_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1122, + "loc": { + "line": 31, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 31, + "name": "31:3", + "place": { + "line": 31, + "column": 3, + "offset": 1122 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/plugins.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 190, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 190 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/podgroup.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2962, + "loc": { + "line": 76, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 76, + "name": "76:3", + "place": { + "line": 76, + "column": 3, + "offset": 2962 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/referrals.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 236, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 236 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/schduler_introduction.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 576, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 576 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/tf_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 655, + "loc": { + "line": 21, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 21, + "name": "21:3", + "place": { + "line": 21, + "column": 3, + "offset": 655 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 30, + "name": "30:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 30, + "column": 28, + "offset": 1739 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/architecture.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 204, + "loc": { + "line": 14, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 14, + "name": "14:3", + "place": { + "line": 14, + "column": 3, + "offset": 204 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/colocation.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 5053, + "loc": { + "line": 48, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 48, + "name": "48:37", + "place": { + "line": 48, + "column": 37, + "offset": 5053 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/descheduler.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3519, + "loc": { + "line": 38, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 38, + "name": "38:37", + "place": { + "line": 38, + "column": 37, + "offset": 3519 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/gpu_virtualization.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3279, + "loc": { + "line": 123, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 123, + "name": "123:37", + "place": { + "line": 123, + "column": 37, + "offset": 3279 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/hierarchical_queue.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3255, + "loc": { + "line": 63, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 63, + "name": "63:3", + "place": { + "line": 63, + "column": 3, + "offset": 3255 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/kubeflow_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1314, + "loc": { + "line": 19, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 19, + "name": "19:3", + "place": { + "line": 19, + "column": 3, + "offset": 1314 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/mpi_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1122, + "loc": { + "line": 31, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 31, + "name": "31:3", + "place": { + "line": 31, + "column": 3, + "offset": 1122 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/multi_cluster_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2152, + "loc": { + "line": 37, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 37, + "name": "37:37", + "place": { + "line": 37, + "column": 37, + "offset": 2152 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/network_topology_aware_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2615, + "loc": { + "line": 32, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 32, + "name": "32:37", + "place": { + "line": 32, + "column": 37, + "offset": 2615 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/plugins.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 190, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 190 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/podgroup.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3086, + "loc": { + "line": 85, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 85, + "name": "85:3", + "place": { + "line": 85, + "column": 3, + "offset": 3086 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/queue_resource_management.md" +Cause: Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_` +Details: +{ + "column": 71, + "file": "", + "message": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "line": 28, + "name": "28:71", + "place": { + "_bufferIndex": 59, + "_index": 0, + "line": 28, + "column": 71, + "offset": 1360 + }, + "reason": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/referrals.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 236, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 236 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/schduler_introduction.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 576, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 576 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/tf_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 655, + "loc": { + "line": 21, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 21, + "name": "21:3", + "place": { + "line": 21, + "column": 3, + "offset": 655 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 30, + "name": "30:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 30, + "column": 28, + "offset": 1739 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/architecture.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 204, + "loc": { + "line": 14, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 14, + "name": "14:3", + "place": { + "line": 14, + "column": 3, + "offset": 204 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/colocation.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 5053, + "loc": { + "line": 48, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 48, + "name": "48:37", + "place": { + "line": 48, + "column": 37, + "offset": 5053 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/descheduler.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3519, + "loc": { + "line": 38, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 38, + "name": "38:37", + "place": { + "line": 38, + "column": 37, + "offset": 3519 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/hierarchical_queue.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3255, + "loc": { + "line": 63, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 63, + "name": "63:3", + "place": { + "line": 63, + "column": 3, + "offset": 3255 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/kubeflow_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1314, + "loc": { + "line": 19, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 19, + "name": "19:3", + "place": { + "line": 19, + "column": 3, + "offset": 1314 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/mpi_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 1122, + "loc": { + "line": 31, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 31, + "name": "31:3", + "place": { + "line": 31, + "column": 3, + "offset": 1122 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/multi_cluster_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2147, + "loc": { + "line": 37, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 37, + "name": "37:37", + "place": { + "line": 37, + "column": 37, + "offset": 2147 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/network_topology_aware_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2588, + "loc": { + "line": 30, + "column": 36 + }, + "raisedAt": 2 + }, + "column": 37, + "file": "", + "message": "Could not parse expression with acorn", + "line": 30, + "name": "30:37", + "place": { + "line": 30, + "column": 37, + "offset": 2588 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/plugins.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 190, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 190 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/podgroup.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3086, + "loc": { + "line": 85, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 85, + "name": "85:3", + "place": { + "line": 85, + "column": 3, + "offset": 3086 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/queue_resource_management.md" +Cause: Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_` +Details: +{ + "column": 71, + "file": "", + "message": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "line": 28, + "name": "28:71", + "place": { + "_bufferIndex": 59, + "_index": 0, + "line": 28, + "column": 71, + "offset": 1360 + }, + "reason": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/referrals.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 236, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 236 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/schduler_introduction.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 576, + "loc": { + "line": 15, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 15, + "name": "15:3", + "place": { + "line": 15, + "column": 3, + "offset": 576 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/tf_on_volcano.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 655, + "loc": { + "line": 21, + "column": 2 + }, + "raisedAt": 2 + }, + "column": 3, + "file": "", + "message": "Could not parse expression with acorn", + "line": 21, + "name": "21:3", + "place": { + "line": 21, + "column": 3, + "offset": 655 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} diff --git a/build_log_2.txt b/build_log_2.txt new file mode 100644 index 00000000..58c39a83 --- /dev/null +++ b/build_log_2.txt @@ -0,0 +1,661 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[webpackbar] ℹ Compiling Client +[webpackbar] ℹ Compiling Server +[webpackbar] ✔ Server: Compiled with some errors in 2.36s + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/multi_cluster_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/network_topology_aware_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/network_topology_aware_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/queue_resource_management.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/queue_resource_management.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/descheduler.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/colocation.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/network_topology_aware_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/descheduler.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/multi_cluster_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/gpu_virtualization.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/descheduler.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/multi_cluster_scheduling.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/queue_resource_management.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/colocation.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/actions.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/colocation.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage +[webpackbar] ✔ Client: Compiled with some errors in 5.57s +[ERROR] Client bundle compiled with errors therefore further build is impossible. +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/colocation.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 5020, + "loc": { + "line": 48, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 48, + "name": "48:4", + "place": { + "line": 48, + "column": 4, + "offset": 5020 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/descheduler.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3486, + "loc": { + "line": 38, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 38, + "name": "38:4", + "place": { + "line": 38, + "column": 4, + "offset": 3486 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/multi_cluster_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2114, + "loc": { + "line": 37, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 37, + "name": "37:4", + "place": { + "line": 37, + "column": 4, + "offset": 2114 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/network_topology_aware_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2555, + "loc": { + "line": 30, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 30, + "name": "30:4", + "place": { + "line": 30, + "column": 4, + "offset": 2555 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/queue_resource_management.md" +Cause: Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_` +Details: +{ + "column": 71, + "file": "", + "message": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "line": 28, + "name": "28:71", + "place": { + "_bufferIndex": 59, + "_index": 0, + "line": 28, + "column": 71, + "offset": 1360 + }, + "reason": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-7-0/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 29, + "name": "29:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 29, + "column": 28, + "offset": 1215 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-8-2/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 29, + "name": "29:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 29, + "column": 28, + "offset": 1215 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-9-0/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 29, + "name": "29:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 29, + "column": 28, + "offset": 1215 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-10-0/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 29, + "name": "29:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 29, + "column": 28, + "offset": 1215 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 30, + "name": "30:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 30, + "column": 28, + "offset": 1739 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/colocation.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 5020, + "loc": { + "line": 48, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 48, + "name": "48:4", + "place": { + "line": 48, + "column": 4, + "offset": 5020 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/descheduler.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3486, + "loc": { + "line": 38, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 38, + "name": "38:4", + "place": { + "line": 38, + "column": 4, + "offset": 3486 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/gpu_virtualization.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3246, + "loc": { + "line": 123, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 123, + "name": "123:4", + "place": { + "line": 123, + "column": 4, + "offset": 3246 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/multi_cluster_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2119, + "loc": { + "line": 37, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 37, + "name": "37:4", + "place": { + "line": 37, + "column": 4, + "offset": 2119 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/network_topology_aware_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2582, + "loc": { + "line": 32, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 32, + "name": "32:4", + "place": { + "line": 32, + "column": 4, + "offset": 2582 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/queue_resource_management.md" +Cause: Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_` +Details: +{ + "column": 71, + "file": "", + "message": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "line": 28, + "name": "28:71", + "place": { + "_bufferIndex": 59, + "_index": 0, + "line": 28, + "column": 71, + "offset": 1360 + }, + "reason": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/actions.md" +Cause: Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag +Details: +{ + "column": 28, + "file": "", + "message": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "line": 30, + "name": "30:28", + "place": { + "_bufferIndex": 27, + "_index": 0, + "line": 30, + "column": 28, + "offset": 1739 + }, + "reason": "Unexpected character `,` (U+002C) after name, expected a character that can start an attribute name, such as a letter, `$`, or `_`; whitespace before attributes; or the end of the tag", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/colocation.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 5020, + "loc": { + "line": 48, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 48, + "name": "48:4", + "place": { + "line": 48, + "column": 4, + "offset": 5020 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/descheduler.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 3486, + "loc": { + "line": 38, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 38, + "name": "38:4", + "place": { + "line": 38, + "column": 4, + "offset": 3486 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/multi_cluster_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2114, + "loc": { + "line": 37, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 37, + "name": "37:4", + "place": { + "line": 37, + "column": 4, + "offset": 2114 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/network_topology_aware_scheduling.md" +Cause: Could not parse expression with acorn +Details: +{ + "cause": { + "pos": 2555, + "loc": { + "line": 30, + "column": 3 + }, + "raisedAt": 2 + }, + "column": 4, + "file": "", + "message": "Could not parse expression with acorn", + "line": 30, + "name": "30:4", + "place": { + "line": 30, + "column": 4, + "offset": 2555 + }, + "reason": "Could not parse expression with acorn", + "ruleId": "acorn", + "source": "micromark-extension-mdx-expression", + "url": "https://github.com/micromark/micromark-extension-mdx-expression/tree/main/packages/micromark-extension-mdx-expression#could-not-parse-expression-with-acorn" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/queue_resource_management.md" +Cause: Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_` +Details: +{ + "column": 71, + "file": "", + "message": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "line": 28, + "name": "28:71", + "place": { + "_bufferIndex": 59, + "_index": 0, + "line": 28, + "column": 71, + "offset": 1360 + }, + "reason": "Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} diff --git a/build_log_3.txt b/build_log_3.txt new file mode 100644 index 00000000..4443eb13 --- /dev/null +++ b/build_log_3.txt @@ -0,0 +1,201 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[webpackbar] ℹ Compiling Client +[webpackbar] ℹ Compiling Server +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +[WARNING] Markdown link with URL `../../installer/README.md` in source file "docs/v1-11-0/gpu_virtualization.md" (26:10) couldn't be resolved. +Make sure it references a local Markdown file that exists within the current plugin. +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +[WARNING] Markdown link with URL `../../installer/README.md` in source file "docs/v1-11-0/gpu_virtualization.md" (26:10) couldn't be resolved. +Make sure it references a local Markdown file that exists within the current plugin. +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +[webpackbar] ✔ Server: Compiled with some errors in 2.16s + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/colocation.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/colocation.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/colocation.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage +[webpackbar] ✔ Client: Compiled with some errors in 4.16s +[ERROR] Client bundle compiled with errors therefore further build is impossible. +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/colocation.md" +Cause: Unexpected closing tag ``, expected corresponding closing tag for `
` (404:117-404:121) +Details: +{ + "column": 239, + "file": "", + "message": "Unexpected closing tag ``, expected corresponding closing tag for `
` (404:117-404:121)", + "line": 404, + "name": "404:239-404:244", + "place": { + "start": { + "_bufferIndex": 238, + "_index": 2, + "line": 404, + "column": 239, + "offset": 25553 + }, + "end": { + "_bufferIndex": -1, + "_index": 3, + "line": 404, + "column": 244, + "offset": 25558 + } + }, + "reason": "Unexpected closing tag ``, expected corresponding closing tag for `
` (404:117-404:121)", + "ruleId": "end-tag-mismatch", + "source": "mdast-util-mdx-jsx" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-11-0/colocation.md" +Cause: Unexpected closing tag ``, expected corresponding closing tag for `
` (404:117-404:121) +Details: +{ + "column": 239, + "file": "", + "message": "Unexpected closing tag ``, expected corresponding closing tag for `
` (404:117-404:121)", + "line": 404, + "name": "404:239-404:244", + "place": { + "start": { + "_bufferIndex": 238, + "_index": 2, + "line": 404, + "column": 239, + "offset": 25553 + }, + "end": { + "_bufferIndex": -1, + "_index": 3, + "line": 404, + "column": 244, + "offset": 25558 + } + }, + "reason": "Unexpected closing tag ``, expected corresponding closing tag for `
` (404:117-404:121)", + "ruleId": "end-tag-mismatch", + "source": "mdast-util-mdx-jsx" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/docs/v1-12-0/colocation.md" +Cause: Unexpected closing tag ``, expected corresponding closing tag for `
` (404:117-404:121) +Details: +{ + "column": 239, + "file": "", + "message": "Unexpected closing tag ``, expected corresponding closing tag for `
` (404:117-404:121)", + "line": 404, + "name": "404:239-404:244", + "place": { + "start": { + "_bufferIndex": 238, + "_index": 2, + "line": 404, + "column": 239, + "offset": 25553 + }, + "end": { + "_bufferIndex": -1, + "_index": 3, + "line": 404, + "column": 244, + "offset": 25558 + } + }, + "reason": "Unexpected closing tag ``, expected corresponding closing tag for `
` (404:117-404:121)", + "ruleId": "end-tag-mismatch", + "source": "mdast-util-mdx-jsx" +} diff --git a/build_log_4.txt b/build_log_4.txt new file mode 100644 index 00000000..da3ffb84 --- /dev/null +++ b/build_log_4.txt @@ -0,0 +1,138 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[webpackbar] ℹ Compiling Client +[webpackbar] ℹ Compiling Server +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +[WARNING] Markdown link with URL `../../installer/README.md` in source file "docs/v1-11-0/gpu_virtualization.md" (26:10) couldn't be resolved. +Make sure it references a local Markdown file that exists within the current plugin. +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +[webpackbar] ✔ Server: Compiled successfully in 1.73s +[webpackbar] ✔ Client: Compiled successfully in 4.07s + +[ERROR] Error: Unable to build website for locale en. + at tryToBuildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:83:15) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:35:9 + ... 4 lines matching cause stack trace ... + at async file:///Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 { + [cause]: Error: Docusaurus static site generation failed for 3 paths: + - "/docs/colocation" + - "/docs/v1-11-0/colocation" + - "/docs/v1-12-0/colocation" + at throwSSGError (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/ssg/ssgGlobalResult.js:56:11) + at createGlobalSSGResult (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/ssg/ssgGlobalResult.js:66:9) + ... 11 lines matching cause stack trace ... + at async file:///Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 { + [cause]: AggregateError + at throwSSGError (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/ssg/ssgGlobalResult.js:57:16) + at createGlobalSSGResult (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/ssg/ssgGlobalResult.js:66:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/ssg/ssgExecutor.js:30:68 + at async executeSSG (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/ssg/ssgExecutor.js:144:20) + at async buildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/buildLocale.js:84:31) + at async runBuildLocaleTask (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:98:5) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:79:13 + at async tryToBuildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:75:9) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:35:9 + at async mapAsyncSequential (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/utils/lib/jsUtils.js:21:24) + at async Command.build (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:34:5) + at async Promise.all (index 0) + at async runCLI (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/cli.js:56:5) + at async file:///Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 { + [errors]: [ + Error: Can't render static file for pathname "/docs/colocation" + at generateStaticFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/ssg/ssgRenderer.js:131:20) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/p-map/index.js:57:22 { + [cause]: Error: The `style` prop expects a mapping from style properties to values, not a string. For example, style={{marginRight: spacing + 'em'}} when using JSX. + at pushStyleAttribute (server.bundle.js:41249:11) + at pushAttribute (server.bundle.js:41408:7) + at pushStartGenericElement (server.bundle.js:41799:13) + at pushStartInstance (server.bundle.js:42855:10) + at renderElement (server.bundle.js:45370:22) + at retryNode (server.bundle.js:46065:16) + at renderNodeDestructive (server.bundle.js:45859:7) + at renderNode (server.bundle.js:46426:14) + at renderChildrenArray (server.bundle.js:46208:7) + at retryNode (server.bundle.js:46079:9) + }, + Error: Can't render static file for pathname "/docs/v1-11-0/colocation" + at generateStaticFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/ssg/ssgRenderer.js:131:20) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/p-map/index.js:57:22 { + [cause]: Error: The `style` prop expects a mapping from style properties to values, not a string. For example, style={{marginRight: spacing + 'em'}} when using JSX. + at pushStyleAttribute (server.bundle.js:41249:11) + at pushAttribute (server.bundle.js:41408:7) + at pushStartGenericElement (server.bundle.js:41799:13) + at pushStartInstance (server.bundle.js:42855:10) + at renderElement (server.bundle.js:45370:22) + at retryNode (server.bundle.js:46065:16) + at renderNodeDestructive (server.bundle.js:45859:7) + at renderNode (server.bundle.js:46426:14) + at renderChildrenArray (server.bundle.js:46208:7) + at retryNode (server.bundle.js:46079:9) + }, + Error: Can't render static file for pathname "/docs/v1-12-0/colocation" + at generateStaticFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/ssg/ssgRenderer.js:131:20) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/p-map/index.js:57:22 { + [cause]: Error: The `style` prop expects a mapping from style properties to values, not a string. For example, style={{marginRight: spacing + 'em'}} when using JSX. + at pushStyleAttribute (server.bundle.js:41249:11) + at pushAttribute (server.bundle.js:41408:7) + at pushStartGenericElement (server.bundle.js:41799:13) + at pushStartInstance (server.bundle.js:42855:10) + at renderElement (server.bundle.js:45370:22) + at retryNode (server.bundle.js:46065:16) + at renderNodeDestructive (server.bundle.js:45859:7) + at renderNode (server.bundle.js:46426:14) + at renderChildrenArray (server.bundle.js:46208:7) + at retryNode (server.bundle.js:46079:9) + } + ] + } + } +} +[INFO] Docusaurus version: 3.9.2 +Node version: v22.21.0 diff --git a/build_log_5.txt b/build_log_5.txt new file mode 100644 index 00000000..12decdd4 --- /dev/null +++ b/build_log_5.txt @@ -0,0 +1,414 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[webpackbar] ℹ Compiling Client +[webpackbar] ℹ Compiling Server +[webpackbar] ✔ Server: Compiled successfully in 733.03ms +[webpackbar] ✔ Client: Compiled successfully in 1.07s +[WARNING] Docusaurus found broken links! + +Please check the pages of your site in the list below, and make sure you don't reference any path that does not exist. +Note: it's possible to ignore broken links with the 'onBrokenLinks' Docusaurus configuration, and let the build pass. + +It looks like some of the broken links we found appear in many pages of your site. +Maybe those broken links appear on all pages through your site layout? +We recommend that you check your theme configuration for such links (particularly, theme navbar and footer). +Frequent broken links are linking to: +- /blog + +Exhaustive list of all broken links found: +- Broken link on source page path = /404.html: + -> linking to /blog +- Broken link on source page path = /markdown-page: + -> linking to /blog +- Broken link on source page path = /docs/actions: + -> linking to /blog +- Broken link on source page path = /docs/architecture: + -> linking to /blog +- Broken link on source page path = /docs/cli: + -> linking to /blog +- Broken link on source page path = /docs/colocation: + -> linking to /blog +- Broken link on source page path = /docs/contribution: + -> linking to /blog +- Broken link on source page path = /docs/cron_volcanoJob: + -> linking to /blog +- Broken link on source page path = /docs/descheduler: + -> linking to /blog +- Broken link on source page path = /docs/flink_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/gpu_virtualization: + -> linking to /blog +- Broken link on source page path = /docs/hierarchical_queue: + -> linking to /blog +- Broken link on source page path = /docs/installation: + -> linking to /blog +- Broken link on source page path = /docs/introduction: + -> linking to /blog +- Broken link on source page path = /docs/kubeflow_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/membership: + -> linking to /blog +- Broken link on source page path = /docs/mindspore_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/mpi_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/multi_cluster_scheduling: + -> linking to /blog +- Broken link on source page path = /docs/network_topology_aware_scheduling: + -> linking to /blog +- Broken link on source page path = /docs/plugins/: + -> linking to /blog +- Broken link on source page path = /docs/plugins/binpack: + -> linking to /blog +- Broken link on source page path = /docs/plugins/drf: + -> linking to /blog +- Broken link on source page path = /docs/plugins/gang: + -> linking to /blog +- Broken link on source page path = /docs/plugins/nodeorder: + -> linking to /blog +- Broken link on source page path = /docs/plugins/numa-aware: + -> linking to /blog +- Broken link on source page path = /docs/plugins/predicates: + -> linking to /blog +- Broken link on source page path = /docs/plugins/priority: + -> linking to /blog +- Broken link on source page path = /docs/plugins/proportion: + -> linking to /blog +- Broken link on source page path = /docs/plugins/sla: + -> linking to /blog +- Broken link on source page path = /docs/plugins/task-topology: + -> linking to /blog +- Broken link on source page path = /docs/plugins/tdm: + -> linking to /blog +- Broken link on source page path = /docs/podgroup: + -> linking to /blog +- Broken link on source page path = /docs/pp_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/queue: + -> linking to /blog +- Broken link on source page path = /docs/queue_resource_management: + -> linking to /blog +- Broken link on source page path = /docs/ray_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/referrals: + -> linking to /blog +- Broken link on source page path = /docs/scheduler_introduction: + -> linking to /blog +- Broken link on source page path = /docs/spark_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/tf_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/tutorials: + -> linking to /blog +- Broken link on source page path = /docs/unified_scheduling: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/actions: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/architecture: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/cli: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/contribution: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/flink_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/installation: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/kubeflow_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/membership: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/mindspore_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/mpi_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/plugins: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/podgroup: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/pp_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/queue: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/referrals: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/schduler_introduction: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/spark_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/tf_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/tutorials: + -> linking to /blog +- Broken link on source page path = /docs/v1-10-0/vcjob: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/actions: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/architecture: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/cli: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/colocation: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/contribution: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/descheduler: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/flink_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/gpu_virtualization: + -> linking to /blog + -> linking to ../../installer/README.md (resolved as: /installer/README.md) +- Broken link on source page path = /docs/v1-11-0/hierarchical_queue: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/installation: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/kubeflow_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/membership: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/mindspore_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/mpi_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/multi_cluster_scheduling: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/network_topology_aware_scheduling: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/plugins: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/podgroup: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/pp_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/queue: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/queue_resource_management: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/referrals: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/schduler_introduction: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/spark_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/tf_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/tutorials: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/unified_scheduling: + -> linking to /blog +- Broken link on source page path = /docs/v1-11-0/vcjob: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/actions: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/architecture: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/cli: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/colocation: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/contribution: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/descheduler: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/flink_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/gpu_virtualization: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/hierarchical_queue: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/installation: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/kubeflow_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/membership: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/mindspore_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/mpi_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/multi_cluster_scheduling: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/network_topology_aware_scheduling: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/plugins: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/podgroup: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/pp_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/queue: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/queue_resource_management: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/referrals: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/schduler_introduction: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/spark_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/tf_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/tutorials: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/unified_scheduling: + -> linking to /blog +- Broken link on source page path = /docs/v1-12-0/vcjob: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/actions: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/architecture: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/cli: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/contribution: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/flink_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/installation: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/kubeflow_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/membership: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/mindspore_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/mpi_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/plugins: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/podgroup: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/pp_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/queue: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/referrals: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/schduler_introduction: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/spark_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/tf_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/tutorials: + -> linking to /blog +- Broken link on source page path = /docs/v1-7-0/vcjob: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/actions: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/architecture: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/cli: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/contribution: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/flink_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/installation: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/kubeflow_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/membership: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/mindspore_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/mpi_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/plugins: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/podgroup: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/pp_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/queue: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/referrals: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/schduler_introduction: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/spark_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/tf_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/tutorials: + -> linking to /blog +- Broken link on source page path = /docs/v1-8-2/vcjob: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/actions: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/architecture: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/cli: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/contribution: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/flink_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/installation: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/kubeflow_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/membership: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/mindspore_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/mpi_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/plugins: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/podgroup: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/pp_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/queue: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/referrals: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/schduler_introduction: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/spark_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/tf_on_volcano: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/tutorials: + -> linking to /blog +- Broken link on source page path = /docs/v1-9-0/vcjob: + -> linking to /blog +- Broken link on source page path = /docs/vcjob: + -> linking to /blog +- Broken link on source page path = /: + -> linking to /blog + +[WARNING] Docusaurus found broken anchors! + +Please check the pages of your site in the list below, and make sure you don't reference any anchor that does not exist. +Note: it's possible to ignore broken anchors with the 'onBrokenAnchors' Docusaurus configuration, and let the build pass. + +Exhaustive list of all broken anchors found: +- Broken anchor on source page path = /docs/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) +- Broken anchor on source page path = /docs/v1-12-0/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/v1-12-0/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) + +[SUCCESS] Generated static files in "build". +[INFO] Use `npm run serve` command to test your build locally. diff --git a/build_log_blog.txt b/build_log_blog.txt new file mode 100644 index 00000000..ccca5df5 --- /dev/null +++ b/build_log_blog.txt @@ -0,0 +1,93 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[ERROR] The following front matter: +--- +title: Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios +description: Join Volcano at KubeCon + CloudNativeCon Europe, 19-22 March in Paris! +subtitle: '' +date: '2024-03-08' +lastmod: '2024-03-08' +datemonth: Mar +dateyear: '2024' +dateday: '08' +draft: 'false # Is this a draft? true/false' +toc: 'true # Show table of contents? true/false' +type: '"posts" # Do not modify.' +authors: + - volcano +tags: + - News +summary: Join Volcano at KubeCon + CloudNativeCon Europe, 19-22 March in Paris! +linktitle: Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios +parent: news +weight: '6' +--- +contains invalid values for field(s): `draft`. + +- "draft" must be a boolean + +[ERROR] Error while parsing blog post file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios.md". + +[ERROR] Error: Unable to build website for locale en. + at tryToBuildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:83:15) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:35:9 + at async mapAsyncSequential (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/utils/lib/jsUtils.js:21:24) + at async Command.build (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:34:5) + at async Promise.all (index 0) + at async runCLI (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/cli.js:56:5) + at async file:///Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 { + [cause]: Error: Processing of blog source file path=Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios.md failed. + at doProcessBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:271:19) + at async Promise.all (index 2) + at async generateBlogPosts (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:274:24) + at async Object.loadContent (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/index.js:155:29) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:40:23 + at async Promise.all (index 2) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:146:25 + at async loadSite (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/site.js:155:45) + at async buildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/buildLocale.js:33:18) + at async runBuildLocaleTask (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:98:5) { + [cause]: [Error [ValidationError]: "draft" must be a boolean] { + _original: { + title: 'Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios', + description: 'Join Volcano at KubeCon + CloudNativeCon Europe, 19-22 March in Paris!', + subtitle: '', + date: '2024-03-08', + lastmod: '2024-03-08', + datemonth: 'Mar', + dateyear: '2024', + dateday: '08', + draft: 'false # Is this a draft? true/false', + toc: 'true # Show table of contents? true/false', + type: '"posts" # Do not modify.', + authors: [ 'volcano' ], + tags: [ 'News' ], + summary: 'Join Volcano at KubeCon + CloudNativeCon Europe, 19-22 March in Paris!', + linktitle: 'Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios', + parent: 'news', + weight: '6' + }, + details: [ + { + message: '"draft" must be a boolean', + path: [ 'draft' ], + type: 'boolean.base', + context: { + label: 'draft', + value: 'false # Is this a draft? true/false', + key: 'draft' + } + } + ] + } + } +} +[INFO] Docusaurus version: 3.9.2 +Node version: v22.21.0 diff --git a/build_log_blog_2.txt b/build_log_blog_2.txt new file mode 100644 index 00000000..9ccc2ec5 --- /dev/null +++ b/build_log_blog_2.txt @@ -0,0 +1,45 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. + +[ERROR] Error: Unable to build website for locale en. + at tryToBuildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:83:15) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:35:9 + at async mapAsyncSequential (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/utils/lib/jsUtils.js:21:24) + at async Command.build (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:34:5) + at async Promise.all (index 0) + at async runCLI (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/cli.js:56:5) + at async file:///Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 { + [cause]: Error: Processing of blog source file path=1.4 release-en.md failed. + at doProcessBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:271:19) + at async Promise.all (index 0) + at async generateBlogPosts (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:274:24) + at async Object.loadContent (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/index.js:155:29) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:40:23 + at async Promise.all (index 2) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:146:25 + at async loadSite (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/site.js:155:45) + at async buildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/buildLocale.js:33:18) + at async runBuildLocaleTask (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:98:5) { + [cause]: Error: Can't reference blog post authors by a key (such as 'Thor-wl') because no authors map file could be loaded. + Please double-check your blog plugin config (in particular 'authorsMapPath'), ensure the file exists at the configured path, is not empty, and is valid! + at getAuthorsMapAuthor (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:82:23) + at toAuthor (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:100:16) + at Array.map () + at getFrontMatterAuthors (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:58:42) + at getBlogPostAuthors (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:114:21) + at processBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:215:54) + at async doProcessBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:268:20) + at async Promise.all (index 0) + at async generateBlogPosts (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:274:24) + at async Object.loadContent (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/index.js:155:29) + } +} +[INFO] Docusaurus version: 3.9.2 +Node version: v22.21.0 diff --git a/build_log_blog_3.txt b/build_log_blog_3.txt new file mode 100644 index 00000000..506723fb --- /dev/null +++ b/build_log_blog_3.txt @@ -0,0 +1,56 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. + +[ERROR] Error: Unable to build website for locale en. + at tryToBuildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:83:15) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:35:9 + at async mapAsyncSequential (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/utils/lib/jsUtils.js:21:24) + at async Command.build (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:34:5) + at async Promise.all (index 0) + at async runCLI (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/cli.js:56:5) + at async file:///Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 { + [cause]: Error: Processing of blog source file path=ruitian-en.md failed. + at doProcessBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:271:19) + at async Promise.all (index 22) + at async generateBlogPosts (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:274:24) + at async Object.loadContent (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/index.js:155:29) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:40:23 + at async Promise.all (index 2) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:146:25 + at async loadSite (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/site.js:155:45) + at async buildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/buildLocale.js:33:18) + at async runBuildLocaleTask (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:98:5) { + [cause]: Error: Blog author with key "Zhengyu Xu/R&D in Ruitian" not found in the authors map file. + Valid author keys are: + - volcano + - Volcano + - Adam Korczynski + - Hao Li/Senior R&D Engineer in iQIYI + - Zhe Jin/Senior Engineer in HUAWEI CLOUD + - Junmao Li/System Architect in Pengcheng Laboratory + - Thor-wl + - Yunzhi Cheng/Ruitian Tech-Leader + - Yunzhi Cheng/Tech-Leader in Ruitian + - Jian Zhu/Senior Architect in Leinao OS + - Yi Guo/Tech-Leader + at getAuthorsMapAuthor (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:87:23) + at toAuthor (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:100:16) + at Array.map () + at getFrontMatterAuthors (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:58:42) + at getBlogPostAuthors (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:114:21) + at processBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:215:54) + at async doProcessBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:268:20) + at async Promise.all (index 22) + at async generateBlogPosts (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:274:24) + at async Object.loadContent (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/index.js:155:29) + } +} +[INFO] Docusaurus version: 3.9.2 +Node version: v22.21.0 diff --git a/build_log_blog_4.txt b/build_log_blog_4.txt new file mode 100644 index 00000000..277f68f3 --- /dev/null +++ b/build_log_blog_4.txt @@ -0,0 +1,57 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. + +[ERROR] Error: Unable to build website for locale en. + at tryToBuildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:83:15) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:35:9 + at async mapAsyncSequential (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/utils/lib/jsUtils.js:21:24) + at async Command.build (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:34:5) + at async Promise.all (index 0) + at async runCLI (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/cli.js:56:5) + at async file:///Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 { + [cause]: Error: Processing of blog source file path=volcano-2025-security-audit.md failed. + at doProcessBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:271:19) + at async Promise.all (index 24) + at async generateBlogPosts (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:274:24) + at async Object.loadContent (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/index.js:155:29) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:40:23 + at async Promise.all (index 2) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:146:25 + at async loadSite (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/site.js:155:45) + at async buildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/buildLocale.js:33:18) + at async runBuildLocaleTask (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:98:5) { + [cause]: Error: Blog author with key "Ada Logics" not found in the authors map file. + Valid author keys are: + - volcano + - Volcano + - Adam Korczynski + - Hao Li/Senior R&D Engineer in iQIYI + - Zhe Jin/Senior Engineer in HUAWEI CLOUD + - Junmao Li/System Architect in Pengcheng Laboratory + - Thor-wl + - Yunzhi Cheng/Ruitian Tech-Leader + - Yunzhi Cheng/Tech-Leader in Ruitian + - Jian Zhu/Senior Architect in Leinao OS + - Yi Guo/Tech-Leader + - Zhengyu Xu/R&D in Ruitian + at getAuthorsMapAuthor (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:87:23) + at toAuthor (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:100:16) + at Array.map () + at getFrontMatterAuthors (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:58:42) + at getBlogPostAuthors (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:114:21) + at processBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:215:54) + at async doProcessBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:268:20) + at async Promise.all (index 24) + at async generateBlogPosts (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:274:24) + at async Object.loadContent (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/index.js:155:29) + } +} +[INFO] Docusaurus version: 3.9.2 +Node version: v22.21.0 diff --git a/build_log_blog_5.txt b/build_log_blog_5.txt new file mode 100644 index 00000000..4884288f --- /dev/null +++ b/build_log_blog_5.txt @@ -0,0 +1,58 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. + +[ERROR] Error: Unable to build website for locale en. + at tryToBuildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:83:15) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:35:9 + at async mapAsyncSequential (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/utils/lib/jsUtils.js:21:24) + at async Command.build (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:34:5) + at async Promise.all (index 0) + at async runCLI (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/cli.js:56:5) + at async file:///Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 { + [cause]: Error: Processing of blog source file path=volcano-2025-security-audit.md failed. + at doProcessBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:271:19) + at async Promise.all (index 24) + at async generateBlogPosts (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:274:24) + at async Object.loadContent (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/index.js:155:29) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:40:23 + at async Promise.all (index 2) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:146:25 + at async loadSite (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/site.js:155:45) + at async buildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/buildLocale.js:33:18) + at async runBuildLocaleTask (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:98:5) { + [cause]: Error: Blog author with key "Xavier Chang" not found in the authors map file. + Valid author keys are: + - volcano + - Volcano + - Adam Korczynski + - Hao Li/Senior R&D Engineer in iQIYI + - Zhe Jin/Senior Engineer in HUAWEI CLOUD + - Junmao Li/System Architect in Pengcheng Laboratory + - Thor-wl + - Yunzhi Cheng/Ruitian Tech-Leader + - Yunzhi Cheng/Tech-Leader in Ruitian + - Jian Zhu/Senior Architect in Leinao OS + - Yi Guo/Tech-Leader + - Zhengyu Xu/R&D in Ruitian + - Ada Logics + at getAuthorsMapAuthor (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:87:23) + at toAuthor (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:100:16) + at Array.map () + at getFrontMatterAuthors (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:58:42) + at getBlogPostAuthors (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:114:21) + at processBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:215:54) + at async doProcessBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:268:20) + at async Promise.all (index 24) + at async generateBlogPosts (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:274:24) + at async Object.loadContent (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/index.js:155:29) + } +} +[INFO] Docusaurus version: 3.9.2 +Node version: v22.21.0 diff --git a/build_log_blog_6.txt b/build_log_blog_6.txt new file mode 100644 index 00000000..2e3dcbe1 --- /dev/null +++ b/build_log_blog_6.txt @@ -0,0 +1,383 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[WARNING] Error: Failed to retrieve the git history for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md" because the file is not tracked by git. +[WARNING] Error: Failed to retrieve the git history for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/paddlepaddle-en.md" because the file is not tracked by git. +[WARNING] Docusaurus found blog posts without truncation markers: +- "blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md" +- "blog/paddlepaddle-en.md" +- "blog/Volcano-1.13.0-release.md" +- "blog/iflytek_case_study.md" +- "blog/Volcano-1.12.0-release.md" +- "blog/volcano-2025-security-audit.md" +- "blog/how-volcano-boosts-distributed-training-and-inference-performance.md" +- "blog/Volcano-1.11.0-release.md" +- "blog/Volcano-1.10.0-release.md" +- "blog/Volcano-1.9.0-release.md" +- "blog/Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios.md" +- "blog/Volcano-1.8.2-release.md" +- "blog/Volcano-community-co-construction-program.md" +- "blog/Volcano-1.7.0-release-en.md" +- "blog/ING_case-en.md" +- "blog/1.4 release-en.md" +- "blog/ruitian2-en.md" +- "blog/pengcheng-en.md" +- "blog/xiaohongshu-en.md" +- "blog/ruitian-en.md" +- "blog/leinao-en.md" +- "blog/hpc-en.md" +- "blog/aiqiyi-en.md" +- "blog/Quick-Start-Volcano.md" +- "blog/kube-batch-customers.md" +- "blog/kube-batch-startup.md" + +We recommend using truncation markers (`` or `{/* truncate */}`) in blog posts to create shorter previews on blog paginated lists. +Tip: turn this security off with the `onUntruncatedBlogPosts: 'ignore'` blog plugin option. +[webpackbar] ℹ Compiling Client +[webpackbar] ℹ Compiling Server +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +[webpackbar] ✔ Server: Compiled with some errors in 3.32s + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[9].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/markdownLoader.js??ruleSet[1].rules[9].use[1]!/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/ruitian-en.md?truncated=true': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[9].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/markdownLoader.js??ruleSet[1].rules[9].use[1]!/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/ruitian-en.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[9].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/markdownLoader.js??ruleSet[1].rules[9].use[1]!/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[9].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/markdownLoader.js??ruleSet[1].rules[9].use[1]!/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md?truncated=true': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[9].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/markdownLoader.js??ruleSet[1].rules[9].use[1]!/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/Volcano-community-co-construction-program.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[9].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/markdownLoader.js??ruleSet[1].rules[9].use[1]!/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/Volcano-community-co-construction-program.md?truncated=true': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[9].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/markdownLoader.js??ruleSet[1].rules[9].use[1]!/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/aiqiyi-en.md': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage + [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[9].use[0]!/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/markdownLoader.js??ruleSet[1].rules[9].use[1]!/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/aiqiyi-en.md?truncated=true': No serializer registered for VFileMessage + while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> Error -> VFileMessage +[webpackbar] ✔ Client: Compiled with some errors in 6.77s +[ERROR] Client bundle compiled with errors therefore further build is impossible. +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/Volcano-community-co-construction-program.md" +Cause: Unexpected character `4` (U+0034) before attribute value, expected a character that can start an attribute value, such as `"`, `'`, or `{` +Details: +{ + "column": 25, + "file": "", + "message": "Unexpected character `4` (U+0034) before attribute value, expected a character that can start an attribute value, such as `\"`, `'`, or `{`", + "line": 26, + "name": "26:25", + "place": { + "_bufferIndex": 24, + "_index": 49, + "line": 26, + "column": 25, + "offset": 605 + }, + "reason": "Unexpected character `4` (U+0034) before attribute value, expected a character that can start an attribute value, such as `\"`, `'`, or `{`", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/Volcano-community-co-construction-program.md" +Cause: Unexpected character `4` (U+0034) before attribute value, expected a character that can start an attribute value, such as `"`, `'`, or `{` +Details: +{ + "column": 25, + "file": "", + "message": "Unexpected character `4` (U+0034) before attribute value, expected a character that can start an attribute value, such as `\"`, `'`, or `{`", + "line": 26, + "name": "26:25", + "place": { + "_bufferIndex": 24, + "_index": 49, + "line": 26, + "column": 25, + "offset": 605 + }, + "reason": "Unexpected character `4` (U+0034) before attribute value, expected a character that can start an attribute value, such as `\"`, `'`, or `{`", + "ruleId": "unexpected-character", + "source": "micromark-extension-mdx-jsx", + "url": "https://github.com/micromark/micromark-extension-mdx-jsx#unexpected-character-at-expected-expect" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/aiqiyi-en.md" +Cause: Expected a closing tag for `` (182:181-182:192) before the end of `paragraph` +Details: +{ + "column": 1, + "file": "", + "message": "Expected a closing tag for `` (182:181-182:192) before the end of `paragraph`", + "line": 182, + "name": "182:1-182:254", + "place": { + "start": { + "_bufferIndex": 0, + "_index": 0, + "line": 182, + "column": 1, + "offset": 9738 + }, + "end": { + "_bufferIndex": -1, + "_index": 1, + "line": 182, + "column": 254, + "offset": 9991 + } + }, + "reason": "Expected a closing tag for `` (182:181-182:192) before the end of `paragraph`", + "ruleId": "end-tag-mismatch", + "source": "mdast-util-mdx-jsx" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/aiqiyi-en.md" +Cause: Expected a closing tag for `` (182:181-182:192) before the end of `paragraph` +Details: +{ + "column": 1, + "file": "", + "message": "Expected a closing tag for `` (182:181-182:192) before the end of `paragraph`", + "line": 182, + "name": "182:1-182:254", + "place": { + "start": { + "_bufferIndex": 0, + "_index": 0, + "line": 182, + "column": 1, + "offset": 9738 + }, + "end": { + "_bufferIndex": -1, + "_index": 1, + "line": 182, + "column": 254, + "offset": 9991 + } + }, + "reason": "Expected a closing tag for `` (182:181-182:192) before the end of `paragraph`", + "ruleId": "end-tag-mismatch", + "source": "mdast-util-mdx-jsx" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md" +Cause: Expected a closing tag for `
` (124:1-124:54) before the end of `paragraph` +Details: +{ + "column": 1, + "file": "", + "message": "Expected a closing tag for `
` (124:1-124:54) before the end of `paragraph`", + "line": 124, + "name": "124:1-124:137", + "place": { + "start": { + "_bufferIndex": 0, + "_index": 0, + "line": 124, + "column": 1, + "offset": 8410 + }, + "end": { + "_bufferIndex": -1, + "_index": 1, + "line": 124, + "column": 137, + "offset": 8546 + } + }, + "reason": "Expected a closing tag for `
` (124:1-124:54) before the end of `paragraph`", + "ruleId": "end-tag-mismatch", + "source": "mdast-util-mdx-jsx" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md" +Cause: Expected a closing tag for `
` (124:1-124:54) before the end of `paragraph` +Details: +{ + "column": 1, + "file": "", + "message": "Expected a closing tag for `
` (124:1-124:54) before the end of `paragraph`", + "line": 124, + "name": "124:1-124:137", + "place": { + "start": { + "_bufferIndex": 0, + "_index": 0, + "line": 124, + "column": 1, + "offset": 8410 + }, + "end": { + "_bufferIndex": -1, + "_index": 1, + "line": 124, + "column": 137, + "offset": 8546 + } + }, + "reason": "Expected a closing tag for `
` (124:1-124:54) before the end of `paragraph`", + "ruleId": "end-tag-mismatch", + "source": "mdast-util-mdx-jsx" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/ruitian-en.md" +Cause: Expected a closing tag for `` (164:14-164:23) before the end of `paragraph` +Details: +{ + "column": 3, + "file": "", + "message": "Expected a closing tag for `` (164:14-164:23) before the end of `paragraph`", + "line": 164, + "name": "164:3-164:178", + "place": { + "start": { + "_bufferIndex": 0, + "_index": 0, + "line": 164, + "column": 3, + "offset": 6288 + }, + "end": { + "_bufferIndex": -1, + "_index": 1, + "line": 164, + "column": 178, + "offset": 6463 + } + }, + "reason": "Expected a closing tag for `` (164:14-164:23) before the end of `paragraph`", + "ruleId": "end-tag-mismatch", + "source": "mdast-util-mdx-jsx" +} + +-------------------------- + +Error: MDX compilation failed for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/ruitian-en.md" +Cause: Expected a closing tag for `` (164:14-164:23) before the end of `paragraph` +Details: +{ + "column": 3, + "file": "", + "message": "Expected a closing tag for `` (164:14-164:23) before the end of `paragraph`", + "line": 164, + "name": "164:3-164:178", + "place": { + "start": { + "_bufferIndex": 0, + "_index": 0, + "line": 164, + "column": 3, + "offset": 6288 + }, + "end": { + "_bufferIndex": -1, + "_index": 1, + "line": 164, + "column": 178, + "offset": 6463 + } + }, + "reason": "Expected a closing tag for `` (164:14-164:23) before the end of `paragraph`", + "ruleId": "end-tag-mismatch", + "source": "mdast-util-mdx-jsx" +} diff --git a/build_log_blog_7.txt b/build_log_blog_7.txt new file mode 100644 index 00000000..86bc6a54 --- /dev/null +++ b/build_log_blog_7.txt @@ -0,0 +1,109 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[WARNING] Error: Failed to retrieve the git history for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md" because the file is not tracked by git. +[WARNING] Error: Failed to retrieve the git history for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/paddlepaddle-en.md" because the file is not tracked by git. +[WARNING] Docusaurus found blog posts without truncation markers: +- "blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md" +- "blog/paddlepaddle-en.md" +- "blog/Volcano-1.13.0-release.md" +- "blog/iflytek_case_study.md" +- "blog/Volcano-1.12.0-release.md" +- "blog/volcano-2025-security-audit.md" +- "blog/how-volcano-boosts-distributed-training-and-inference-performance.md" +- "blog/Volcano-1.11.0-release.md" +- "blog/Volcano-1.10.0-release.md" +- "blog/Volcano-1.9.0-release.md" +- "blog/Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios.md" +- "blog/Volcano-1.8.2-release.md" +- "blog/Volcano-community-co-construction-program.md" +- "blog/Volcano-1.7.0-release-en.md" +- "blog/ING_case-en.md" +- "blog/1.4 release-en.md" +- "blog/ruitian2-en.md" +- "blog/pengcheng-en.md" +- "blog/xiaohongshu-en.md" +- "blog/ruitian-en.md" +- "blog/leinao-en.md" +- "blog/hpc-en.md" +- "blog/aiqiyi-en.md" +- "blog/Quick-Start-Volcano.md" +- "blog/kube-batch-customers.md" +- "blog/kube-batch-startup.md" + +We recommend using truncation markers (`` or `{/* truncate */}`) in blog posts to create shorter previews on blog paginated lists. +Tip: turn this security off with the `onUntruncatedBlogPosts: 'ignore'` blog plugin option. +[webpackbar] ℹ Compiling Client +[webpackbar] ℹ Compiling Server +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +[webpackbar] ✔ Server: Compiled successfully in 2.70s +[webpackbar] ✔ Client: Compiled successfully in 6.25s +[WARNING] Docusaurus found broken links! + +Please check the pages of your site in the list below, and make sure you don't reference any path that does not exist. +Note: it's possible to ignore broken links with the 'onBrokenLinks' Docusaurus configuration, and let the build pass. + +Exhaustive list of all broken links found: +- Broken link on source page path = /docs/v1-11-0/gpu_virtualization: + -> linking to ../../installer/README.md (resolved as: /installer/README.md) + +[WARNING] Docusaurus found broken anchors! + +Please check the pages of your site in the list below, and make sure you don't reference any anchor that does not exist. +Note: it's possible to ignore broken anchors with the 'onBrokenAnchors' Docusaurus configuration, and let the build pass. + +Exhaustive list of all broken anchors found: +- Broken anchor on source page path = /docs/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) +- Broken anchor on source page path = /docs/v1-12-0/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/v1-12-0/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) + +[SUCCESS] Generated static files in "build". +[INFO] Use `npm run serve` command to test your build locally. diff --git a/build_log_blog_fix_svg.txt b/build_log_blog_fix_svg.txt new file mode 100644 index 00000000..71db55e3 --- /dev/null +++ b/build_log_blog_fix_svg.txt @@ -0,0 +1,151 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[WARNING] Error: Failed to retrieve the git history for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/paddlepaddle-en.md" because the file is not tracked by git. +[WARNING] Error: Failed to retrieve the git history for file "/Users/kushkumargupta/Documents/Volcano/volcano-website/blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md" because the file is not tracked by git. +[WARNING] Docusaurus found blog posts without truncation markers: +- "blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md" +- "blog/paddlepaddle-en.md" +- "blog/Volcano-1.13.0-release.md" +- "blog/iflytek_case_study.md" +- "blog/Volcano-1.12.0-release.md" +- "blog/volcano-2025-security-audit.md" +- "blog/how-volcano-boosts-distributed-training-and-inference-performance.md" +- "blog/Volcano-1.11.0-release.md" +- "blog/Volcano-1.10.0-release.md" +- "blog/Volcano-1.9.0-release.md" +- "blog/Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios.md" +- "blog/Volcano-1.8.2-release.md" +- "blog/Volcano-community-co-construction-program.md" +- "blog/Volcano-1.7.0-release-en.md" +- "blog/ING_case-en.md" +- "blog/1.4 release-en.md" +- "blog/ruitian2-en.md" +- "blog/pengcheng-en.md" +- "blog/xiaohongshu-en.md" +- "blog/ruitian-en.md" +- "blog/leinao-en.md" +- "blog/hpc-en.md" +- "blog/aiqiyi-en.md" +- "blog/Quick-Start-Volcano.md" +- "blog/kube-batch-customers.md" +- "blog/kube-batch-startup.md" + +We recommend using truncation markers (`` or `{/* truncate */}`) in blog posts to create shorter previews on blog paginated lists. +Tip: turn this security off with the `onUntruncatedBlogPosts: 'ignore'` blog plugin option. +[webpackbar] ℹ Compiling Client +[webpackbar] ℹ Compiling Server +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +[webpackbar] ✔ Server: Compiled successfully in 2.72s +[webpackbar] ✔ Client: Compiled successfully in 3.60s +[WARNING] Docusaurus found broken links! + +Please check the pages of your site in the list below, and make sure you don't reference any path that does not exist. +Note: it's possible to ignore broken links with the 'onBrokenLinks' Docusaurus configuration, and let the build pass. + +Exhaustive list of all broken links found: +- Broken link on source page path = /docs/v1-11-0/gpu_virtualization: + -> linking to ../../installer/README.md (resolved as: /installer/README.md) + +[WARNING] Docusaurus found broken anchors! + +Please check the pages of your site in the list below, and make sure you don't reference any anchor that does not exist. +Note: it's possible to ignore broken anchors with the 'onBrokenAnchors' Docusaurus configuration, and let the build pass. + +Exhaustive list of all broken anchors found: +- Broken anchor on source page path = /docs/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) +- Broken anchor on source page path = /docs/v1-12-0/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/v1-12-0/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) + +[SUCCESS] Generated static files in "build". +[INFO] Use `npm run serve` command to test your build locally. diff --git a/build_log_debug_blog_2.txt b/build_log_debug_blog_2.txt new file mode 100644 index 00000000..8eff64dd --- /dev/null +++ b/build_log_debug_blog_2.txt @@ -0,0 +1,60 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. + +[ERROR] Error: Unable to build website for locale en. + at tryToBuildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:83:15) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:35:9 + at async mapAsyncSequential (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/utils/lib/jsUtils.js:21:24) + at async Command.build (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:34:5) + at async Promise.all (index 0) + at async runCLI (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/cli.js:56:5) + at async file:///Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 { + [cause]: Error: Processing of blog source file path=introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md failed. + at doProcessBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:271:19) + at async Promise.all (index 16) + at async generateBlogPosts (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:274:24) + at async Object.loadContent (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/index.js:155:29) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:40:23 + at async Promise.all (index 2) + at async /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:146:25 + at async loadSite (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/server/site.js:155:45) + at async buildLocale (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/buildLocale.js:33:18) + at async runBuildLocaleTask (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/core/lib/commands/build/build.js:98:5) { + [cause]: Error: Blog author with key "Volcano Engine Team" not found in the authors map file. + Valid author keys are: + - volcano + - Volcano + - Adam Korczynski + - Hao Li/Senior R&D Engineer in iQIYI + - Zhe Jin/Senior Engineer in HUAWEI CLOUD + - Junmao Li/System Architect in Pengcheng Laboratory + - Thor-wl + - Yunzhi Cheng/Ruitian Tech-Leader + - Yunzhi Cheng/Tech-Leader in Ruitian + - Jian Zhu/Senior Architect in Leinao OS + - Yi Guo/Tech-Leader + - Zhengyu Xu/R&D in Ruitian + - Ada Logics + - Xavier Chang + - Huawei and Volcano maintainer + at getAuthorsMapAuthor (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:87:23) + at toAuthor (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:100:16) + at Array.map () + at getFrontMatterAuthors (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:58:42) + at getBlogPostAuthors (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/authors.js:114:21) + at processBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:215:54) + at async doProcessBlogSourceFile (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:268:20) + at async Promise.all (index 16) + at async generateBlogPosts (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/blogUtils.js:274:24) + at async Object.loadContent (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/@docusaurus/plugin-content-blog/lib/index.js:155:29) + } +} +[INFO] Docusaurus version: 3.9.2 +Node version: v22.21.0 diff --git a/build_log_debug_blog_3.txt b/build_log_debug_blog_3.txt new file mode 100644 index 00000000..d353256e --- /dev/null +++ b/build_log_debug_blog_3.txt @@ -0,0 +1,237 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[WARNING] Docusaurus found blog posts without truncation markers: +- "blog/Volcano-1.13.0-release.md" +- "blog/iflytek_case_study.md" +- "blog/Volcano-1.12.0-release.md" +- "blog/volcano-2025-security-audit.md" +- "blog/how-volcano-boosts-distributed-training-and-inference-performance.md" +- "blog/Volcano-1.11.0-release.md" +- "blog/Volcano-1.10.0-release.md" +- "blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md" +- "blog/Volcano-1.9.0-release.md" +- "blog/Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios.md" +- "blog/Volcano-1.8.2-release.md" +- "blog/Volcano-community-co-construction-program.md" +- "blog/Volcano-1.7.0-release-en.md" +- "blog/ING_case-en.md" +- "blog/1.4-release-en.md" +- "blog/ruitian2-en.md" +- "blog/pengcheng-en.md" +- "blog/xiaohongshu-en.md" +- "blog/ruitian-en.md" +- "blog/leinao-en.md" +- "blog/hpc-en.md" +- "blog/aiqiyi-en.md" +- "blog/paddlepaddle-en.md" +- "blog/Quick-Start-Volcano.md" +- "blog/kube-batch-customers.md" +- "blog/kube-batch-startup.md" + +We recommend using truncation markers (`` or `{/* truncate */}`) in blog posts to create shorter previews on blog paginated lists. +Tip: turn this security off with the `onUntruncatedBlogPosts: 'ignore'` blog plugin option. +[webpackbar] ℹ Compiling Client +[webpackbar] ℹ Compiling Server +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +[WARNING] Markdown link with URL `../../installer/README.md` in source file "docs/v1-11-0/gpu_virtualization.md" (26:10) couldn't be resolved. +Make sure it references a local Markdown file that exists within the current plugin. +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +[WARNING] Markdown link with URL `../../installer/README.md` in source file "docs/v1-11-0/gpu_virtualization.md" (26:10) couldn't be resolved. +Make sure it references a local Markdown file that exists within the current plugin. +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/descheduler/descheduler_EN.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +TypeError: unsupported file type: undefined + at imageSize (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1013:9) + at /Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1040:16 + at async Promise.allSettled (index 0) + at async processQueue (/Users/kushkumargupta/Documents/Volcano/volcano-website/node_modules/image-size/dist/fromFile.cjs:1047:3) +[WARNING] The image at "/Users/kushkumargupta/Documents/Volcano/volcano-website/static/img/multi-cluster/volcano_global_design.svg" can't be read correctly. Please ensure it's a valid image. +unsupported file type: undefined +[webpackbar] ✔ Server: Compiled successfully in 12.41s +[webpackbar] ✔ Client: Compiled successfully in 16.30s +[WARNING] Docusaurus found broken links! + +Please check the pages of your site in the list below, and make sure you don't reference any path that does not exist. +Note: it's possible to ignore broken links with the 'onBrokenLinks' Docusaurus configuration, and let the build pass. + +Exhaustive list of all broken links found: +- Broken link on source page path = /docs/v1-11-0/gpu_virtualization: + -> linking to ../../installer/README.md (resolved as: /installer/README.md) + +[WARNING] Docusaurus found broken anchors! + +Please check the pages of your site in the list below, and make sure you don't reference any anchor that does not exist. +Note: it's possible to ignore broken anchors with the 'onBrokenAnchors' Docusaurus configuration, and let the build pass. + +Exhaustive list of all broken anchors found: +- Broken anchor on source page path = /docs/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) +- Broken anchor on source page path = /docs/v1-12-0/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/v1-12-0/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) + +[SUCCESS] Generated static files in "build". +[INFO] Use `npm run serve` command to test your build locally. diff --git a/build_log_debug_blog_4.txt b/build_log_debug_blog_4.txt new file mode 100644 index 00000000..074a9fc8 --- /dev/null +++ b/build_log_debug_blog_4.txt @@ -0,0 +1,65 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[WARNING] Docusaurus found blog posts without truncation markers: +- "blog/Volcano-1.13.0-release.md" +- "blog/iflytek_case_study.md" +- "blog/Volcano-1.12.0-release.md" +- "blog/volcano-2025-security-audit.md" +- "blog/how-volcano-boosts-distributed-training-and-inference-performance.md" +- "blog/Volcano-1.11.0-release.md" +- "blog/Volcano-1.10.0-release.md" +- "blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md" +- "blog/Volcano-1.9.0-release.md" +- "blog/Meet Cloud Native Batch Computing with Volcano in AI & Big Data Scenarios.md" +- "blog/Volcano-1.8.2-release.md" +- "blog/Volcano-community-co-construction-program.md" +- "blog/Volcano-1.7.0-release-en.md" +- "blog/ING_case-en.md" +- "blog/1.4-release-en.md" +- "blog/ruitian2-en.md" +- "blog/pengcheng-en.md" +- "blog/xiaohongshu-en.md" +- "blog/ruitian-en.md" +- "blog/leinao-en.md" +- "blog/hpc-en.md" +- "blog/aiqiyi-en.md" +- "blog/paddlepaddle-en.md" +- "blog/Quick-Start-Volcano.md" +- "blog/kube-batch-customers.md" +- "blog/kube-batch-startup.md" + +We recommend using truncation markers (`` or `{/* truncate */}`) in blog posts to create shorter previews on blog paginated lists. +Tip: turn this security off with the `onUntruncatedBlogPosts: 'ignore'` blog plugin option. +[webpackbar] ℹ Compiling Client +[webpackbar] ℹ Compiling Server +[webpackbar] ✔ Server: Compiled successfully in 1.88s +[webpackbar] ✔ Client: Compiled successfully in 3.39s +[WARNING] Docusaurus found broken links! + +Please check the pages of your site in the list below, and make sure you don't reference any path that does not exist. +Note: it's possible to ignore broken links with the 'onBrokenLinks' Docusaurus configuration, and let the build pass. + +Exhaustive list of all broken links found: +- Broken link on source page path = /docs/v1-11-0/gpu_virtualization: + -> linking to ../../installer/README.md (resolved as: /installer/README.md) + +[WARNING] Docusaurus found broken anchors! + +Please check the pages of your site in the list below, and make sure you don't reference any anchor that does not exist. +Note: it's possible to ignore broken anchors with the 'onBrokenAnchors' Docusaurus configuration, and let the build pass. + +Exhaustive list of all broken anchors found: +- Broken anchor on source page path = /docs/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) +- Broken anchor on source page path = /docs/v1-12-0/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/v1-12-0/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) + +[SUCCESS] Generated static files in "build". +[INFO] Use `npm run serve` command to test your build locally. diff --git a/build_log_debug_blog_5.txt b/build_log_debug_blog_5.txt new file mode 100644 index 00000000..4a8fb0e9 --- /dev/null +++ b/build_log_debug_blog_5.txt @@ -0,0 +1,65 @@ + +> temp-docusaurus@0.0.0 build +> docusaurus build + +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[INFO] [en] Creating an optimized production build... +[WARNING] The `siteConfig.onBrokenMarkdownLinks` config option is deprecated and will be removed in Docusaurus v4. +Please migrate and move this option to `siteConfig.markdown.hooks.onBrokenMarkdownLinks` instead. +[WARNING] Docusaurus found blog posts without truncation markers: +- "blog/Volcano-1.13.0-release.md" +- "blog/iflytek_case_study.md" +- "blog/Volcano-1.12.0-release.md" +- "blog/volcano-2025-security-audit.md" +- "blog/how-volcano-boosts-distributed-training-and-inference-performance.md" +- "blog/Volcano-1.11.0-release.md" +- "blog/Volcano-1.10.0-release.md" +- "blog/introducing-kthena-redefining-llm-inference-for-the-cloud-native-era.md" +- "blog/Volcano-1.9.0-release.md" +- "blog/Volcano-1.8.2-release.md" +- "blog/Volcano-community-co-construction-program.md" +- "blog/Volcano-1.7.0-release-en.md" +- "blog/ING_case-en.md" +- "blog/1.4-release-en.md" +- "blog/meet-cloud-native-batch-computing-with-volcano-in-ai-big-data-scenarios.md" +- "blog/ruitian2-en.md" +- "blog/pengcheng-en.md" +- "blog/xiaohongshu-en.md" +- "blog/ruitian-en.md" +- "blog/leinao-en.md" +- "blog/hpc-en.md" +- "blog/aiqiyi-en.md" +- "blog/paddlepaddle-en.md" +- "blog/Quick-Start-Volcano.md" +- "blog/kube-batch-customers.md" +- "blog/kube-batch-startup.md" + +We recommend using truncation markers (`` or `{/* truncate */}`) in blog posts to create shorter previews on blog paginated lists. +Tip: turn this security off with the `onUntruncatedBlogPosts: 'ignore'` blog plugin option. +[webpackbar] ℹ Compiling Client +[webpackbar] ℹ Compiling Server +[webpackbar] ✔ Server: Compiled successfully in 1.59s +[webpackbar] ✔ Client: Compiled successfully in 3.41s +[WARNING] Docusaurus found broken links! + +Please check the pages of your site in the list below, and make sure you don't reference any path that does not exist. +Note: it's possible to ignore broken links with the 'onBrokenLinks' Docusaurus configuration, and let the build pass. + +Exhaustive list of all broken links found: +- Broken link on source page path = /docs/v1-11-0/gpu_virtualization: + -> linking to ../../installer/README.md (resolved as: /installer/README.md) + +[WARNING] Docusaurus found broken anchors! + +Please check the pages of your site in the list below, and make sure you don't reference any anchor that does not exist. +Note: it's possible to ignore broken anchors with the 'onBrokenAnchors' Docusaurus configuration, and let the build pass. + +Exhaustive list of all broken anchors found: +- Broken anchor on source page path = /docs/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) +- Broken anchor on source page path = /docs/v1-12-0/unified_scheduling: + -> linking to #2-1-2-enable-dra-dynamic-resource-allocation-in-volcano (resolved as: /docs/v1-12-0/unified_scheduling#2-1-2-enable-dra-dynamic-resource-allocation-in-volcano) + +[SUCCESS] Generated static files in "build". +[INFO] Use `npm run serve` command to test your build locally. diff --git a/content/en/author/volcano/_index.md b/content/en/author/volcano/_index.md deleted file mode 100644 index 630afd69..00000000 --- a/content/en/author/volcano/_index.md +++ /dev/null @@ -1,79 +0,0 @@ -+++ -# Display name -name = "Volcano" - -# Is this the primary user of the site? -superuser = true - -# Organizations/Affiliations -# Separate multiple entries with a comma, using the form: `[ {name="Org1", url=""}, {name="Org2", url=""} ]`. -organizations = [] - -# Short bio (displayed in user profile at end of posts) -# bio = "My research interests include distributed robotics, mobile computing and programmable matter." - -# Enter email to display Gravatar (if Gravatar enabled in Config) -email = "" - -# List (academic) interests or hobbies -interests = [] - -[[social]] - icon = "/img/icon_email.svg" - icon_pack = "fas" - link = "#contact" # For a direct email link, use "mailto:test@example.org". - - - - [[social]] - icon = "/img/icon_github.svg" - icon_pack = "fab" - link = "https://github.com/volcano-sh/volcano" - -#[[social]] - # icon = "/img/google-scholar" - # icon_pack = "ai" - # link = "https://scholar.google.co.uk/citations?user=sIwtMXoAAAAJ" - -[[social]] - icon = "/img/icon_twitter.svg" - icon_pack = "fab" - link = "https://x.com/volcano_sh" - -+++ -
-
- -

Unified Scheduling

-

Supports integrated job scheduling for both Kubernetes native workloads and mainstream computing frameworks (such as TensorFlow, Spark, PyTorch, Ray, Flink, etc.).

- - -

Queue Management

-

Provides multi-level queue management capabilities, enabling fine-grained resource quota control and task priority scheduling.

- - -

Heterogeneous Device Support

-

Efficiently schedules heterogeneous devices like GPU and NPU, fully unleashing hardware computing potential.

- - -

Network Topology Aware Scheduling

-

Greatly enhancing model training efficiency in AI distributed training scenarios.

- - -

Multi-cluster Scheduling

-

Supports cross cluster job scheduling, improving resource pool management capabilities and achieving large scale load balancing.

- - -

Online and Offline Workloads Colocation

-

Enables online and offline workloads colocation, improving cluster resource utilization through intelligent scheduling strategies.

- - -

Load Aware Descheduling

-

Optimizing cluster load distribution and enhancing system stability.

- - -

Multiple Scheduling Policies

-

Supports various scheduling strategies such as Gang scheduling, Fair-Share, Binpack, DeviceShare, NUMA-aware scheduling, Task Topology, etc.

- -
-
diff --git a/content/en/author/volcano/avatar.png b/content/en/author/volcano/avatar.png deleted file mode 100644 index 7c4c8a734321bc8ebd0e5f1da39d525c85288fac..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 5677 zcma)AXH-+!w+|%{dQqtX0?L3CK}Km22pvS)D8-00X+a2Oq)G1}B`5+q5D|%jBS?!B z=|vL+=}5U0=?H{QKoEEreE+xJhxg&Fb=O&E|IY4b?|s+3iDoAHY|H}85D0|L&_L%e z2!uu!+%Llz!1bDV(-s7BZre~t^Hw0u%J7hnc;m*W4R#t+qf3XAWF$HG$Cuzl-!)jI zs62gKkEjQXl_qEr%8x3z^-jutmyt5v1yKuKC|%lmxO=DY$N521#Y2+D(FQL3*UY4b zZT%y&ms+$wv(B|qJq_zz)cF;Q7}}j!`f|S%e`wy}_wy_5Ki|QiOA-bM!EYXn(7aH2 z4n%ammyz(OicLXkzHUjk*x5o6(MS&i)7q#U#p&}oACoFzQ1^Ix$K^$TsLI_oeP!n? zS1#s(^A|V;lYn!xSGYu`Zr#P}N}BN+AQ-;6S#heWg~Z%^5XzH>x~ob1-0dr9x!+L` z^+tbEwC4VTw-1pVnGNb(f-pHHPf?h)3LALo4YZ3F6LJ`8-iDVGED~beD-S2(e5>^NO#G|?U5U`QG43L z`9F?NxCPTtsvd9eYKDkIe@JpL#jdE4x%#>uJ zu6WcR(!^AD-;@d6ufvYf5c05N=V6|?1=jnW%k+Nv3iN3V-4vqe(W{Fp*B%~I+In9> z)elrP#}im-W?RRq`WXV*3cBk}%u<|T2qBv);5=yq)!y5C;3bjDb2a41FM{})tH6oBNtJ9NpiPou5Md?|jjU;TCbtk#S1uHKfeB;#LOr7TB;e#NgG*O9 zBwHNDdMn0xQ+YyX6i{*Qpa&{Z<)IE0bv;tMuNvlJyB<~vlt5nH(`B|l;uE% zJinAZ^ZjpgLg4jpKu$4Q5IgXBB&dS;aNo674lcV-b&jDOOo-aPcQEU}B)=5jeiSkA z9#3iTAub}bz%nsB>!i}^UwM;G#Nh?(Q}6rz@B(`$+l4Ntq|)?G>Jrfw*sw-4XZ?+7vXiE2fDG$Sa;R@dqwd^Rv@qfjF`ZScYb}B(maGfe{klZJn!tUg**t3Yof|loe^

qgnVPnJ?cHFF zRGz~{Db2eECu|+zt{EVk(J340M4S8Y3sH|6JO#|ihDFpz1<>pimoQ2-*{z$ivZu>` z`h4Ajs-}>RCWFAp3(?Z?kN7>9oY4C-1I=fhBKRr%w16y-h?C!(xqp-Nn^Hkog(Rgkp?9y7Zq%OhQ1am z!D0xIa>t)yje)-ZFEb!ZfTk+4T_oqryg<%ETtP+h&F*>)2; zl4iIx6zhrVgNtQ2KIxY8W6?6+o!;WLR48K`6@pv;UOz6F`mF>DI_V{z2DLr11vbO) z>jX!1Qdh%`0(w?cyVs9epR3ql@+qL=+GDnH&06VTfg=9a48;jyL`8kDHQ3UBD5D!2 z63t$Qp=fU-wD^y+wMXfkDwV=2j6oGqGkZs^{N9D#UgEA4nu*eSpKZ^j#_`5bR?9n^ zKo>PnfAS5O8riMvY%rk5+zD)zf=p?=L}`FB%)kdX(e?%JD=FXWX65db+adK2u+pB; zD+mK@0Nw04AcL|s!eA|Aa_4`zx<55Pv~#Qttv&3OQxB`8(_aoy0t>&HoBU-M+b@hT zb+fAM7JcD3#lJ+~O*bl%XVOCLchd;Oc+I%gN2)ew(@;3lM_B;VH;3P=gd*~H4gd)M z5{jRe3{kDFM>!z@(g&adx5$e?@)hy;t#6c_Tut;;xk!P83$2# z{C>a|QI4lDQ2q#ssf3c*8Q$qq)YU61@cDE0)uYkQ25i46=OtVfjluvsXo$hhH8_$? zg$}P~HZE2|Mw;zE6gHDc{*`|d;jC8R2zmS8)Q!U*im14KCUC&9_3NF;I1jJphPLt1 z_}(vt_jrbdoRpPx)65-*@7~U^{87eGxa!_auzfA&Kn*xdv5Z*RK4yZFQ;)5TN3dh; zX~V0|#8odkr@}RG>M0ft+9IMna++9iqaJsCUo4L6 zcsUMEsM|<@h8qA_Qzsm&Sw|@x2KdSfAviIcEvvZ~;KBUWl%|!Be=t zd;?t7;vOpR#}WI%2@^o^z?pRd6n&PXhXrQHi)01@{(~#5*3&gOrhJe(g3A+5Z4eE@ z?)oWv1jyUCPV#y+HnUd|lGYT-QOIEyF8U6FvG zNbWnj>mEvZEC9>c?Z(>18P|ix@S^wMA&wCgkF}jTt%H1HBw%#_VH>K201F+dz+_QR zST_}GDoxZV+6f1^6CT52DgCDttL?bxB|(HF;R(v;$)x zb^9mLIf8XV3@#PO7f;Jk*E-l6!9GNJxCTd4*?3UPq$mCQZeUFsb0mOK^t@FH+1+*6E}~1PG8^U{LH4f< z0Fy72KlfQ<1TQeEBUyEN{T|)K zPoFJN{q)wv=ii(7rCSB6gKsY#(7p?9Z13p$y1mN=2Q)DKDtJQwtaI|M_Ka6;*Gi4w zatjmlo`6ayiko61Plx-ZR;Z1w<8TJh#oY*H6MaZEi~|F+*DWn4>C7 za%XNZ>uk%zh;OUpc;RE#um?WvudlMqQ+Q&-I-?lH8I90q+{h^%h{EH<@}f^oIXxb? z(6Neg=Up?B2-z|3ugVltMI|)R5_`ut9&Uyli2;3ZbvQF)*p<|j=G9#aXpUy;TOvCJZn-5&++ua>a5h8J&|C3cD1vp1Xw*ul z9A>_DhA{Viq0XQtc=pq;#Uh272~ke&;=z|Ci!U^)uzK~Y{<0w-$Vy)`ZtZKyZO>G2 z`G-iK9#Y^t7~CTCs8M@9u$uTt6NArqMNa)QdOj>0>dg2YA=~N`F=bh>6~4P>)bU!E za2IVq>%GgH_mL?7%RMM)ur?G^L$pCNPy1pV_69dMYZMOpd_ib^KtFh=&M0Z;scGKQ z;oDQ}W%YVt%j?)FtB#I;CvxRvvoeiXcoEt@HN?G)<-g#0FDFrhU%QAT9*XYkxhd)Tox`DfCX0ns!I}^{bymVew z1q{6tZ9xgKr3rW=mgsj2m!Xxoa5`MFg|PMDCcs$yVIkd~?Ty9OInE|gWxFCPE6018 zAy$fm=%=Co{6YKG4P%SUmRXC z#nCko<+;{cTD+3P1YXk^M*h%f%RQUeGQ&>1PoG5Z!8#y+@&N@xG6wp?Tpz1uR45k%jlcjI+#HY%T$F!g#eI7`d5jLe`Q>faP>xMa60t@+czI8)Bn z*YO4|5?k>G_nSm%rNcP9LR3NyDFlltM6N_}Hjz(8rhBqv=%ORpMNG()d~^ZhIek1+ zlYbZ6CxEo;Fbu5C_G%@it_&nwnBicapP$G44JBsW$xZ)>zcKFEZDb|y$=-vDFLEYM zJ#M}X?{(#PyHc7MPn>qBPlo?u9{_#`5)b$tCD$)bDU}zy8^{^iz3E~^rDHoZxc_o(PYZafQxf-(- z?}B^>yKRx&eAJ?}Xo+Lx3p?i<6;1qOsi(})@lNUe-F%oB`7f_*_ic^~I`$0_M5)go znAMp5L{VRo$~pEc&6G@6y9bP4m~Jx#agYlgS&T*2!?rV!GM~q!mbzMoCa$C3UTvG( zRq1n=Z=by$FY2J^+M&0qSM#Q1un)F`j)HXZJE|9nWMO&8ky)m*Llg~g{t(E6*4Bn&udujQ|x?9FsJDTMVOn*(I3HSxRA~p5R=1&!R^yG-F1m&)4MIgoV9#8c%yX z&{0#x=3(vaNsobx7Hbj&i^jD0O-8oy%Z4l!Bp4}#5##es?wfgJJfe7q9^*r`W~$Vz z(RY$Bi1j3M^-asbRTbOu6)jsygirRbwzk5h9rpiD;OpVpQY(WV&Ua2aQHE^$RKw%h zXPIlPEgbF}h_&*_D^GiGwvFvR;mJ%O|6LbN6niRQJZ^|+Oe9vCWWSahl52h0WR9C9 zRtsgv7&BwlSrOHL60V9plX4b=o%_|+)4h9PUf#H26);S?_c4i*{o#k?^t0J>c*0;( z&GuKq2Y<3mT_lm=sY9)#w=5<$`T$E-yF{0e(`!S3#`6%$k|-mXzUk!2(a~A=m3`Md zm#0a^<3!$N~4jL!MBpFDz4;4Rn zq4=C{EVu11H_na7$%j?xVJsNkd6f6Albc!aW+%{U{dx5P>;gJJNXIdV4pW9ZGirlu z%Ml1s7`=P`Pi)RPQa&Npvkr|(DbD?n-jBXvf8pudG8shYK+re0%QIK6;`5`4?#o74 z%2&PlnUPud(P4)(Px|RNS{Xbp{{Vi__Wf&Gxx#;lAGm8aCGoC3B~J1E&h1N`oS_Oc z8h_4rRw*yrX``ExHxa_G;K-QLh~bAEFTUbShT=L5jjr`bZHCY-!_&A3*nw74^vX^D zYx33qF62Zkde`CK(IFDss@GIxQ>3`R!gra2rH_S%?d)sgh(s)c|5eW=_Mp!6zFi*8 zH$2N(Pk3zkd*=9uJ}cUvCtvWltup+DL;bP~YxO2{FP4N1!eAebzs#r5$hMYbGgxZ; zWmjfwk2D`JE)+!64#}lTn0wz8%h|IpI^r0OIKC)&_G!}5!w`dm3!FWl3rT6|jYbna z711Gs@l1PRoj;b>JHi(@n*^1x-r~1b^D)@5>W$)rkgyGP)8)@8f|IQCFE@?RO4zZz j>)&Z+_et))_OCC6(RSvdv;mq)m#-5=Kcxmh6PFD^j8iN_wN$QefYc z`Qp@-t6(Y9I37OpE@jtoY?kvj^=j)joP6;qLDG@eshmX~gY>u@@70<<;%b#SZ~#tnNNO%;7`$n;Jzf~YbCGTr)XlR_ZxaeCE3a0SB?mtwyjKw5@_uDlF5ZCNIK?&-Yp zw{fVs$Q4SLZCD_PR`B@tz|T|*Ka_fLj!U$R*C9dt3O-VMQTdH}slULM6zEvCG2rhH zlqDVotoSifuCu#m`VbMYuqRli zs~x~~Cxx1$;zK@NUu+C{YHWdy{;_W@sppnwS_HscL?%-_dvopC<>|9z+3DLMN17wA zsh^MX6hb(Z;BS3np7yL^hB2aFQid8HZ@O3i{?+V0bF+Dz;)`g#C&cIY7t{R(q8aAn z+6}aJe+OCeFMm~(&elzhd8t_|e$psnbeVvTz?bW3whjhccZ1*D`>+F;o%vB9U!x{| z-Nl#O#a#znDca)bK_7K6udEBQo=QLJ2!FH=eQ)-!y8_L_Jdju^^fKq`@~oy;xI8q5 z8h&#&xlh3o<#D4p502MRK@%R?#_PxQ~V_WDC5Av--kRs82XhLic7qBkUg&T03 zunre|)@D1C{`WWQKm!BO*ggP%5@ZDKnYwphhuHmz2Y*ocmBHUrDA|R-X{~xYNAgIc z0GJZGc`ya%OkGwRqTiMh^B+M(FpPF$L(yv1pFn?jCld(H4n$Mzne!FOFCB(DEap<; zyTf^Z3eDFM0Yv9n+LnFBmH}xRjaHX&^Re9)R?TcF%f#MPA}`^LueB(XyH2f(U3s90 z{y+iiCJ7vLdb%-vCAP0r69&Qv-IH|Lj!@3-gJJd^4-SaJr`#y5L@T6G{Xb282~F=S zhXYoRI^5U}BR%XGymi>>!`l@xw-%Zl4Ok?R^5@hxUyAE@%%pj4#<~bktr};&$jm8=!hq%q#z|2|lvC~7$hp6mJNegG4;hvQ$-$w~ar>1tT_?@~b) zrdafZn}?OmPDkO_!C;+j5wXrYj%{+$r>Zs7X@sP!8bFx}^-6IXy>~ty%oS7-<5eEt zECg;rx(NTAJ?tw2QTVA3vLtj-WETkG`Q&4aXxq>I!Iww|%cu3K1Stb5`K9!B&xf=N zB;4kkI&Fo-uFmItVx1F1W0GunP|I21QgHM$!2=8Eubp+J=WG;PX*Lw#d|QxTgWqna zp`=VSPan>vN1Sm+r6w^B+O|IQOfr)?SDv48yhb6aLBclQSHk8d?_k3aQ|KJU-G-9_ zyvY?g#)vFK!c--6pn67zRyqM{|BMwSSu_sG7kctqD@r66aLJ75wq)*GCg@y+;%FOf z=Z$t{XXDm0fh(<AySA2+u5NO(-jmyFnOt|A>mio_jPw!XX^wr>i+_u`oS*oaflzV@v$C)`+D7xDq%UY7*wdwa&pN2y@s=U2pjiC|J%mr9MRVB(7CO$Umm_TLIik*kU=mpBHwyP$x5lrw8x7Dn*AHI zPEwkaE6uR^EDh#Wni@$bDNQ7zkus{s?WL2T?`eaDg9~} zEl@7B2{d%FrMe`s@*c*-;?b%GHW7k}EDiX0)U=%$1VgH1$*%;emmmeJdp7tUP$Xs{m^pEfHwS8;^12{q!omd~F#I>Zc95L6Mj z(U(=%)TqMSCw1#oK$XV|D%J4@zny+ZGx2)eylaxIz4ztPH*6-OhH^!W1b+-X4#$P+ z%IyL}T5)SZvnOi$Ra?%3scS_?%^<^_JiM|3;`evD(Sw0rF&Ds{mj^$+aCWS{lDp8# z%&rQz(#;>1&5+^5*1aPF6%x;Fvs~6PGMzupsUf0nQF5(<3M?*l&UQCF4NxRSYt0PL z7xg2fVWL4%8As9r(%%^tKW6 z*(~h-)0nm>;sf)4BMP}4Xi9sLshYHRR$GErV(!pUV(Ga4`%%=I{gy2A6#!J6yE;9_ zs-pM9+&XKzujakVv)W~8^|HRZ1WN@pzN_pl@LRm*8}?k*#bQR8EAlx$z99PRcI~Oe zrZCVBy3TLch+8tkf}lJ|HK9WMI`xm4m#gcM<936#3Zp6Zg(Kx1H-z{Ms0Thh^myLX z5@c}yS6YLg!i|6_C#yD9QhOLvxX4x10skr_DSoMNLjxfYf2N|`Ggv+RMah;p7>DJ? zi8;n0F)|6$U09a6avZ6&aVhcBxi9;a1RV`J7i<+Q)V36{Y?eLB_+X_hAdY zEDp4ns+PY*kv^UnTzLWnB)bRbrkM)J>QGVcqO{E73ViXFPKgwEG3}|vn%4!Hc>%Ok zS|65rPhW1}z)6KpeCrl93L0A2>%tAVTfifsX=Elkj0Dktd>41 z!|BaPFc^O|k~Z!0;A-$u3w}gjC9lfhd7l!B{iTeQSqB}CSa0boCz;UhGzyOT4=S!p z!V@dE3XHEIX*14L*Q6EpKQx2Y%43D1F#-OJT+jV+!+mgwQRFnK~|`aZeF znk%_({L0={@Q z39u_y87>?pKlNlZ^7HmfND&KNbHT#KzyuWkl?ih@k0PsH%o{g>5elJngZUs!p`von z!3{lv;`1enteIZkn(S-{hHd|G0I&DJjcn^hRAlvdjUO&b(G=*M7;6FswI9icR=xpU za=SyeHDr5?ahHK6vV!uY)zP;oy5usQnN0%h?>5(mB#tijj{k=53P#PzoVwjeS@Dxe zuWj(3^Oe~$G-`F`N9&GSPMs4Uah@Zj*G?Ji*56p&2A8Hn+n(54d*OUs6GtPYr1bbc z*psJ$f-DV!Zv9RLb6^LuuY?9I>5W86ALpRvITYEujGc^ETgh>?AwnCvd9GtVoW_(R zWMK=X7W6hqKz|ej742Bb_Cy4Sjv=EZ3(t*Z@WJXgLNK;#uz4_#QS42HJX~_sCf^ze zG$qvbxbt5yPyArIEuAl{nYM^d@6y6fe^hfoiV;s28D`d@V!ljT^m0Tr%e=N^_2VY| z?DVEtKW$| zc2b3Zk2d~|1-wHFI`k3BzVf4_wkN>;nH-h}GYU+}T|ntlZ{Jxkfj^;Y52J1e7FdGV z7AuiURs9jUnsGZj!WWP%=>ozCBX#gVtp58$H*fEok!806RAQ<)C?V>60xv&qF^Vy4 zUJH*3Vnp!qI|tTJ=_>dH-Rd8;9^}N^;cd#O&Lf8!SU;9lyt@wtFd~rbR*fOU`N*Fu zkq|`Cgn3<;pnrVa;+)h}ky9)k5tygZ+$R>jlI1ejwjp^R=X?^ea69lx`#&c{(2Lm_ zgt)ZD+Ifh-_pP$Z{vEk`+4*#ur7qRfg&MPHKlFW{<@8@0*Gh)z>f(MZYn;NU383}) z7O>QRi$HFjQayJlFvf&SaB%qB{vB|=EO#+sO@}(+8q>1qHy_TwlUnC{MM>@NkfUv= z`~@(q{hu#ec+m>qSpKJB7jqUN`G;bpbd&0udIzq4F(9H7Y+V-BweD?f!75Lv+`2(~ z(Y*(}05;)lnmBc677=Iq0f!ui*`+g5MUZ_?qe8;}@SiS&GeZ5u(WtQw(g^ zY|Ml)zUUwP=1liij8H!EqmxGz)iHr8&)sp2q^ zR&f2sMcXQOd);C}bWj};e2lnwwa}y8!$9TQP_n_=~6;lIsOAL))uuzS-W+#p- z<4-7-_~NDtjSDH29Jg9so%RD6alw_{bBRr?c5muj=$Pmf7dxT9hBFG1_<3HJ^-Cyh z%$P6DUSiFuiE~cSCr+=h0!$}{Gul=(zDnI)Vo?YdMJv%A+*W-72D;fMf19( zkkr@PiZuDl2j?b+F(8kqz+JCn*CH8*+e4T$k diff --git a/content/en/blog/_index.md b/content/en/blog/_index.md deleted file mode 100644 index 9283b958..00000000 --- a/content/en/blog/_index.md +++ /dev/null @@ -1,16 +0,0 @@ -+++ -title = "BLOGS" -date = 2017-01-01T00:00:00 -type = "blogList" - -# List format. -# 0 = Simple -# 1 = Detailed -# 2 = Stream -list_format = 2 - -# Optional featured image (relative to `static/img/` folder). -[header] -image = "" -caption = "" -+++ diff --git a/content/en/docs/.DS_Store b/content/en/docs/.DS_Store deleted file mode 100644 index 9633916c63c2acb9e8d64b20a6df55487e9cb73c..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 6148 zcmeHKJx{|h5PdEk3M?HNy(1Cv52C7Uhy}4jS_DYYR&65I{B+)ZC{QI`s|wvk_OtK9 zc3z@f4nUeU%UfUyV8SLiI%hW^VwcvFk!!}pq-(6P!VWk1!m{Tbic@4jp4|mrut1Fu z{CR$xy1H+6^_KQ)Jn^Z%V#~gu?}pbW=LOyvQ{x>CD$c&CtLP&-u`YBy8Ly}*6brr? zYg%Nb!#h9t&Vwh@I5|P1`*&~Lkxo3k^Y4iV>^yNuFc1s`1E<6QbGFFj(lGj9AQ%V+ zItJwXkl6%F$JS6k9qhCOAj)r~3dd4QNKAGt9a}?MD9J*J7Mi@oNES|gcE8fGHMDRf zFFum*oV-Fwx;xj;>W)+zMjs3W1N#h|*>o!T|L$=8zaJE{U?3RyR}7@_{Bb_xEqQM3 v-JImw%=XMCB7Uu57s7$2V){xdzG73MKFfeuI<|(?P~t~GX^1WuI4T1_CwNmw diff --git a/content/en/docs/v1-10-0/architecture.md b/content/en/docs/v1-10-0/architecture.md deleted file mode 100644 index 25e72699..00000000 --- a/content/en/docs/v1-10-0/architecture.md +++ /dev/null @@ -1,42 +0,0 @@ -+++ -title = "Architecture" - -date = 2024-09-29 -lastmod = 2024-09-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Architecture" -[menu.v1-10-0] - parent = "home" - weight = 2 -+++ - -## Overall Architecture - - -{{

}} - - -Volcano is designed for high-performance workloads running on Kubernetes. It follows the design and mechanisms of Kubernetes. - - -{{
}} - - -Volcano consists of **scheduler** / **controllermanager** / **admission** / **vcctl**: - -##### Scheduler -Volcano Scheduler schedules jobs to the most suitable node based on actions and plug-ins. Volcano supplements Kubernetes to support multiple scheduling algorithms for jobs. - -##### ControllerManager (CM) -Volcano CMs manage the lifecycle of Custom Resource Definitions (CRDs). You can use the **Queue CM**, **PodGroup CM**, and **VCJob CM**. - -##### Admission -Volcano Admission is responsible for the CRD API validation. - -##### vcctl -Volcano vcctl is the command line client for Volcano. diff --git a/content/en/docs/v1-10-0/flink_on_volcano.md b/content/en/docs/v1-10-0/flink_on_volcano.md deleted file mode 100644 index 2fccf30f..00000000 --- a/content/en/docs/v1-10-0/flink_on_volcano.md +++ /dev/null @@ -1,279 +0,0 @@ -+++ -title = "Flink on Volcano" - -date = 2024-09-29 -lastmod = 2024-09-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Flink" -[menu.v1-10-0] - parent = "ecosystem" - weight = 2 - -+++ - - - -### Flink introduction - -Apache Flink is an open-source streaming framework developed by the Apache Software Foundation. At its core, Apache Flink is a distributed streaming data streaming engine written in Java and Scala. Flink executes any stream data program in data parallelism and pipelining. Flink's pipelined runtime system can execute both batch and stream programs. In addition, the Flink runtime itself supports the execution of iterative algorithms. - -### The premise condition - -Make sure the deployed Kubernetes, Kubectl, Volcano are installed correctly. - -### Deployment process - -##### 1. Download - -To run Flink, which requires a Java 8 or 11 environment, use the following instructions to determine the Java version. - -```bash -java -version -``` - -Download the package and go to the directory. - -``` -$ wget https://www.apache.org/dyn/closer.lua/flink/flink-1.12.2/flink-1.12.2-src.tgz -$ cd flink-1.12.2 -``` - -##### 2. Start a Cluster - -Running the script completes the deployment of Flink on the cluster. - -```bash -$ ./bin/start-cluster.sh -``` - -##### 3. Submit a job - -Submit the job using the following instructions. - -```bash -$ ./bin/flink run examples/streaming/WordCount.jar -$ tail log/flink-*-taskexecutor-*.out -``` - -### Flink on Volcano - -##### 1. The deployment of the component - -Deploying a Flink Cluster requires creating two deploys, a Service, and a ConfigMap. The scheduling strategy is Volcano.The contents of `flink-configuration-configmap.yaml` are shown below. - -``` -apiVersion: v1 -kind: ConfigMap -metadata: - name: flink-config - labels: - app: flink -data: - flink-conf.yaml: |+ - jobmanager.rpc.address: flink-jobmanager - taskmanager.numberOfTaskSlots: 2 - blob.server.port: 6124 - jobmanager.rpc.port: 6123 - taskmanager.rpc.port: 6122 - queryable-state.proxy.ports: 6125 - jobmanager.memory.process.size: 1600m - taskmanager.memory.process.size: 1728m - parallelism.default: 2 - log4j-console.properties: |+ - # This affects logging for both user code and Flink - rootLogger.level = INFO - rootLogger.appenderRef.console.ref = ConsoleAppender - rootLogger.appenderRef.rolling.ref = RollingFileAppender - - # Uncomment this if you want to _only_ change Flink's logging - #logger.flink.name = org.apache.flink - #logger.flink.level = INFO - - # The following lines keep the log level of common libraries/connectors on - # log level INFO. The root logger does not override this. You have to manually - # change the log levels here. - logger.akka.name = akka - logger.akka.level = INFO - logger.kafka.name= org.apache.kafka - logger.kafka.level = INFO - logger.hadoop.name = org.apache.hadoop - logger.hadoop.level = INFO - logger.zookeeper.name = org.apache.zookeeper - logger.zookeeper.level = INFO - - # Log all infos to the console - appender.console.name = ConsoleAppender - appender.console.type = CONSOLE - appender.console.layout.type = PatternLayout - appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n - - # Log all infos in the given rolling file - appender.rolling.name = RollingFileAppender - appender.rolling.type = RollingFile - appender.rolling.append = false - appender.rolling.fileName = ${sys:log.file} - appender.rolling.filePattern = ${sys:log.file}.%i - appender.rolling.layout.type = PatternLayout - appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n - appender.rolling.policies.type = Policies - appender.rolling.policies.size.type = SizeBasedTriggeringPolicy - appender.rolling.policies.size.size=100MB - appender.rolling.strategy.type = DefaultRolloverStrategy - appender.rolling.strategy.max = 10 - - # Suppress the irrelevant (wrong) warnings from the Netty channel handler - logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline - logger.netty.level = OFF -``` - -Service is used to provide services for the REST and UI ports of the JobManager.The contents of `jobManager-Service.yaml` are as follows. - -``` -apiVersion: v1 -kind: Service -metadata: - name: flink-jobmanager -spec: - type: ClusterIP - ports: - - name: rpc - port: 6123 - - name: blob-server - port: 6124 - - name: webui - port: 8081 - selector: - app: flink - component: jobmanager -``` - -The contents of `jobmanager-session-deployment.yaml` are as follows. - -``` -apiVersion: apps/v1 -kind: Deployment -metadata: - name: flink-jobmanager -spec: - replicas: 1 - selector: - matchLabels: - app: flink - component: jobmanager - template: - metadata: - labels: - app: flink - component: jobmanager - spec: - containers: - - name: jobmanager - image: flink:1.11.0-scala_2.11 - args: ["jobmanager"] - ports: - - containerPort: 6123 - name: rpc - - containerPort: 6124 - name: blob-server - - containerPort: 8081 - name: webui - livenessProbe: - tcpSocket: - port: 6123 - initialDelaySeconds: 30 - periodSeconds: 60 - volumeMounts: - - name: flink-config-volume - mountPath: /opt/flink/conf - securityContext: - runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary - volumes: - - name: flink-config-volume - configMap: - name: flink-config - items: - - key: flink-conf.yaml - path: flink-conf.yaml - - key: log4j-console.properties - path: log4j-console.properties -``` - -The contents of `taskmanager-session-deployment.yaml` are as follows. - -``` -apiVersion: apps/v1 -kind: Deployment -metadata: - name: flink-taskmanager -spec: - replicas: 2 - selector: - matchLabels: - app: flink - component: taskmanager - template: - metadata: - labels: - app: flink - component: taskmanager - spec: - containers: - - name: taskmanager - image: flink:1.11.0-scala_2.11 - args: ["taskmanager"] - ports: - - containerPort: 6122 - name: rpc - - containerPort: 6125 - name: query-state - livenessProbe: - tcpSocket: - port: 6122 - initialDelaySeconds: 30 - periodSeconds: 60 - volumeMounts: - - name: flink-config-volume - mountPath: /opt/flink/conf/ - securityContext: - runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary - volumes: - - name: flink-config-volume - configMap: - name: flink-config - items: - - key: flink-conf.yaml - path: flink-conf.yaml - - key: log4j-console.properties - path: log4j-console.properties -``` - -Create the above four YAML configuration files on the cluster node and deploy them using the following instructions. - -``` -kubectl create -f flink-configuration-configmap.yaml -kubectl create -f jobmanager-service.yaml -kubectl create -f jobmanager-session-deployment.yaml -kubectl create -f taskmanager-session-deployment.yaml -``` - -Query to see if the payload was successfully created： - -``` -kubectl get cm| grep flink -kubectl get svc | grep flink -kubectl get pod -owide | grep Flink -``` - -##### 2. Outward publishing service - -Once the Flink payload is created, you need to publish the service externally。 - -- If you use Huawei Cloud CCE for testing, go to the "Workloads - Stateless Loads" page of CCE. Select Flink-JobManager and click Access Mode. -- Click "Add Service", select node access, and enter container port bit 8081. -- Click Network Management in CCE, you can see the service we just added, and visit the link for external publication. -- Go to the Dashboard page of Flink and click Submit New Job to submit the task. Here you have the option to submit an officially-provided WordCount sample.The directory is `flink-1.12.2/examples/streaming/WordCount.jar` \ No newline at end of file diff --git a/content/en/docs/v1-10-0/membership.md b/content/en/docs/v1-10-0/membership.md deleted file mode 100644 index a8d225b4..00000000 --- a/content/en/docs/v1-10-0/membership.md +++ /dev/null @@ -1,135 +0,0 @@ -+++ -title = "Community Membership" - -date = 2024-09-29 -lastmod = 2024-09-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Community Membership" -[menu.v1-10-0] - parent = "contribution" - weight = 2 -+++ - -# Volcano Community Membership - -**Note:** We regularly update this document with the latest information and incorporate feedback from the Volcano community. - -This document gives a brief overview of the Volcano community roles and the requirements and responsibilities associated with them. - -| Role | Requirements | Responsibilities | Privileges | -| -----| ---------------- | ------------ | -------| -| [Member](#member) | Sponsored by 2 approvers, active in the community, has contributed to Volcano | Gets new contributors on board| Member of the Volcano GitHub organization | -| [Approver](#approver) | Sponsored by 2 maintainers, has extensive experience and knowledge of a particular domain, has actively contributed to code and review | Reviews and approves contributions from community members | Has write access to specific packages in relevant repositories | -| [Maintainer](#maintainer) | Sponsored by 2 owners, has shown good technical judgment in feature design/development and PR review | Release planning and feature development/maintenance | Has top-level write access to relevant repositories; name entry in the Maintainers file of the repositories | -| [Owner](#owner) | Sponsored by 3 owners, has proven ability to lead the entire Volcano project | Drives the overall technical roadmap of the project and sets priorities of activities in release planning | Has admin access to the Volcano GitHub organization | - - -**Note:** All Volcano community members must follow the Volcano [Code of Conduct](https://github.com/volcano-sh/website/blob/master/CODE_OF_CONDUCT.md). - -## Member - -Members are active participants in the community. They contribute by authoring PRs, -reviewing issues/PRs or participating in community discussions on slack/mailing list. - - -### Requirements - -- Sponsored by 2 approvers -- Has enabled two-factor authentication for GitHub account -- Actively contributed to the community. Contributions include but are not limited to: - - Authoring PRs - - Reviewing issues/PRs authored by other community members - - Participating in community discussions on slack/mailing list - - Attending Volcano community meetings - - -### Responsibilities and privileges - -- Becomes a member of the Volcano GitHub organization -- Can be assigned issues and PRs and can work on review requests from community members -- Assigns issues and PRs -- Gets new contributors on board -- Guides new contributors on maintaining docs/files -- Helps/Motivates new members in contributing to Volcano - - -## Approver - -Approvers are active members who have extensive experience and knowledge of a particular domain. -They have actively participated in the issue/PR reviews and have identified noteworthy issues during the reviews. - - -### Requirements - -- Sponsored by 2 maintainers -- Has been a member for at least 2 months -- Has reviewed a large number of PRs -- Has good codebase knowledge - - -### Responsibilities and privileges - -- Reviews code to maintain/improve code quality -- Acknowledges and works on review requests from community members -- Approves code contributions for acceptance related to a particular domain -- Has write access to specific packages inside a repository, enforced via bot -- Continues to contribute and guide other community members to contribute to Volcano - -## Maintainer - -Maintainers are approvers who have shown good technical judgment in feature design/development. -They can grasp the big picture of the project and features in the project. - -### Requirements - -- Sponsored by 2 owners -- Has been an approver for at least 2 months -- Is nominated by a project owner -- Has good technical judgment in feature design/development - -### Responsibilities and privileges - -- Participates in release planning -- Maintains project code quality -- Ensures the forward/backword compatibility of APIs based on feature graduation criteria -- Analyzes and proposes new features/enhancements in the Volcano project -- Demonstrates sound technical judgment -- Mentors contributors and approvers -- Has top-level write access to relevant repositories (including the permission to click the Merge PR button when manual check-in is necessary) -- Name entry in the Maintainers file of the repositories -- Participates & drives design/development of multiple features - -## Owner - -Owners are maintainers who have helped lead the project. -They have deep understanding of Volcano and the related domains, and have facilitated major decision making in issues such as release planning. - -### Requirements - -- Sponsored by 3 owners -- Has been a maintainer for at least 2 months -- Is nominated by a project owner -- Is not opposed by any project owner -- Has helped lead the entire project - -### Responsibilities and privileges - -- Makes overall technical decisions of the project -- Makes the overall technical roadmap of the project -- Sets priorities for release planning -- Guides and mentors all other community members -- Ensures all community members follow the Code of Conduct -- Makes sure all PRs are properly reviewed and merged, despite the admin access to all repositories -- May obtain admin access to relevant repositories as required -- Participates & drives design/development of multiple features - - -**Note:** These roles are applicable only to the Volcano Github organization and repositories. We are currently developing a formal process for review and acceptance into these roles. - - -[two-factor authentication]: https://help.github.com/articles/about-two-factor-authentication diff --git a/content/en/docs/v1-10-0/mindspore_on_volcano.md b/content/en/docs/v1-10-0/mindspore_on_volcano.md deleted file mode 100644 index 17a2075d..00000000 --- a/content/en/docs/v1-10-0/mindspore_on_volcano.md +++ /dev/null @@ -1,77 +0,0 @@ -+++ -title = "MindSpore on Volcano" - -date = 2024-09-29 -lastmod = 2024-09-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "MindSpore" -[menu.v1-10-0] - parent = "ecosystem" - weight = 4 - -+++ - - - -### MindSpore introduction - -MindSpore is a new generation of deep learning framework launched by Huawei. It is derived from the best practices of the whole industry, which best matches the computing power of the rise processor, supports the flexible deployment of the whole scene of terminal, edge and cloud, and creates a brand new AI programming paradigm and lowers the threshold of AI development. - -### MindSpore on Volcano - -Create a new `mindSpore-cpu.yaml` in the cluster as follows. - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: mindspore-cpu -spec: - minAvailable: 1 - schedulerName: volcano - policies: - - event: PodEvicted - action: RestartJob - plugins: - ssh: [] - env: [] - svc: [] - maxRetry: 5 - queue: default - tasks: - - replicas: 8 - name: "pod" - template: - spec: - containers: - - command: ["/bin/bash", "-c", "python /tmp/lenet.py"] - image: lyd911/mindspore-cpu-example:0.2.0 - imagePullPolicy: IfNotPresent - name: mindspore-cpu-job - resources: - limits: - cpu: "1" - requests: - cpu: "1" - restartPolicy: OnFailure -``` - -Deploy. - -``` -kubectl apply -f mindspore-cpu.yaml -``` - -Query the status of the jobs in the cluster. - -``` -kubectl get pods -``` - - - diff --git a/content/en/docs/v1-10-0/mpi_on_volcano.md b/content/en/docs/v1-10-0/mpi_on_volcano.md deleted file mode 100644 index 82474e55..00000000 --- a/content/en/docs/v1-10-0/mpi_on_volcano.md +++ /dev/null @@ -1,112 +0,0 @@ -+++ -title = "MPI on Volcano" - -date = 2024-09-29 -lastmod = 2024-09-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "MPI" -[menu.v1-10-0] - parent = "ecosystem" - weight = 5 - -+++ - - - -### HPC introduction - -High Performance Computing (HPC) refers to the use of aggregated Computing power to handle data-intensive Computing tasks that cannot be performed by standard workstations. - -HPC = PBS + Maui + OpenMPI[1] - -- PBS：Resource manager, which is responsible for managing resources for all nodes in the cluster -- Maui：Third-party task scheduler, support resource reservation, support various complex priority policies, support preemption, etc. -- OpenMPI：The upper communication environment, taking into account the functions of communication library, compilation and distributed start task. - -### openMPI introduction - -The OpenMPI project is an open source messaging interface implementation developed and maintained by a coalition of academic, research, and industry partners. Through it we can carry on the parallelization program design. - -### How is opensMPI implemented - -Here is a simple 4-thread MPI program example. - -{{
}} - - - -### MPI on Volcano - -Create `mpi-example.yaml`. - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: lm-mpi-job -spec: - minAvailable: 3 - schedulerName: volcano - plugins: - ssh: [] - svc: [] - tasks: - - replicas: 1 - name: mpimaster - policies: - - event: TaskCompleted - action: CompleteJob - template: - spec: - containers: - - command: - - /bin/sh - - -c - - | - MPI_HOST=`cat /etc/volcano/mpiworker.host | tr "\n" ","`; - mkdir -p /var/run/sshd; /usr/sbin/sshd; - mpiexec --allow-run-as-root --host ${MPI_HOST} -np 2 mpi_hello_world > /home/re; - image: volcanosh/example-mpi:0.0.1 - name: mpimaster - ports: - - containerPort: 22 - name: mpijob-port - workingDir: /home - restartPolicy: OnFailure - - replicas: 2 - name: mpiworker - template: - spec: - containers: - - command: - - /bin/sh - - -c - - | - mkdir -p /var/run/sshd; /usr/sbin/sshd -D; - image: volcanosh/example-mpi:0.0.1 - name: mpiworker - ports: - - containerPort: 22 - name: mpijob-port - workingDir: /home - restartPolicy: OnFailure ---- - -``` - -Delopy `mpi-example.yaml`. - -``` -kubectl apply -f mpi-example.yaml -``` - -View job performance under the cluster. - -``` -kubectl get pod -``` \ No newline at end of file diff --git a/content/en/docs/v1-10-0/plugins.md b/content/en/docs/v1-10-0/plugins.md deleted file mode 100644 index 42404573..00000000 --- a/content/en/docs/v1-10-0/plugins.md +++ /dev/null @@ -1,176 +0,0 @@ -+++ -title = "Plugins" - -date = 2024-09-29 -lastmod = 2024-09-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Plugins" -[menu.v1-10-0] - parent = "scheduler" - weight = 3 -+++ - - - -### Gang - -{{
}} - -#### Overview - -The Gang scheduling strategy is one of the core scheduling algorithms of the Volcano-Scheduler. It meets the scheduling requirements of "All or nothing" in the scheduling process and avoids the waste of cluster resources caused by arbitrary scheduling of Pod. The Gang scheduler algorithm is to observe whether the scheduled number of Pods under Job meets the minimum number of runs. When the minimum number of runs of Job is satisfied, the scheduling action is executed for all Pods under Job; otherwise, it is not executed. - -#### Scenario - -The Gang scheduling algorithm based on the container group concept is well suited for scenarios that require multi-process collaboration. AI scenes often contain complex processes. Data Ingestion, Data Analysts, Data Splitting, trainers, Serving, Logging, etc., which require a group of containers to work together, are suitable for container-based Gang scheduling strategies. Multi-thread parallel computing communication scenarios under MPI computing framework are also suitable for Gang scheduling because master and slave processes need to work together. Containers under the container group are highly correlated, and there may be resource contention. The overall scheduling allocation can effectively solve the deadlock. - -In the case of insufficient cluster resources, the scheduling strategy of Gang can significantly improve the utilization of cluster resources. - -### Binpack - -#### Overview - -The goal of the BinPack scheduling algorithm is to fill as many existing nodes as possible (try not to allocate blank nodes). In the concrete implementation, BinPack scheduling algorithm scores the nodes that can be delivered, and the higher the score, the higher the resource utilization rate of nodes. Binpack algorithm can fill up the nodes as much as possible to close the application load to some nodes, which is very conducive to the automatic expansion capacity function of K8s cluster nodes. - -The BinPack algorithm is injected into the Volcano-Scheduler process as a plug-in and will be applied during the Pod stage of node selection. When calculating the Binpack algorithm, the Volcano-Scheduler considers the various resources requested by Pod and averages them according to the weights configured for each resource. The weight of each resource in the node score calculation is different, depending on the weight value configured by the administrator for each resource. Different plug-ins also need to assign different weights when calculating node scores, and the Scheduler also sets the score weights for BinPack plugins. - -#### Scenario - -The BinPack algorithm is good for small jobs that can fill as many nodes as possible. For example, the single query job in the big data scene, the order generation in the e-commerce seckill scene, the single identification job in the AI scene, and the high concurrency service scene on the Internet, etc. This scheduling algorithm can reduce the fragmentation in the node as much as possible, and reserve enough resource space on the idle machine for Pod which has applied for more resource requests, so as to maximize the utilization of idle resources under the cluster. - - - -### Priority - -{{
}} - -#### Overview - -The Priority Plugin provides the implementation of job, Task sorting, and PreempTablefn, a function that calculates sacrifice jobs. Job sorting according to priorityClassName, the task of sorting by priorityClassName, createTime, id in turn. - -#### Scenario - -When the cluster runs multiple jobs but is low on resources, and each Job has a different number of Pods waiting to be scheduled, if you use the Kubernetes default scheduler, the Job with more Pods will eventually get more of the cluster's resources. In this case, the Volcano-Scheduler provides algorithms that enable different jobs to share cluster resources in a fair-share. - -The Priority Plugin enables users to customize their job and task priorities, and to customize scheduling policies at different levels according to their own needs. Priority is arranged according to Job's PriorityClassName at the application level. For example, there are financial scenarios, Internet of Things monitoring scenarios and other applications requiring high real-time performance in the cluster, and the Priority Plugin can ensure that they are scheduled in Priority. - - - -### DRF - -{{
}} - -#### Overview - -The full name of DRF scheduling algorithm is Dominant Resource Fairness, which is a scheduling algorithm based on the container group Dominant Resource. Dominant Resource is the largest percentage of all required resources for a container group. The DRF algorithm selects the Dominant Resource that is the smallest in a series of container groups for priority scheduling. This can meet more job, not because a fat business, starve a large number of small business. DRF scheduling algorithm can ensure that many types of resources coexist in the environment, as far as possible to meet the fair principle of allocation. - -#### Scenario - -The DRF scheduling algorithm gives priority to the throughput of the business in the cluster and is suitable for batch small business scenarios such as a single AI training, a single big data calculation and a query. - - - -### Proportion - -#### Overview - -Proportion scheduling algorithm uses the concept of queue to control the Proportion of total resources allocated in the cluster. Each queue allocates a certain proportion of cluster resources. For example, there are three teams that share A pool of resources on A cluster: Team A uses up to 40% of the total cluster, Team B uses up to 30%, and Team C uses up to 30%. If the amount of work delivered exceeds the team's maximum available resources, there is a queue. - -#### Scenario - -Proportion scheduling algorithm improves the flexibility and elasticity of cluster scheduling. The most typical scenario is that when multiple development teams in a company share a cluster, this scheduling algorithm can handle the requirements of shared resource matching and isolation between different departments very well. In multi-service mixed scenarios, such as computation-intensive AI business, network IO-intensive MPI and HPC business, and storage-intensive big data business, Proportion scheduling algorithm can allocate shared resources according to demand through matching. - - - -### Task-topology - -#### Overview - -The task-topology algorithm is an algorithm that computes the priority of tasks and nodes based on the affinity and anti-affinity configuration between tasks within a Job. By configuring the affinity and anti-affinity policies between tasks within the Job and using the Task-Topology algorithm, tasks with affinity configurations can be scheduled to the same node first, and PODs with anti-affinity configurations to different nodes. - -#### Scenario - -node affinity： - -- Task-topology is important for improving computational efficiency in deep learning computing scenarios. Using the TensorFlow calculation as an example, configure the affinity between "ps" and "worker". Task-topology algorithm enables "ps" and "worker" to be scheduled to the same node as far as possible, so as to improve the efficiency of network and data interaction between "ps" and "worker", thus improving the computing efficiency. -- Tasks in HPC and MPI scenarios are highly synchronized and need high-speed network IO. - -Anti-affinity： - -- Take the TensorFlow calculation as an example, the anti-affinity between "ps" and "ps" - -- Master and slave backup of e-commerce service scene, data disaster tolerant, to ensure that there are spare jobs to continue to provide service after a job fails. - - - -### Predicates - -#### Overview - -The Predicate Plugin calls the PredicateGPU with pod and nodeInfo as parameters to evaluate and pre-select jobs based on the results. - -#### Scenario - -In AI scenarios where GPU resources are required, the Predicate Plugin can quickly filter out those that require the GPU for centralized scheduling. - - -### Nodeorder - -#### Overview - -The NodeOrder Plugin is a scheduling optimization strategy that scores nodes from various dimensions through simulated assignments to find the node that is best suited for the current job. The scoring parameters are configured by the user. The parameter contains the Affinity、reqResource、LeastReqResource、MostResource、balanceReqResouce. - -#### Scenario - -NodeOrder Plugin provides scoring criteria of multiple dimensions for scheduling, and the combination of different dimensions enables users to flexibly configure appropriate scheduling policies according to their own needs. - - - -### SLA - -#### Overview - -When users apply jobs to Volcano, they may need adding some particular constraints to job, for example, longest Pending time aiming to prevent job from starving. And these constraints can be regarded as Service Level Agreement (SLA) which are agreed between volcano and user. So sla plugin is provided to receive and realize SLA settings for both individual job and whole cluster. - -#### Scenario - -Users can customize SLA related parameters in their own cluster according to business needs. For example, for clusters with high real-time service requirements, JobWaitingTime can be set as small as possible. For clusters with bulk computing jobs, JobWaitingTime can be set to larger. The parameters of a specific SLA and the optimization of the parameters need to be combined with the specific business and related performance measurement results. - -### TDM - -#### Overview - -The full name of TDM is Time Division Multiplexing. In a co-located environment, some nodes are in both Kubernetes cluster and Yarn cluster. For these nodes, Kubernetes and Yarn cluster can use these resource by time-sharing multiplexing.The TDM Plugin marks these nodes as `revocable nodes`. TDM plugin will try to dispatch `preemptable task` to `revocable node` in node revocable time and evict the `preemptable task` from `revocable node` out of revocable time.. TDM Plugin improves the time-division multiplexing ability of node resources in the scheduling process of Volcano. - -#### Scenario - -In ToB business, cloud vendors provide cloud-based resources for merchants, and different merchants adopt different container arrangement frameworks (Kubernetes/YARN, etc.). TDM Plugin improves the time-sharing efficiency of common node resources and further improves the utilization rate of resources. - - - -### Numa-aware - -#### Overview - -When the node runs many CPU-bound pods, the workload can move to different CPU cores depending on whether the pod is throttled and which CPU cores are available at scheduling time. Many workloads are not sensitive to this migration and thus work fine without any intervention. However, in workloads where CPU cache affinity and scheduling latency significantly affect workload performance, the kubelet allows alternative CPU management policies to determine some placement preferences on the node. - -The CPU Manager and the Topology Manager are all Kubelet components, However There is the following limitation: - -- The scheduler is not topology-aware. so it is possible to be scheduled on a node and then fail on the node due to the Topology Manager. this is unacceptable for TensorFlow job. If any worker or ps failed on node, the job will fail. -- The managers are node-level that results in an inability to match the best node for NUMA topology in the whole cluster. - -The Numa-Aware Plugin aims to address these limitations. - -- Support cpu resource topology scheduling. -- Support pod-level topology policies. - -#### Scenario - -Common scenarios for NUMA-Aware are computation-intensive jobs that are sensitive to CPU parameters, scheduling delays. Such as scientific calculation, video decoding, animation rendering, big data offline processing and other specific scenes. - - diff --git a/content/en/docs/v1-10-0/pp_on_volcano.md b/content/en/docs/v1-10-0/pp_on_volcano.md deleted file mode 100644 index 029aff35..00000000 --- a/content/en/docs/v1-10-0/pp_on_volcano.md +++ /dev/null @@ -1,242 +0,0 @@ -+++ -title = "PaddlePaddle on Volcano" - -date = 2024-09-29 -lastmod = 2024-09-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "PaddlePaddle" -[menu.v1-10-0] - parent = "ecosystem" - weight = 6 - -+++ - - - -### paddlepaddle introduction - -PaddlePaddle is a deep learning framework open source by Baidu in September 2016, aiming to provide a safe, efficient, flexible, easy to use and extensible deep learning platform. - -In October 2018, the Paddle Fluid 1.0 version was released by the Paddle Fluid team, which comprehensively upgraded core capabilities such as neural network description, large-scale distributed training, and high-performance inference engine. For example, in the latest Paddle Fluid 1.5.2, the Paddle fly supports data parallelism, model parallelism, pipeline parallelism, and other parallel modes. Parameter server architecture and point-to-point synchronous training architecture fully support large-scale training on hardware resource devices such as CPU and GPU [1]. - -### paddlepaddle on valcano - -Upload `ctr-Volcano.yaml` on the cluster node as follows - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: ctr-volcano -spec: - minAvailable: 4 - schedulerName: volcano - policies: - - event: PodEvicted - action: RestartJob - - event: PodFailed - action: RestartJob - tasks: - - replicas: 2 - name: pserver - template: - metadata: - labels: - paddle-job-pserver: fluid-ctr - spec: - imagePullSecrets: - - name: default-secret - volumes: - - hostPath: - path: /home/work/ - type: "" - name: seqdata - containers: - - image: volcanosh/edlctr:v1 - command: - - paddle_k8s - - start_fluid - imagePullPolicy: IfNotPresent - name: pserver - volumeMounts: - - mountPath: /mnt/seqdata - name: seqdata - resources: - limits: - cpu: 10 - memory: 30Gi - ephemeral-storage: 10Gi - requests: - cpu: 1 - memory: 100M - ephemeral-storage: 1Gi - env: - - name: GLOG_v - value: "0" - - name: GLOG_logtostderr - value: "1" - - name: TOPOLOGY - value: "" - - name: TRAINER_PACKAGE - value: /workspace - - name: NAMESPACE - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.namespace - - name: POD_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: POD_NAME - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.name - - name: PADDLE_CURRENT_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: PADDLE_JOB_NAME - value: fluid-ctr - - name: PADDLE_IS_LOCAL - value: "0" - - name: PADDLE_TRAINERS_NUM - value: "2" - - name: PADDLE_PSERVERS_NUM - value: "2" - - name: FLAGS_rpc_deadline - value: "36000000" - - name: ENTRY - value: cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1 - - name: PADDLE_PORT - value: "30236" - - name: LD_LIBRARY_PATH - value: /usr/local/lib:/usr/local/nvidia/lib64:/usr/local/rdma/lib64:/usr/lib64/mlnx_ofed/valgrind - - name: PADDLE_TRAINING_ROLE - value: PSERVER - - name: TRAINING_ROLE - value: PSERVER - restartPolicy: OnFailure - - replicas: 2 - policies: - - event: TaskCompleted - action: CompleteJob - name: trainer - template: - metadata: - labels: - paddle-job: fluid-ctr - spec: - imagePullSecrets: - - name: default-secret - volumes: - - hostPath: - path: /home/work/ - type: "" - name: seqdata - containers: - - image: volcanosh/edlctr:v1 - command: - - paddle_k8s - - start_fluid - imagePullPolicy: IfNotPresent - name: trainer - volumeMounts: - - mountPath: /mnt/seqdata - name: seqdata - resources: - limits: - cpu: 10 - memory: 30Gi - ephemeral-storage: 10Gi - requests: - cpu: 1 - memory: 100M - ephemeral-storage: 10Gi - env: - - name: GLOG_v - value: "0" - - name: GLOG_logtostderr - value: "1" - - name: TOPOLOGY - - name: TRAINER_PACKAGE - value: /workspace - - name: CPU_NUM - value: "2" - - name: NAMESPACE - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.namespace - - name: POD_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: POD_NAME - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.name - - name: PADDLE_CURRENT_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: PADDLE_JOB_NAME - value: fluid-ctr - - name: PADDLE_IS_LOCAL - value: "0" - - name: FLAGS_rpc_deadline - value: "36000000" - - name: PADDLE_PORT - value: "30236" - - name: PADDLE_PSERVERS_NUM - value: "2" - - name: PADDLE_TRAINERS_NUM - value: "2" - - name: PADDLE_TRAINING_ROLE - value: TRAINER - - name: TRAINING_ROLE - value: TRAINER - - name: LD_LIBRARY_PATH - value: /usr/local/lib:/usr/local/nvidia/lib64:/usr/local/rdma/lib64:/usr/lib64/mlnx_ofed/valgrind - - name: ENTRY - value: cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1 - restartPolicy: OnFailure - -``` - -Deploy under the cluster terminal. - -``` -kubectl apply -f ctr-volcano.yaml -``` - -Query if the job is running properly.If the PodGroup cannot meet the scheduling conditions, check that the cluster has sufficient resources available. - -``` -kubectl get podgroup -kubectl describe podgroup ctr-volcano -kubectl get pods | grep ctr-volcano -``` - -You can select a PServer task to view the log. - -``` -kubectl logs ctr-volcano-pserver-0 -``` - -Select a Tariner task to view the log. - -``` -kubectl logs ctr-volcano-trainer-0 -``` \ No newline at end of file diff --git a/content/en/docs/v1-10-0/referrals.md b/content/en/docs/v1-10-0/referrals.md deleted file mode 100644 index c952c3c8..00000000 --- a/content/en/docs/v1-10-0/referrals.md +++ /dev/null @@ -1,88 +0,0 @@ -+++ -title = "Recommended Configurations for Common Scenarios" - -date = 2024-09-29 -lastmod = 2024-09-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Scenario & Configuration" -[menu.v1-10-0] - parent = "scheduler" - weight = 4 - -+++ - - - -### Background - -{{
}} - -Deep learning generally involves four stages: data acquisition and processing, model training and evolution, model deployment, and model evaluation [1]. At the beginning of an enterprise's involvement in deep learning, due to the lack of relevant experience and resources for deep learning, the team members usually choose to complete the above steps manually step by step in order to quickly meet the immediate needs of the project development period. However, no matter how domain-specific a deep learning expert may have, scaling up a project without the support of an infrastructure platform becomes extremely difficult. Large-scale operations in a production environment require more rapid replication and iterative upgrades than local operations. Production environment has higher requirements for reliability of data, reproducibility of model training, scalability of training model, reliability of automated deployment and operation and maintenance. Therefore, when the enterprise grows to a certain scale, the enterprise will want to build a stable scientific computing platform, so that the upper business personnel can focus on algorithm optimization and business development. - -In general, the construction of a deep learning platform will encounter many challenges, mainly reflected in the following aspects: - -- Data management automation -- The efficient use of resources -- Mask the complexity of the underlying technology - - - -##### Data management - -In a business scenario of deep learning, practitioners spend a significant amount of time capturing and transforming the data needed to build the model. As business scenarios expand, the automation of data transformation becomes a bottleneck in the efficiency of model building. In the process of data processing, new data will be generated, and these data will not only be used for this training, but also for the subsequent reasoning process. And the newly generated data does not need to be passed to the data source, but instead wants to be placed in a new storage space. This requires the underlying platform to provide a scalable storage system. Flexible, extensible and secure data storage system will greatly promote the improvement of data management ability. - -##### The efficient use of resources - -Deep learning-related applications are resource-intensive, and the use of resources fluctuates greatly, with peaks and valleys. Fast access to computing resources when the application starts running; At the end of the application, it is very important to recycle the unsuitable computing resources for the improvement of resource utilization. The types and time of computing resources used in data processing, model training and reasoning are different, which requires a flexible resource supply mechanism provided by the computing platform. Deep learning tasks require a large amount of computing resources, so it is impossible to build computing resources for each user separately. The computing platform should be able to guarantee the multi-lease of resource use, allowing multiple users to use the computing resource pool at the same time, rather than being monopolized by a few users. - -##### Shield the underlying technical complexity - -AI practitioners focus on model building and product development, ignoring the importance of infrastructure to AI development. Machine Learning/Deep Learning's own growing technology stack of users includes computing frameworks for machine learning such as TensorFlow, the Spark data processing engine, and underlying drivers such as CUDA. Manually managing these dependencies can consume significant resources and effort on the part of practitioners. - - - -Based on the above goals and challenges, deep learning and other AI computing choose to use containers and Kubernetes to build and manage deep learning platforms. Need to achieve user isolation, queuing mechanism, user rights management, resource management, container ceiling management and many other functions. - - - -### Common scenarios - -1. Priority issues:There are many roles of algorithm engineers in the company, such as interns, full-time employees and senior engineers. They all do the same thing and need GPU resources to do experiments or training. However, when resources are tight, they need to queue according to priority. Even in the same position, there is preemption for urgent assignments (such as paper deadlines, project launches). The priority level of the task needs to be reported by the engineer. -2. storage of data:The data sets of deep learning scenarios are very large, so it is necessary to optimize the design at the storage level to meet this requirement. When computing, if one person trains a model, then simple storage is OK. However, if there are 100 people and each person has different training models and data sets, and needs to run multiple tasks, what kind of storage can help you solve this problem? This is a realistic problem facing us. How storage ADAPTS to the ever-expanding computing needs of users is a matter of concern to architects -3. Parallel computing problem:A general requirement of deep learning platform based on K8S is to support multi-machine scheduling and make model training as fast as possible through multi-machine parallel training. -4. User isolation problem :For larger companies, there are many business departments, which require multi-user management and multi-tenant isolation. (User isolation problem) -5. Batch scheduling requirements for pod under deep learning scenarios. -6. Resource preemption problem:The upper limit on the amount of scarce resources (GPUs) used and the upper limit on how long they can be used. -7. Reservation:The details of the underlying application resources are shielded. It only cares about the total amount of available GPU, not the distribution of GPU. - - - -## referrals of the configuration - -- Priority issues: Customize the priority hierarchy. Focus on the `PriorityClassName` of the job itself. This field represents the priority of the job and is in effect for preemptive scheduling and prioritization. - -- Data storage: Recommends the `block storage` service in extremely performing businesses such as databases. Recommend the use of `object storage` service in the business scenarios of HPC, big data and AI. - -- High performance computing, model training parallelism :WRF, Kubegene, MPI high performance computing scenarios, multi-machine parallel training. - - 1. Job Plugin: `ssh` provides secret login, and `svc` provides network information needed for job operation, such as host file, headless service, etc, to provide automatic configuration for calculating cluster parameters. - - 2. `Policies field`: Once any `pod` has been killed, restart the entire job `restartJob`/`compeletedJob`. Because of this HPC scenario, Gang-based training job, if a worker fails, it is usually pointless for the entire job to run. - - 3. This scenario will generally define multiple `tasks`, 1 master + multiple workers. - -- Multi-tenancy, isolation issues :TensorFlow has no concept of tenants. The new resource splitting tool `Queue` introduced by Volcano is configured around Queue in various ways. To ensure the multi-lease of the cluster resources, the surplus of the cluster resources, and the flexibility of the tenants to use the cluster resources. When the cluster resources are redivided, the tenant resource quota can be guaranteed through Reclaim. Configure `preempt, reclaim` in `Volcano-Scheduler-ConfigMap`; configure `proportion` in plugins. - -- Batch scheduling: The `Gang Plugin` meets the requirements of pod batch scheduling for deep learning scenarios. The class field `minAvailable`, the main indicator of the Gang scheduling algorithm. The plugins field needs to use `env` and `svc`. The definition of Policies is similar to the high performance computing scenario where a `pod` ejection restarts the job. - - - -Reference: - -[1][Challenges facing AI infrastructure ](https://zhuanlan.zhihu.com/p/75634193) - diff --git a/content/en/docs/v1-10-0/spark_on_volcano.md b/content/en/docs/v1-10-0/spark_on_volcano.md deleted file mode 100644 index f5ac4c47..00000000 --- a/content/en/docs/v1-10-0/spark_on_volcano.md +++ /dev/null @@ -1,101 +0,0 @@ -+++ -title = "Spark on Volcano" - -date = 2024-09-29 -lastmod = 2024-09-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Spark" -[menu.v1-10-0] - parent = "ecosystem" - weight = 8 - -+++ - - - -### Spark introduction - -Spark is a fast and versatile big data clustering computing system. It provides high-level APIs for Scala, Java, Python, and R, as well as an optimization engine that supports a generic computational graph for data analysis. It also supports a rich set of advanced tools, including Spark SQL for SQL and Dataframes, MLLib for machine learning, GraphX for graphics processing, and Spark Streaming for Streaming. - -### Spark on Volcano - -Currently, there are two ways to support the integration of Spark on Kubernetes and volcano. -- Spark on Kubernetes native support: maintained by the [Apache Spark community](https://github.com/apache/spark) and Volcano community -- Spark Operator support: maintained by the [GoogleCloudPlatform community](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) and Volcano community - -#### Spark on Kubernetes native support (spark-submit) - -Spark on Kubernetes with Volcano as a custom scheduler is supported since Spark v3.3.0 and Volcano v1.5.1. See more detail in [link](https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-volcano-as-customized-scheduler-for-spark-on-kubernetes). - -#### Spark Operator support (spark-operator) - -Install Spark-Operator through Helm. - -``` -$ helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator - -$ helm install my-release spark-operator/spark-operator --namespace spark-operator --create-namespace -``` - -To ensure that the Spark-Operator is up and running, check with the following directive. - -``` -$ kubectl get po -nspark-operator -``` - -Here's the official `spark.yaml`. - -``` -apiVersion: "sparkoperator.k8s.io/v1beta2" -kind: SparkApplication -metadata: - name: spark-pi - namespace: default -spec: - type: Scala - mode: cluster - image: "gcr.io/spark-operator/spark:v3.0.0" - imagePullPolicy: Always - mainClass: org.apache.spark.examples.SparkPi - mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar" - sparkVersion: "3.0.0" - batchScheduler: "volcano" #Note: the batch scheduler name must be specified with `volcano` - restartPolicy: - type: Never - volumes: - - name: "test-volume" - hostPath: - path: "/tmp" - type: Directory - driver: - cores: 1 - coreLimit: "1200m" - memory: "512m" - labels: - version: 3.0.0 - serviceAccount: spark - volumeMounts: - - name: "test-volume" - mountPath: "/tmp" - executor: - cores: 1 - instances: 1 - memory: "512m" - labels: - version: 3.0.0 - volumeMounts: - - name: "test-volume" - mountPath: "/tmp" -``` - -Deploy the Spark application and see the status. - -``` -$ kubectl apply -f spark-pi.yaml -$ kubectl get SparkApplication -``` diff --git a/content/en/docs/v1-10-0/tf_on_volcano.md b/content/en/docs/v1-10-0/tf_on_volcano.md deleted file mode 100644 index 4f7ebf8f..00000000 --- a/content/en/docs/v1-10-0/tf_on_volcano.md +++ /dev/null @@ -1,111 +0,0 @@ -+++ -title = "TensorFlow on Volcano" - -date = 2024-09-29 -lastmod = 2024-09-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "TensorFlow" -[menu.v1-10-0] - parent = "ecosystem" - weight = 7 - -+++ - - - -### TensorFlow introduction - -TensorFlow is a symbolic mathematical system based on data flow programming, which is widely used in programming and realization of various machine learning algorithms. Its predecessor is DistBelief, a neural network algorithm library of Google. - -### TensorFlow on Volcano - -PS-worker model: Parameter Server performs model-related services, Work Server trains related services, inference calculation, gradient calculation, etc[1]. - -{{
}} - -TensorFlow on Kubernetes has many problems: - -- Resource isolation. -- Lack of GPU scheduling, Gang Schuler. -- Process Legacy. -- Training log is not convenient to save. - -Create `tftest.yaml`. - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: tensorflow-dist-mnist -spec: - minAvailable: 3 - schedulerName: volcano - plugins: - env: [] - svc: [] - policies: - - event: PodEvicted - action: RestartJob - queue: default - tasks: - - replicas: 1 - name: ps - template: - spec: - containers: - - command: - - sh - - -c - - | - PS_HOST=`cat /etc/volcano/ps.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - export TF_CONFIG={\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"ps\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"}; - python /var/tf_dist_mnist/dist_mnist.py - image: volcanosh/dist-mnist-tf-example:0.0.1 - name: tensorflow - ports: - - containerPort: 2222 - name: tfjob-port - resources: {} - restartPolicy: Never - - replicas: 2 - name: worker - policies: - - event: TaskCompleted - action: CompleteJob - template: - spec: - containers: - - command: - - sh - - -c - - | - PS_HOST=`cat /etc/volcano/ps.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - export TF_CONFIG={\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"worker\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"}; - python /var/tf_dist_mnist/dist_mnist.py - image: volcanosh/dist-mnist-tf-example:0.0.1 - name: tensorflow - ports: - - containerPort: 2222 - name: tfjob-port - resources: {} - restartPolicy: Never -``` - -Deploy `tftest.yaml`. - -``` -kubectl apply -f tftest.yaml -``` - -View job health. - -``` -kubectl get pod -``` \ No newline at end of file diff --git a/content/en/docs/v1-11-0/architecture.md b/content/en/docs/v1-11-0/architecture.md deleted file mode 100644 index d06c0362..00000000 --- a/content/en/docs/v1-11-0/architecture.md +++ /dev/null @@ -1,43 +0,0 @@ -+++ -title = "Architecture" - - -date = 2019-01-28 -lastmod = 2020-08-28 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Architecture" -[menu.v1-11-0] - parent = "home" - weight = 2 -+++ - -## Overall Architecture - - -{{
}} - - -Volcano is designed for high-performance workloads running on Kubernetes. It follows the design and mechanisms of Kubernetes. - - -{{
}} - - -Volcano consists of **scheduler** / **controllermanager** / **admission** / **vcctl**: - -##### Scheduler -Volcano Scheduler schedules jobs to the most suitable node based on actions and plug-ins. Volcano supplements Kubernetes to support multiple scheduling algorithms for jobs. - -##### ControllerManager (CM) -Volcano CMs manage the lifecycle of Custom Resource Definitions (CRDs). You can use the **Queue CM**, **PodGroup CM**, and **VCJob CM**. - -##### Admission -Volcano Admission is responsible for the CRD API validation. - -##### vcctl -Volcano vcctl is the command line client for Volcano. diff --git a/content/en/docs/v1-11-0/cli.md b/content/en/docs/v1-11-0/cli.md deleted file mode 100644 index f58e5e84..00000000 --- a/content/en/docs/v1-11-0/cli.md +++ /dev/null @@ -1,74 +0,0 @@ -+++ -title = "CLI" - - -date = 2019-01-28 -lastmod = 2020-09-01 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Commandline" -[menu.v1-11-0] - parent = "cli" - weight = 1 -+++ - -## Introduction -A Command Line Interface (CLI) is provided for you to manage resources. -## Configuration - -1. You can obtain the latest executable file by cloning the code from GitHub and running the following command in the root directory of the project: -``` -# make vcctl -``` -2. Copy the executable file to $PATH. You then can execute it anywhere. - -## Command Line List -### Listing all jobs -vcctl job list - -```html -# vcctl job list -Name Creation Phase JobType Replicas Min Pending Running Succeeded Failed Unknown RetryCount -job-1 2020-09-01 Running Batch 1 1 0 1 0 0 0 0 -``` - -### Deleting a specific job -vcctl job delete --name job-name [--namespace job-namespace] - -```html -# vcctl delete job --name job-1 --namespaces default -delete job job-1 successfully -``` - -### Suspending a job -vcctl job suspend --name job-name [--namespace job-namespace] - -```html -# vcctl job suspend --name job-1 --namespace default -``` - -### Resuming a job (opposite to "vcctl job suspend") -vcctl job resume --name job-name [--namespace job-namespace] - -```html -# vcctl job resume --name job-1 --namespace default -``` - -### Running a job -vcctl job run --name job-name [--namespace job-namespace] - -```html -# vcctl job run --name job-1 --namespace default -``` - -## Note -For more information about Volcano command lines, run the following commands: - -```html -# vcctl -h -# vcctl [command] -h -``` diff --git a/content/en/docs/v1-11-0/contribution.md b/content/en/docs/v1-11-0/contribution.md deleted file mode 100644 index 086b5460..00000000 --- a/content/en/docs/v1-11-0/contribution.md +++ /dev/null @@ -1,165 +0,0 @@ -+++ -title = "Contribution" - - -date = 2019-01-28 -lastmod = 2020-09-01 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Volcano Contribution" -[menu.v1-11-0] - parent = "contribution" - weight = 1 -+++ - -# Welcome - -Welcome to Volcano! - -- [Before You Start](#before-you-start) - - [Code of Conduct](#code-of-conduct) - - [Community discussions](#community-discussions) - - [Community Expectations](#community-expectations) -- [Getting Started](#getting-started) -- [Your First Contribution](#your-first-contribution) - - [Find Something to Work On](#find-something-to-work-on) - - [Find a Good Topic](#find-a-good-topic) - - [Work on an Issue](#work-on-an-issue) - - [File an Issue](#file-an-issue) -- [Contribution Workflow](#contribution-workflow) - - [Open a Pull Request](#open-a-pull-request) -- [Code Review](#code-review) -- [Commit Message Format](#commit-message-format) - - [Testing](#testing) - -## Before You Start - -### Code of Conduct - -All Volcano contributors must read and observe the [Code of Conduct](https://github.com/volcano-sh/website/blob/master/CODE_OF_CONDUCT.md). - -### Community discussions - -To better communicate with the developers in the Volcano community, please subscribe to the Volcano channel in the following way. - -- Sign up and visit `slack.cncf. IO/` to join the CNCF workspace. -- Participate in community discussions by adding the channel search `volcano`. - -### Community Expectations - -Volcano is an open-source project driven by the Volcano community, which strives to promote a healthy, friendly, and productive environment. -The community is committed to developing a system that helps running high-performance workloads, such as AI, ML, and deep learning applications, on Kubernetes. Building such a system would be impossible without the support of community contributors with similar aspirations. - -- For details about the community roles, see [Community Membership](https://github.com/volcano-sh/website/blob/master/content/en/docs/membership.md). If you make significant contributions, you will have a more advanced role in the community. - - -## Getting Started - -- For more information on building and deployment, see [setup](https://github.com/volcano-sh/website/blob/master/content/en/docs/installation.md). - - -## Your First Contribution - -You can contribute in different areas, including filing issues, developing features, fixing critical bugs, and getting your work reviewed and merged. - -If you have any question about the development process, visit the [Slack Channel](https://cloud-native.slack.com/archives/C011GJDQS0N) ([sign up](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk)) -or join our [mailing list](https://groups.google.com/forum/#!forum/volcano-sh). - -#### Find Something to Work On - -You are welcome to open an issue concerning documentation, report bugs, and push changes to the repositories. -Feel free to optimize code that does not follow the best coding practices, perform code refactoring, or compile test cases. -The following steps will help you get started. - -#### Find a Good Topic - -There are [multiple repositories](https://github.com/volcano-sh/) within the Volcano organization with each repository containing a beginner-friendly issue that does not require deep understanding of the Volcano project. -For example, in [Volcano-Issues](https://github.com/volcano-sh/volcano), you can choose issues labeled with [help wanted](https://github.com/volcano-sh/volcano/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) or [good first issue](https://github.com/volcano-sh/volcano/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22). -New contributors are welcome to work on these issues. - -Another good way to start is to find a document that needs improvement, for example, a document that is missing a link or contains a broken link. For details on the workflow, see [Contribution Workflow](#contributor-workflow). - -#### Work on an Issue - -When you are ready to work on an issue, reply with `/assign` or `/assign @yourself` on an issue. -The bot then will assign the issue to you. Your name will then be displayed on the `Assignees` list. - -#### File an Issue - -You are welcome to file issues to Volcano sub-repositories. - -*Example:* You can file an issue for [Volcano](https://github.com/volcano-sh/volcano/issues). - -Follow the submission guidelines when you open an issue. - -## Contribution Workflow - -All contributors are welcome to open issues and create pull requests. - -The contribution workflow is as follows: - -- Create a topic branch from the existing branch (usually the master branch). -- Edit and commit the code. -- Make sure [commit message format](#commit-message-format) is followed. -- Push changes in the topic branch to your remote personal fork of the repository. -- Submit a pull request (PR) to [Volcano](https://github.com/volcano-sh/volcano). The PR must receive approval from at least two community maintainers before it can be merged. - -### Open a Pull Request - -Volcano follows the standard [GitHub pull request](https://help.github.com/articles/about-pull-requests/) process. - -Volcano bot will apply structured labels to your PRs. - -It also provides suggestions on commands in your PRs to facilitate review. -These `/command` options can be annotated to trigger auto-labeling and notifications. For more information, see [command reference documentation](https://go.k8s.io/bot-commands). - -### Code Review - -To make it easier for your PRs to receive reviews, - -* Follow [good coding guidelines](https://github.com/golang/go/wiki/CodeReviewComments). -* Write [good commit messages](https://chris.beams.io/posts/git-commit/). -* Break down large chunks of modification into smaller unites that are logically independent and easy to understand. -* Label your PRs properly so that they can be sent to appropriate reviewers. The bot will help you through the entire PR submission process. - - - -### Commit Message Format - -In the subject line mention the changes you have made, and in the message body provide the reasons for making these changes. - -```shell -scripts: add test code for metamanager - -Unit test code is added to improve code coverage for metamanager. - -Fixes #12 -``` - -A more formal format is as follows: - -```shell -: - - - -
-``` - -The first line is the subject, which can contain up to 70 characters. The second line is normally a blank line. The third line can contain up to 80 characters and mentions why the change was made. Ensure that the other lines do not contain more than 80 characters so that the message can be easily read on GitHub as well as using various Git tools. - -Note: If your pull request does not receive enough attention, you can reach out to the reviewers on Slack. - -### Testing - -After a PR is opened, the Volcano bot automatically performs multiple types of testing on it. The location of the test code and the environment requirements vary depending on the type of test. - -* Unit testing: used to check whether a specific function works properly. Unit test source code is in the same package as its corresponding source code. You can easily run it at local. -* Integration testing: used to check the interactions between package components or between Volcano components and Kubernetes control plane components like API server. -* End-to-end ("E2E") testing: used to test system consistency. [Volcano E2E test](https://github.com/volcano-sh/volcano/tree/master/test/e2e). - -Each PR has to pass all the test cases before it can be reviewed. diff --git a/content/en/docs/v1-11-0/flink_on_volcano.md b/content/en/docs/v1-11-0/flink_on_volcano.md deleted file mode 100644 index 74bd8ebc..00000000 --- a/content/en/docs/v1-11-0/flink_on_volcano.md +++ /dev/null @@ -1,279 +0,0 @@ -+++ -title = "Flink on Volcano" - -date = 2021-06-29 -lastmod = 2021-06-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Flink" -[menu.v1-11-0] - parent = "ecosystem" - weight = 2 - -+++ - - - -### Flink introduction - -Apache Flink is an open-source streaming framework developed by the Apache Software Foundation. At its core, Apache Flink is a distributed streaming data streaming engine written in Java and Scala. Flink executes any stream data program in data parallelism and pipelining. Flink's pipelined runtime system can execute both batch and stream programs. In addition, the Flink runtime itself supports the execution of iterative algorithms. - -### The premise condition - -Make sure the deployed Kubernetes, Kubectl, Volcano are installed correctly. - -### Deployment process - -##### 1. Download - -To run Flink, which requires a Java 8 or 11 environment, use the following instructions to determine the Java version. - -```bash -java -version -``` - -Download the package and go to the directory. - -``` -$ wget https://www.apache.org/dyn/closer.lua/flink/flink-1.12.2/flink-1.12.2-src.tgz -$ cd flink-1.12.2 -``` - -##### 2. Start a Cluster - -Running the script completes the deployment of Flink on the cluster. - -```bash -$ ./bin/start-cluster.sh -``` - -##### 3. Submit a job - -Submit the job using the following instructions. - -```bash -$ ./bin/flink run examples/streaming/WordCount.jar -$ tail log/flink-*-taskexecutor-*.out -``` - -### Flink on Volcano - -##### 1. The deployment of the component - -Deploying a Flink Cluster requires creating two deploys, a Service, and a ConfigMap. The scheduling strategy is Volcano.The contents of `flink-configuration-configmap.yaml` are shown below. - -``` -apiVersion: v1 -kind: ConfigMap -metadata: - name: flink-config - labels: - app: flink -data: - flink-conf.yaml: |+ - jobmanager.rpc.address: flink-jobmanager - taskmanager.numberOfTaskSlots: 2 - blob.server.port: 6124 - jobmanager.rpc.port: 6123 - taskmanager.rpc.port: 6122 - queryable-state.proxy.ports: 6125 - jobmanager.memory.process.size: 1600m - taskmanager.memory.process.size: 1728m - parallelism.default: 2 - log4j-console.properties: |+ - # This affects logging for both user code and Flink - rootLogger.level = INFO - rootLogger.appenderRef.console.ref = ConsoleAppender - rootLogger.appenderRef.rolling.ref = RollingFileAppender - - # Uncomment this if you want to _only_ change Flink's logging - #logger.flink.name = org.apache.flink - #logger.flink.level = INFO - - # The following lines keep the log level of common libraries/connectors on - # log level INFO. The root logger does not override this. You have to manually - # change the log levels here. - logger.akka.name = akka - logger.akka.level = INFO - logger.kafka.name= org.apache.kafka - logger.kafka.level = INFO - logger.hadoop.name = org.apache.hadoop - logger.hadoop.level = INFO - logger.zookeeper.name = org.apache.zookeeper - logger.zookeeper.level = INFO - - # Log all infos to the console - appender.console.name = ConsoleAppender - appender.console.type = CONSOLE - appender.console.layout.type = PatternLayout - appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n - - # Log all infos in the given rolling file - appender.rolling.name = RollingFileAppender - appender.rolling.type = RollingFile - appender.rolling.append = false - appender.rolling.fileName = ${sys:log.file} - appender.rolling.filePattern = ${sys:log.file}.%i - appender.rolling.layout.type = PatternLayout - appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n - appender.rolling.policies.type = Policies - appender.rolling.policies.size.type = SizeBasedTriggeringPolicy - appender.rolling.policies.size.size=100MB - appender.rolling.strategy.type = DefaultRolloverStrategy - appender.rolling.strategy.max = 10 - - # Suppress the irrelevant (wrong) warnings from the Netty channel handler - logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline - logger.netty.level = OFF -``` - -Service is used to provide services for the REST and UI ports of the JobManager.The contents of `jobManager-Service.yaml` are as follows. - -``` -apiVersion: v1 -kind: Service -metadata: - name: flink-jobmanager -spec: - type: ClusterIP - ports: - - name: rpc - port: 6123 - - name: blob-server - port: 6124 - - name: webui - port: 8081 - selector: - app: flink - component: jobmanager -``` - -The contents of `jobmanager-session-deployment.yaml` are as follows. - -``` -apiVersion: apps/v1 -kind: Deployment -metadata: - name: flink-jobmanager -spec: - replicas: 1 - selector: - matchLabels: - app: flink - component: jobmanager - template: - metadata: - labels: - app: flink - component: jobmanager - spec: - containers: - - name: jobmanager - image: flink:1.11.0-scala_2.11 - args: ["jobmanager"] - ports: - - containerPort: 6123 - name: rpc - - containerPort: 6124 - name: blob-server - - containerPort: 8081 - name: webui - livenessProbe: - tcpSocket: - port: 6123 - initialDelaySeconds: 30 - periodSeconds: 60 - volumeMounts: - - name: flink-config-volume - mountPath: /opt/flink/conf - securityContext: - runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary - volumes: - - name: flink-config-volume - configMap: - name: flink-config - items: - - key: flink-conf.yaml - path: flink-conf.yaml - - key: log4j-console.properties - path: log4j-console.properties -``` - -The contents of `taskmanager-session-deployment.yaml` are as follows. - -``` -apiVersion: apps/v1 -kind: Deployment -metadata: - name: flink-taskmanager -spec: - replicas: 2 - selector: - matchLabels: - app: flink - component: taskmanager - template: - metadata: - labels: - app: flink - component: taskmanager - spec: - containers: - - name: taskmanager - image: flink:1.11.0-scala_2.11 - args: ["taskmanager"] - ports: - - containerPort: 6122 - name: rpc - - containerPort: 6125 - name: query-state - livenessProbe: - tcpSocket: - port: 6122 - initialDelaySeconds: 30 - periodSeconds: 60 - volumeMounts: - - name: flink-config-volume - mountPath: /opt/flink/conf/ - securityContext: - runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary - volumes: - - name: flink-config-volume - configMap: - name: flink-config - items: - - key: flink-conf.yaml - path: flink-conf.yaml - - key: log4j-console.properties - path: log4j-console.properties -``` - -Create the above four YAML configuration files on the cluster node and deploy them using the following instructions. - -``` -kubectl create -f flink-configuration-configmap.yaml -kubectl create -f jobmanager-service.yaml -kubectl create -f jobmanager-session-deployment.yaml -kubectl create -f taskmanager-session-deployment.yaml -``` - -Query to see if the payload was successfully created： - -``` -kubectl get cm| grep flink -kubectl get svc | grep flink -kubectl get pod -owide | grep Flink -``` - -##### 2. Outward publishing service - -Once the Flink payload is created, you need to publish the service externally。 - -- If you use Huawei Cloud CCE for testing, go to the "Workloads - Stateless Loads" page of CCE. Select Flink-JobManager and click Access Mode. -- Click "Add Service", select node access, and enter container port bit 8081. -- Click Network Management in CCE, you can see the service we just added, and visit the link for external publication. -- Go to the Dashboard page of Flink and click Submit New Job to submit the task. Here you have the option to submit an officially-provided WordCount sample.The directory is `flink-1.12.2/examples/streaming/WordCount.jar` \ No newline at end of file diff --git a/content/en/docs/v1-11-0/kubeflow_on_volcano.md b/content/en/docs/v1-11-0/kubeflow_on_volcano.md deleted file mode 100644 index 6c19de38..00000000 --- a/content/en/docs/v1-11-0/kubeflow_on_volcano.md +++ /dev/null @@ -1,246 +0,0 @@ -+++ -title = "Kubeflow on Volcano" - -date = 2021-06-29 -lastmod = 2021-06-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Kubeflow" -[menu.v1-11-0] - parent = "ecosystem" - weight = 3 - -+++ - - - - - -### Kubeflow introduction - -Kubernetes has become the de facto standard for cloud native application choreography and management, and more and more applications are migrating to Kubernetes. The field of artificial intelligence and machine learning naturally contains a large number of computation-intensive tasks. Developers are very willing to build an AI platform based on Kubernetes and make full use of the resource management, application scheduling, operation and maintenance monitoring capabilities provided by Kubernetes. However, it is a very complicated and tedious process to build an end-to-end AI computing platform based on Kubernetes, which needs to deal with many links. In addition to the model training we are familiar with, it also includes data collection, preprocessing, resource management, feature extraction, data verification, model management, model release, monitoring and other links. For an AI algorithm engineer, if he wants to do model training, he has to build a set of AI computing platform. This process is time-consuming and laborious, and requires a lot of knowledge accumulation[1]. - -{{
}} - -Kubeflow was born in 2017. The Kubeflow project is built based on containers and Kubernetes, aiming to provide data scientists, machine learning engineers, system operation and maintenance personnel with an agile platform for machine learning business deployment, development, training, release and management. It takes advantage of cloud native technology to make it faster and easier for users to deploy, use and manage the most popular machine learning software. - -What scenarios can we use Kubeflow for： - -- You want to train the TensorFlow model and you can use the model interface to publish application services in the K8S environment. -- You want to debug your code using Jupyter Notebooks, a multi-user Notebook Server. -- In the training Job, the CPU or GPU resources need to be scheduled and choreographed. -- You want TensorFlow to be combined with other components to publish services. - - - -### Kubeflow on Volcano - -Volcano is an enhanced high performance computing task batch processing system built on Kubernetes. As a platform for high performance computing scenarios, it makes up for Kubernetes' lack of basic capabilities in machine learning, deep learning, HPC, and big data computing scenarios, including gang-schedule scheduling capability, computational task queue management, task-topology, and GPU affinity scheduling. In addition, Volcano has enhanced the batch creation and life cycle management of computing tasks, fair-share, binpack scheduling and other aspects on the basis of the native Kubernetes capability. Volcano has fully solved the problem of distributed training in Kubeflow mentioned above. - -#### download kfctl - -First of all, you need to download kfctl, you can choose the appropriate compressed package file according to the system [1]. - -``` -$ tar -xvf kfctl_v1.0.2-0-ga476281_linux.tar.gz -$ sudo mv ./kfctl /usr/local/bin/kfctl -``` - - - -#### Configure environment variables - -``` -$ export PATH= $PATH:"" -$ export KF_NAME= -$ export BASE_DIR= -$ export KF_DIR=${BASE_DIR}/${KF_NAME} -$ export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml" -``` - - - -#### Install Kubeflow - -``` -$ mkdir -p ${KF_DIR} -$ cd ${KF_DIR} -$ Kfctl apply -V -f ${CONFIG_URI} -``` - -Confirm the installation results with the following instructions. - -``` -$ kubectl -n kubeflow get all -``` - - - -#### deploy Mnist - -Download the official test set provided by Kubuflow. - -``` -git clone https://github.com/kubeflow/examples.git -``` - - - -#### Start using Notebook - -External interface service is provided, where the nodes under the cluster need to be bound to public network IP. If Notebook is not installed, please use pip3 to install it first. - -``` -$ pip3 install jupyter notebook -$ jupyter notebook --allow-root -[W 09:08:03.572 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended. -[I 09:08:03.575 NotebookApp] Serving notebooks from local directory: /root/examples -[I 09:08:03.575 NotebookApp] Jupyter Notebook 6.3.0 is running at: -[I 09:08:03.575 NotebookApp] http://mytest-87034:30200/ -[I 09:08:03.575 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). -``` - -Access your-IP：30200/,Enter the configuration password to enter the Notebook. - - - -#### Run the official instance on the Notebook[2] - -1.Open Notebook and deploy TFJob。Open the notebook `mnist/mnist_vanilla_k8s.ipynb` ,Follow the guidelines to deploy a distributed TF Job. - -2.Add a schedulerName: add `schedulerName: volcano` in mnist.yaml,Be sure to use volcano for scheduling. - -``` -train_spec = f"""apiVersion: kubeflow.org/v1 -kind: TFJob -metadata: - name: {train_name} -spec: - schedulerName: volcano - tfReplicaSpecs: - Ps: - replicas: {num_ps} - template: - metadata: - annotations: - sidecar.istio.io/inject: "false" - spec: - serviceAccount: default-editor - containers: - - name: tensorflow - command: - - python - - /opt/model.py - - --tf-model-dir={model_dir} - - --tf-export-dir={export_path} - - --tf-train-steps={train_steps} - - --tf-batch-size={batch_size} - - --tf-learning-rate={learning_rate} - env: - - name: S3_ENDPOINT - value: {s3_endpoint} - - name: AWS_ENDPOINT_URL - value: {minio_endpoint} - - name: AWS_REGION - value: {minio_region} - - name: BUCKET_NAME - value: {mnist_bucket} - - name: S3_USE_HTTPS - value: "0" - - name: S3_VERIFY_SSL - value: "0" - - name: AWS_ACCESS_KEY_ID - value: {minio_username} - - name: AWS_SECRET_ACCESS_KEY - value: {minio_key} - image: {image} - workingDir: /opt - restartPolicy: OnFailure - Chief: - replicas: 1 - template: - metadata: - annotations: - sidecar.istio.io/inject: "false" - spec: - serviceAccount: default-editor - containers: - - name: tensorflow - command: - - python - - /opt/model.py - - --tf-model-dir={model_dir} - - --tf-export-dir={export_path} - - --tf-train-steps={train_steps} - - --tf-batch-size={batch_size} - - --tf-learning-rate={learning_rate} - env: - - name: S3_ENDPOINT - value: {s3_endpoint} - - name: AWS_ENDPOINT_URL - value: {minio_endpoint} - - name: AWS_REGION - value: {minio_region} - - name: BUCKET_NAME - value: {mnist_bucket} - - name: S3_USE_HTTPS - value: "0" - - name: S3_VERIFY_SSL - value: "0" - - name: AWS_ACCESS_KEY_ID - value: {minio_username} - - name: AWS_SECRET_ACCESS_KEY - value: {minio_key} - image: {image} - workingDir: /opt - restartPolicy: OnFailure - Worker: - replicas: 1 - template: - metadata: - annotations: - sidecar.istio.io/inject: "false" - spec: - serviceAccount: default-editor - containers: - - name: tensorflow - command: - - python - - /opt/model.py - - --tf-model-dir={model_dir} - - --tf-export-dir={export_path} - - --tf-train-steps={train_steps} - - --tf-batch-size={batch_size} - - --tf-learning-rate={learning_rate} - env: - - name: S3_ENDPOINT - value: {s3_endpoint} - - name: AWS_ENDPOINT_URL - value: {minio_endpoint} - - name: AWS_REGION - value: {minio_region} - - name: BUCKET_NAME - value: {mnist_bucket} - - name: S3_USE_HTTPS - value: "0" - - name: S3_VERIFY_SSL - value: "0" - - name: AWS_ACCESS_KEY_ID - value: {minio_username} - - name: AWS_SECRET_ACCESS_KEY - value: {minio_key} - image: {image} - workingDir: /opt - restartPolicy: OnFailure -""" -``` - -3.submit a job - -``` -kubectl apply -f mnist.yaml -``` \ No newline at end of file diff --git a/content/en/docs/v1-11-0/membership.md b/content/en/docs/v1-11-0/membership.md deleted file mode 100644 index 405acc00..00000000 --- a/content/en/docs/v1-11-0/membership.md +++ /dev/null @@ -1,136 +0,0 @@ -+++ -title = "Community Membership" - - -date = 2019-01-28 -lastmod = 2020-09-01 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Community Membership" -[menu.v1-11-0] - parent = "contribution" - weight = 2 -+++ - -# Volcano Community Membership - -**Note:** We regularly update this document with the latest information and incorporate feedback from the Volcano community. - -This document gives a brief overview of the Volcano community roles and the requirements and responsibilities associated with them. - -| Role | Requirements | Responsibilities | Privileges | -| -----| ---------------- | ------------ | -------| -| [Member](#member) | Sponsored by 2 approvers, active in the community, has contributed to Volcano | Gets new contributors on board| Member of the Volcano GitHub organization | -| [Approver](#approver) | Sponsored by 2 maintainers, has extensive experience and knowledge of a particular domain, has actively contributed to code and review | Reviews and approves contributions from community members | Has write access to specific packages in relevant repositories | -| [Maintainer](#maintainer) | Sponsored by 2 owners, has shown good technical judgment in feature design/development and PR review | Release planning and feature development/maintenance | Has top-level write access to relevant repositories; name entry in the Maintainers file of the repositories | -| [Owner](#owner) | Sponsored by 3 owners, has proven ability to lead the entire Volcano project | Drives the overall technical roadmap of the project and sets priorities of activities in release planning | Has admin access to the Volcano GitHub organization | - - -**Note:** All Volcano community members must follow the Volcano [Code of Conduct](https://github.com/volcano-sh/website/blob/master/CODE_OF_CONDUCT.md). - -## Member - -Members are active participants in the community. They contribute by authoring PRs, -reviewing issues/PRs or participating in community discussions on slack/mailing list. - - -### Requirements - -- Sponsored by 2 approvers -- Has enabled two-factor authentication for GitHub account -- Actively contributed to the community. Contributions include but are not limited to: - - Authoring PRs - - Reviewing issues/PRs authored by other community members - - Participating in community discussions on slack/mailing list - - Attending Volcano community meetings - - -### Responsibilities and privileges - -- Becomes a member of the Volcano GitHub organization -- Can be assigned issues and PRs and can work on review requests from community members -- Assigns issues and PRs -- Gets new contributors on board -- Guides new contributors on maintaining docs/files -- Helps/Motivates new members in contributing to Volcano - - -## Approver - -Approvers are active members who have extensive experience and knowledge of a particular domain. -They have actively participated in the issue/PR reviews and have identified noteworthy issues during the reviews. - - -### Requirements - -- Sponsored by 2 maintainers -- Has been a member for at least 2 months -- Has reviewed a large number of PRs -- Has good codebase knowledge - - -### Responsibilities and privileges - -- Reviews code to maintain/improve code quality -- Acknowledges and works on review requests from community members -- Approves code contributions for acceptance related to a particular domain -- Has write access to specific packages inside a repository, enforced via bot -- Continues to contribute and guide other community members to contribute to Volcano - -## Maintainer - -Maintainers are approvers who have shown good technical judgment in feature design/development. -They can grasp the big picture of the project and features in the project. - -### Requirements - -- Sponsored by 2 owners -- Has been an approver for at least 2 months -- Is nominated by a project owner -- Has good technical judgment in feature design/development - -### Responsibilities and privileges - -- Participates in release planning -- Maintains project code quality -- Ensures the forward/backword compatibility of APIs based on feature graduation criteria -- Analyzes and proposes new features/enhancements in the Volcano project -- Demonstrates sound technical judgment -- Mentors contributors and approvers -- Has top-level write access to relevant repositories (including the permission to click the Merge PR button when manual check-in is necessary) -- Name entry in the Maintainers file of the repositories -- Participates & drives design/development of multiple features - -## Owner - -Owners are maintainers who have helped lead the project. -They have deep understanding of Volcano and the related domains, and have facilitated major decision making in issues such as release planning. - -### Requirements - -- Sponsored by 3 owners -- Has been a maintainer for at least 2 months -- Is nominated by a project owner -- Is not opposed by any project owner -- Has helped lead the entire project - -### Responsibilities and privileges - -- Makes overall technical decisions of the project -- Makes the overall technical roadmap of the project -- Sets priorities for release planning -- Guides and mentors all other community members -- Ensures all community members follow the Code of Conduct -- Makes sure all PRs are properly reviewed and merged, despite the admin access to all repositories -- May obtain admin access to relevant repositories as required -- Participates & drives design/development of multiple features - - -**Note:** These roles are applicable only to the Volcano Github organization and repositories. We are currently developing a formal process for review and acceptance into these roles. - - -[two-factor authentication]: https://help.github.com/articles/about-two-factor-authentication diff --git a/content/en/docs/v1-11-0/mindspore_on_volcano.md b/content/en/docs/v1-11-0/mindspore_on_volcano.md deleted file mode 100644 index f2d808e2..00000000 --- a/content/en/docs/v1-11-0/mindspore_on_volcano.md +++ /dev/null @@ -1,77 +0,0 @@ -+++ -title = "MindSpore on Volcano" - -date = 2021-06-29 -lastmod = 2021-06-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "MindSpore" -[menu.v1-11-0] - parent = "ecosystem" - weight = 4 - -+++ - - - -### MindSpore introduction - -MindSpore is a new generation of deep learning framework launched by Huawei. It is derived from the best practices of the whole industry, which best matches the computing power of the rise processor, supports the flexible deployment of the whole scene of terminal, edge and cloud, and creates a brand new AI programming paradigm and lowers the threshold of AI development. - -### MindSpore on Volcano - -Create a new `mindSpore-cpu.yaml` in the cluster as follows. - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: mindspore-cpu -spec: - minAvailable: 1 - schedulerName: volcano - policies: - - event: PodEvicted - action: RestartJob - plugins: - ssh: [] - env: [] - svc: [] - maxRetry: 5 - queue: default - tasks: - - replicas: 8 - name: "pod" - template: - spec: - containers: - - command: ["/bin/bash", "-c", "python /tmp/lenet.py"] - image: lyd911/mindspore-cpu-example:0.2.0 - imagePullPolicy: IfNotPresent - name: mindspore-cpu-job - resources: - limits: - cpu: "1" - requests: - cpu: "1" - restartPolicy: OnFailure -``` - -Deploy. - -``` -kubectl apply -f mindspore-cpu.yaml -``` - -Query the status of the jobs in the cluster. - -``` -kubectl get pods -``` - - - diff --git a/content/en/docs/v1-11-0/mpi_on_volcano.md b/content/en/docs/v1-11-0/mpi_on_volcano.md deleted file mode 100644 index f6682d13..00000000 --- a/content/en/docs/v1-11-0/mpi_on_volcano.md +++ /dev/null @@ -1,112 +0,0 @@ -+++ -title = "MPI on Volcano" - -date = 2021-04-07 -lastmod = 2021-04-07 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "MPI" -[menu.v1-11-0] - parent = "ecosystem" - weight = 5 - -+++ - - - -### HPC introduction - -High Performance Computing (HPC) refers to the use of aggregated Computing power to handle data-intensive Computing tasks that cannot be performed by standard workstations. - -HPC = PBS + Maui + OpenMPI[1] - -- PBS：Resource manager, which is responsible for managing resources for all nodes in the cluster -- Maui：Third-party task scheduler, support resource reservation, support various complex priority policies, support preemption, etc. -- OpenMPI：The upper communication environment, taking into account the functions of communication library, compilation and distributed start task. - -### openMPI introduction - -The OpenMPI project is an open source messaging interface implementation developed and maintained by a coalition of academic, research, and industry partners. Through it we can carry on the parallelization program design. - -### How is opensMPI implemented - -Here is a simple 4-thread MPI program example. - -{{
}} - - - -### MPI on Volcano - -Create `mpi-example.yaml`. - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: lm-mpi-job -spec: - minAvailable: 3 - schedulerName: volcano - plugins: - ssh: [] - svc: [] - tasks: - - replicas: 1 - name: mpimaster - policies: - - event: TaskCompleted - action: CompleteJob - template: - spec: - containers: - - command: - - /bin/sh - - -c - - | - MPI_HOST=`cat /etc/volcano/mpiworker.host | tr "\n" ","`; - mkdir -p /var/run/sshd; /usr/sbin/sshd; - mpiexec --allow-run-as-root --host ${MPI_HOST} -np 2 mpi_hello_world > /home/re; - image: volcanosh/example-mpi:0.0.1 - name: mpimaster - ports: - - containerPort: 22 - name: mpijob-port - workingDir: /home - restartPolicy: OnFailure - - replicas: 2 - name: mpiworker - template: - spec: - containers: - - command: - - /bin/sh - - -c - - | - mkdir -p /var/run/sshd; /usr/sbin/sshd -D; - image: volcanosh/example-mpi:0.0.1 - name: mpiworker - ports: - - containerPort: 22 - name: mpijob-port - workingDir: /home - restartPolicy: OnFailure ---- - -``` - -Delopy `mpi-example.yaml`. - -``` -kubectl apply -f mpi-example.yaml -``` - -View job performance under the cluster. - -``` -kubectl get pod -``` \ No newline at end of file diff --git a/content/en/docs/v1-11-0/plugins.md b/content/en/docs/v1-11-0/plugins.md deleted file mode 100644 index 66ca2c97..00000000 --- a/content/en/docs/v1-11-0/plugins.md +++ /dev/null @@ -1,176 +0,0 @@ -+++ -title = "Plugins" - -date = 2021-05-13 -lastmod = 2021-05-13 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Plugins" -[menu.v1-11-0] - parent = "scheduler" - weight = 3 -+++ - - - -### Gang - -{{
}} - -#### Overview - -The Gang scheduling strategy is one of the core scheduling algorithms of the Volcano-Scheduler. It meets the scheduling requirements of "All or nothing" in the scheduling process and avoids the waste of cluster resources caused by arbitrary scheduling of Pod. The Gang scheduler algorithm is to observe whether the scheduled number of Pods under Job meets the minimum number of runs. When the minimum number of runs of Job is satisfied, the scheduling action is executed for all Pods under Job; otherwise, it is not executed. - -#### Scenario - -The Gang scheduling algorithm based on the container group concept is well suited for scenarios that require multi-process collaboration. AI scenes often contain complex processes. Data Ingestion, Data Analysts, Data Splitting, trainers, Serving, Logging, etc., which require a group of containers to work together, are suitable for container-based Gang scheduling strategies. Multi-thread parallel computing communication scenarios under MPI computing framework are also suitable for Gang scheduling because master and slave processes need to work together. Containers under the container group are highly correlated, and there may be resource contention. The overall scheduling allocation can effectively solve the deadlock. - -In the case of insufficient cluster resources, the scheduling strategy of Gang can significantly improve the utilization of cluster resources. - -### Binpack - -#### Overview - -The goal of the BinPack scheduling algorithm is to fill as many existing nodes as possible (try not to allocate blank nodes). In the concrete implementation, BinPack scheduling algorithm scores the nodes that can be delivered, and the higher the score, the higher the resource utilization rate of nodes. Binpack algorithm can fill up the nodes as much as possible to close the application load to some nodes, which is very conducive to the automatic expansion capacity function of K8s cluster nodes. - -The BinPack algorithm is injected into the Volcano-Scheduler process as a plug-in and will be applied during the Pod stage of node selection. When calculating the Binpack algorithm, the Volcano-Scheduler considers the various resources requested by Pod and averages them according to the weights configured for each resource. The weight of each resource in the node score calculation is different, depending on the weight value configured by the administrator for each resource. Different plug-ins also need to assign different weights when calculating node scores, and the Scheduler also sets the score weights for BinPack plugins. - -#### Scenario - -The BinPack algorithm is good for small jobs that can fill as many nodes as possible. For example, the single query job in the big data scene, the order generation in the e-commerce seckill scene, the single identification job in the AI scene, and the high concurrency service scene on the Internet, etc. This scheduling algorithm can reduce the fragmentation in the node as much as possible, and reserve enough resource space on the idle machine for Pod which has applied for more resource requests, so as to maximize the utilization of idle resources under the cluster. - - - -### Priority - -{{
}} - -#### Overview - -The Priority Plugin provides the implementation of job, Task sorting, and PreempTablefn, a function that calculates sacrifice jobs. Job sorting according to priorityClassName, the task of sorting by priorityClassName, createTime, id in turn. - -#### Scenario - -When the cluster runs multiple jobs but is low on resources, and each Job has a different number of Pods waiting to be scheduled, if you use the Kubernetes default scheduler, the Job with more Pods will eventually get more of the cluster's resources. In this case, the Volcano-Scheduler provides algorithms that enable different jobs to share cluster resources in a fair-share. - -The Priority Plugin enables users to customize their job and task priorities, and to customize scheduling policies at different levels according to their own needs. Priority is arranged according to Job's PriorityClassName at the application level. For example, there are financial scenarios, Internet of Things monitoring scenarios and other applications requiring high real-time performance in the cluster, and the Priority Plugin can ensure that they are scheduled in Priority. - - - -### DRF - -{{
}} - -#### Overview - -The full name of DRF scheduling algorithm is Dominant Resource Fairness, which is a scheduling algorithm based on the container group Dominant Resource. Dominant Resource is the largest percentage of all required resources for a container group. The DRF algorithm selects the Dominant Resource that is the smallest in a series of container groups for priority scheduling. This can meet more job, not because a fat business, starve a large number of small business. DRF scheduling algorithm can ensure that many types of resources coexist in the environment, as far as possible to meet the fair principle of allocation. - -#### Scenario - -The DRF scheduling algorithm gives priority to the throughput of the business in the cluster and is suitable for batch small business scenarios such as a single AI training, a single big data calculation and a query. - - - -### Proportion - -#### Overview - -Proportion scheduling algorithm uses the concept of queue to control the Proportion of total resources allocated in the cluster. Each queue allocates a certain proportion of cluster resources. For example, there are three teams that share A pool of resources on A cluster: Team A uses up to 40% of the total cluster, Team B uses up to 30%, and Team C uses up to 30%. If the amount of work delivered exceeds the team's maximum available resources, there is a queue. - -#### Scenario - -Proportion scheduling algorithm improves the flexibility and elasticity of cluster scheduling. The most typical scenario is that when multiple development teams in a company share a cluster, this scheduling algorithm can handle the requirements of shared resource matching and isolation between different departments very well. In multi-service mixed scenarios, such as computation-intensive AI business, network IO-intensive MPI and HPC business, and storage-intensive big data business, Proportion scheduling algorithm can allocate shared resources according to demand through matching. - - - -### Task-topology - -#### Overview - -The task-topology algorithm is an algorithm that computes the priority of tasks and nodes based on the affinity and anti-affinity configuration between tasks within a Job. By configuring the affinity and anti-affinity policies between tasks within the Job and using the Task-Topology algorithm, tasks with affinity configurations can be scheduled to the same node first, and PODs with anti-affinity configurations to different nodes. - -#### Scenario - -node affinity： - -- Task-topology is important for improving computational efficiency in deep learning computing scenarios. Using the TensorFlow calculation as an example, configure the affinity between "ps" and "worker". Task-topology algorithm enables "ps" and "worker" to be scheduled to the same node as far as possible, so as to improve the efficiency of network and data interaction between "ps" and "worker", thus improving the computing efficiency. -- Tasks in HPC and MPI scenarios are highly synchronized and need high-speed network IO. - -Anti-affinity： - -- Take the TensorFlow calculation as an example, the anti-affinity between "ps" and "ps" - -- Master and slave backup of e-commerce service scene, data disaster tolerant, to ensure that there are spare jobs to continue to provide service after a job fails. - - - -### Predicates - -#### Overview - -The Predicate Plugin calls the PredicateGPU with pod and nodeInfo as parameters to evaluate and pre-select jobs based on the results. - -#### Scenario - -In AI scenarios where GPU resources are required, the Predicate Plugin can quickly filter out those that require the GPU for centralized scheduling. - - -### Nodeorder - -#### Overview - -The NodeOrder Plugin is a scheduling optimization strategy that scores nodes from various dimensions through simulated assignments to find the node that is best suited for the current job. The scoring parameters are configured by the user. The parameter contains the Affinity、reqResource、LeastReqResource、MostResource、balanceReqResouce. - -#### Scenario - -NodeOrder Plugin provides scoring criteria of multiple dimensions for scheduling, and the combination of different dimensions enables users to flexibly configure appropriate scheduling policies according to their own needs. - - - -### SLA - -#### Overview - -When users apply jobs to Volcano, they may need adding some particular constraints to job, for example, longest Pending time aiming to prevent job from starving. And these constraints can be regarded as Service Level Agreement (SLA) which are agreed between volcano and user. So sla plugin is provided to receive and realize SLA settings for both individual job and whole cluster. - -#### Scenario - -Users can customize SLA related parameters in their own cluster according to business needs. For example, for clusters with high real-time service requirements, JobWaitingTime can be set as small as possible. For clusters with bulk computing jobs, JobWaitingTime can be set to larger. The parameters of a specific SLA and the optimization of the parameters need to be combined with the specific business and related performance measurement results. - -### TDM - -#### Overview - -The full name of TDM is Time Division Multiplexing. In a co-located environment, some nodes are in both Kubernetes cluster and Yarn cluster. For these nodes, Kubernetes and Yarn cluster can use these resource by time-sharing multiplexing.The TDM Plugin marks these nodes as `revocable nodes`. TDM plugin will try to dispatch `preemptable task` to `revocable node` in node revocable time and evict the `preemptable task` from `revocable node` out of revocable time.. TDM Plugin improves the time-division multiplexing ability of node resources in the scheduling process of Volcano. - -#### Scenario - -In ToB business, cloud vendors provide cloud-based resources for merchants, and different merchants adopt different container arrangement frameworks (Kubernetes/YARN, etc.). TDM Plugin improves the time-sharing efficiency of common node resources and further improves the utilization rate of resources. - - - -### Numa-aware - -#### Overview - -When the node runs many CPU-bound pods, the workload can move to different CPU cores depending on whether the pod is throttled and which CPU cores are available at scheduling time. Many workloads are not sensitive to this migration and thus work fine without any intervention. However, in workloads where CPU cache affinity and scheduling latency significantly affect workload performance, the kubelet allows alternative CPU management policies to determine some placement preferences on the node. - -The CPU Manager and the Topology Manager are all Kubelet components, However There is the following limitation: - -- The scheduler is not topology-aware. so it is possible to be scheduled on a node and then fail on the node due to the Topology Manager. this is unacceptable for TensorFlow job. If any worker or ps failed on node, the job will fail. -- The managers are node-level that results in an inability to match the best node for NUMA topology in the whole cluster. - -The Numa-Aware Plugin aims to address these limitations. - -- Support cpu resource topology scheduling. -- Support pod-level topology policies. - -#### Scenario - -Common scenarios for NUMA-Aware are computation-intensive jobs that are sensitive to CPU parameters, scheduling delays. Such as scientific calculation, video decoding, animation rendering, big data offline processing and other specific scenes. - - diff --git a/content/en/docs/v1-11-0/pp_on_volcano.md b/content/en/docs/v1-11-0/pp_on_volcano.md deleted file mode 100644 index 2eb94a54..00000000 --- a/content/en/docs/v1-11-0/pp_on_volcano.md +++ /dev/null @@ -1,242 +0,0 @@ -+++ -title = "PaddlePaddle on Volcano" - -date = 2021-06-29 -lastmod = 2021-06-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "PaddlePaddle" -[menu.v1-11-0] - parent = "ecosystem" - weight = 6 - -+++ - - - -### paddlepaddle introduction - -PaddlePaddle is a deep learning framework open source by Baidu in September 2016, aiming to provide a safe, efficient, flexible, easy to use and extensible deep learning platform. - -In October 2018, the Paddle Fluid 1.0 version was released by the Paddle Fluid team, which comprehensively upgraded core capabilities such as neural network description, large-scale distributed training, and high-performance inference engine. For example, in the latest Paddle Fluid 1.5.2, the Paddle fly supports data parallelism, model parallelism, pipeline parallelism, and other parallel modes. Parameter server architecture and point-to-point synchronous training architecture fully support large-scale training on hardware resource devices such as CPU and GPU [1]. - -### paddlepaddle on valcano - -Upload `ctr-Volcano.yaml` on the cluster node as follows - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: ctr-volcano -spec: - minAvailable: 4 - schedulerName: volcano - policies: - - event: PodEvicted - action: RestartJob - - event: PodFailed - action: RestartJob - tasks: - - replicas: 2 - name: pserver - template: - metadata: - labels: - paddle-job-pserver: fluid-ctr - spec: - imagePullSecrets: - - name: default-secret - volumes: - - hostPath: - path: /home/work/ - type: "" - name: seqdata - containers: - - image: volcanosh/edlctr:v1 - command: - - paddle_k8s - - start_fluid - imagePullPolicy: IfNotPresent - name: pserver - volumeMounts: - - mountPath: /mnt/seqdata - name: seqdata - resources: - limits: - cpu: 10 - memory: 30Gi - ephemeral-storage: 10Gi - requests: - cpu: 1 - memory: 100M - ephemeral-storage: 1Gi - env: - - name: GLOG_v - value: "0" - - name: GLOG_logtostderr - value: "1" - - name: TOPOLOGY - value: "" - - name: TRAINER_PACKAGE - value: /workspace - - name: NAMESPACE - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.namespace - - name: POD_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: POD_NAME - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.name - - name: PADDLE_CURRENT_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: PADDLE_JOB_NAME - value: fluid-ctr - - name: PADDLE_IS_LOCAL - value: "0" - - name: PADDLE_TRAINERS_NUM - value: "2" - - name: PADDLE_PSERVERS_NUM - value: "2" - - name: FLAGS_rpc_deadline - value: "36000000" - - name: ENTRY - value: cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1 - - name: PADDLE_PORT - value: "30236" - - name: LD_LIBRARY_PATH - value: /usr/local/lib:/usr/local/nvidia/lib64:/usr/local/rdma/lib64:/usr/lib64/mlnx_ofed/valgrind - - name: PADDLE_TRAINING_ROLE - value: PSERVER - - name: TRAINING_ROLE - value: PSERVER - restartPolicy: OnFailure - - replicas: 2 - policies: - - event: TaskCompleted - action: CompleteJob - name: trainer - template: - metadata: - labels: - paddle-job: fluid-ctr - spec: - imagePullSecrets: - - name: default-secret - volumes: - - hostPath: - path: /home/work/ - type: "" - name: seqdata - containers: - - image: volcanosh/edlctr:v1 - command: - - paddle_k8s - - start_fluid - imagePullPolicy: IfNotPresent - name: trainer - volumeMounts: - - mountPath: /mnt/seqdata - name: seqdata - resources: - limits: - cpu: 10 - memory: 30Gi - ephemeral-storage: 10Gi - requests: - cpu: 1 - memory: 100M - ephemeral-storage: 10Gi - env: - - name: GLOG_v - value: "0" - - name: GLOG_logtostderr - value: "1" - - name: TOPOLOGY - - name: TRAINER_PACKAGE - value: /workspace - - name: CPU_NUM - value: "2" - - name: NAMESPACE - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.namespace - - name: POD_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: POD_NAME - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.name - - name: PADDLE_CURRENT_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: PADDLE_JOB_NAME - value: fluid-ctr - - name: PADDLE_IS_LOCAL - value: "0" - - name: FLAGS_rpc_deadline - value: "36000000" - - name: PADDLE_PORT - value: "30236" - - name: PADDLE_PSERVERS_NUM - value: "2" - - name: PADDLE_TRAINERS_NUM - value: "2" - - name: PADDLE_TRAINING_ROLE - value: TRAINER - - name: TRAINING_ROLE - value: TRAINER - - name: LD_LIBRARY_PATH - value: /usr/local/lib:/usr/local/nvidia/lib64:/usr/local/rdma/lib64:/usr/lib64/mlnx_ofed/valgrind - - name: ENTRY - value: cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1 - restartPolicy: OnFailure - -``` - -Deploy under the cluster terminal. - -``` -kubectl apply -f ctr-volcano.yaml -``` - -Query if the job is running properly.If the PodGroup cannot meet the scheduling conditions, check that the cluster has sufficient resources available. - -``` -kubectl get podgroup -kubectl describe podgroup ctr-volcano -kubectl get pods | grep ctr-volcano -``` - -You can select a PServer task to view the log. - -``` -kubectl logs ctr-volcano-pserver-0 -``` - -Select a Tariner task to view the log. - -``` -kubectl logs ctr-volcano-trainer-0 -``` \ No newline at end of file diff --git a/content/en/docs/v1-11-0/referrals.md b/content/en/docs/v1-11-0/referrals.md deleted file mode 100644 index 4f2a1ceb..00000000 --- a/content/en/docs/v1-11-0/referrals.md +++ /dev/null @@ -1,88 +0,0 @@ -+++ -title = "Recommended Configurations for Common Scenarios" - -date = 2021-04-07 -lastmod = 2021-04-07 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Scenario & Configuration" -[menu.v1-11-0] - parent = "scheduler" - weight = 4 - -+++ - - - -### Background - -{{
}} - -Deep learning generally involves four stages: data acquisition and processing, model training and evolution, model deployment, and model evaluation [1]. At the beginning of an enterprise's involvement in deep learning, due to the lack of relevant experience and resources for deep learning, the team members usually choose to complete the above steps manually step by step in order to quickly meet the immediate needs of the project development period. However, no matter how domain-specific a deep learning expert may have, scaling up a project without the support of an infrastructure platform becomes extremely difficult. Large-scale operations in a production environment require more rapid replication and iterative upgrades than local operations. Production environment has higher requirements for reliability of data, reproducibility of model training, scalability of training model, reliability of automated deployment and operation and maintenance. Therefore, when the enterprise grows to a certain scale, the enterprise will want to build a stable scientific computing platform, so that the upper business personnel can focus on algorithm optimization and business development. - -In general, the construction of a deep learning platform will encounter many challenges, mainly reflected in the following aspects: - -- Data management automation -- The efficient use of resources -- Mask the complexity of the underlying technology - - - -##### Data management - -In a business scenario of deep learning, practitioners spend a significant amount of time capturing and transforming the data needed to build the model. As business scenarios expand, the automation of data transformation becomes a bottleneck in the efficiency of model building. In the process of data processing, new data will be generated, and these data will not only be used for this training, but also for the subsequent reasoning process. And the newly generated data does not need to be passed to the data source, but instead wants to be placed in a new storage space. This requires the underlying platform to provide a scalable storage system. Flexible, extensible and secure data storage system will greatly promote the improvement of data management ability. - -##### The efficient use of resources - -Deep learning-related applications are resource-intensive, and the use of resources fluctuates greatly, with peaks and valleys. Fast access to computing resources when the application starts running; At the end of the application, it is very important to recycle the unsuitable computing resources for the improvement of resource utilization. The types and time of computing resources used in data processing, model training and reasoning are different, which requires a flexible resource supply mechanism provided by the computing platform. Deep learning tasks require a large amount of computing resources, so it is impossible to build computing resources for each user separately. The computing platform should be able to guarantee the multi-lease of resource use, allowing multiple users to use the computing resource pool at the same time, rather than being monopolized by a few users. - -##### Shield the underlying technical complexity - -AI practitioners focus on model building and product development, ignoring the importance of infrastructure to AI development. Machine Learning/Deep Learning's own growing technology stack of users includes computing frameworks for machine learning such as TensorFlow, the Spark data processing engine, and underlying drivers such as CUDA. Manually managing these dependencies can consume significant resources and effort on the part of practitioners. - - - -Based on the above goals and challenges, deep learning and other AI computing choose to use containers and Kubernetes to build and manage deep learning platforms. Need to achieve user isolation, queuing mechanism, user rights management, resource management, container ceiling management and many other functions. - - - -### Common scenarios - -1. Priority issues:There are many roles of algorithm engineers in the company, such as interns, full-time employees and senior engineers. They all do the same thing and need GPU resources to do experiments or training. However, when resources are tight, they need to queue according to priority. Even in the same position, there is preemption for urgent assignments (such as paper deadlines, project launches). The priority level of the task needs to be reported by the engineer. -2. storage of data:The data sets of deep learning scenarios are very large, so it is necessary to optimize the design at the storage level to meet this requirement. When computing, if one person trains a model, then simple storage is OK. However, if there are 100 people and each person has different training models and data sets, and needs to run multiple tasks, what kind of storage can help you solve this problem? This is a realistic problem facing us. How storage ADAPTS to the ever-expanding computing needs of users is a matter of concern to architects -3. Parallel computing problem:A general requirement of deep learning platform based on K8S is to support multi-machine scheduling and make model training as fast as possible through multi-machine parallel training. -4. User isolation problem :For larger companies, there are many business departments, which require multi-user management and multi-tenant isolation. (User isolation problem) -5. Batch scheduling requirements for pod under deep learning scenarios. -6. Resource preemption problem:The upper limit on the amount of scarce resources (GPUs) used and the upper limit on how long they can be used. -7. Reservation:The details of the underlying application resources are shielded. It only cares about the total amount of available GPU, not the distribution of GPU. - - - -## referrals of the configuration - -- Priority issues: Customize the priority hierarchy. Focus on the `PriorityClassName` of the job itself. This field represents the priority of the job and is in effect for preemptive scheduling and prioritization. - -- Data storage: Recommends the `block storage` service in extremely performing businesses such as databases. Recommend the use of `object storage` service in the business scenarios of HPC, big data and AI. - -- High performance computing, model training parallelism :WRF, Kubegene, MPI high performance computing scenarios, multi-machine parallel training. - - 1. Job Plugin: `ssh` provides secret login, and `svc` provides network information needed for job operation, such as host file, headless service, etc, to provide automatic configuration for calculating cluster parameters. - - 2. `Policies field`: Once any `pod` has been killed, restart the entire job `restartJob`/`compeletedJob`. Because of this HPC scenario, Gang-based training job, if a worker fails, it is usually pointless for the entire job to run. - - 3. This scenario will generally define multiple `tasks`, 1 master + multiple workers. - -- Multi-tenancy, isolation issues :TensorFlow has no concept of tenants. The new resource splitting tool `Queue` introduced by Volcano is configured around Queue in various ways. To ensure the multi-lease of the cluster resources, the surplus of the cluster resources, and the flexibility of the tenants to use the cluster resources. When the cluster resources are redivided, the tenant resource quota can be guaranteed through Reclaim. Configure `preempt, reclaim` in `Volcano-Scheduler-ConfigMap`; configure `proportion` in plugins. - -- Batch scheduling: The `Gang Plugin` meets the requirements of pod batch scheduling for deep learning scenarios. The class field `minAvailable`, the main indicator of the Gang scheduling algorithm. The plugins field needs to use `env` and `svc`. The definition of Policies is similar to the high performance computing scenario where a `pod` ejection restarts the job. - - - -Reference: - -[1][Challenges facing AI infrastructure ](https://zhuanlan.zhihu.com/p/75634193) - diff --git a/content/en/docs/v1-11-0/schduler_introduction.md b/content/en/docs/v1-11-0/schduler_introduction.md deleted file mode 100644 index 838941ff..00000000 --- a/content/en/docs/v1-11-0/schduler_introduction.md +++ /dev/null @@ -1,103 +0,0 @@ -+++ -title = "Introduction" - - -date = 2019-01-28 -lastmod = 2020-09-01 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Overview" -[menu.v1-11-0] - parent = "scheduler" - weight = 1 - -+++ - -## Introduction -Volcano scheduler is the component responsible for pod scheduling. It consists of a series of actions and plugins. Actions define the action that should be executed in every step. Plugins provide the action algorithm details in different scenarios. -Volcano scheduler is highly scalable. You can specify and implement actions and plugins based on your requirements. -## Workflow -{{
}} - - -Volcano scheduler works as follows: - -1. Watches for and caches the jobs submitted by the client. -2. Opens a session periodically. A scheduling cycle begins. -3. Sends jobs that are not scheduled in the cache to the to-be-scheduled queue in the session. -4. Traverses all jobs to be scheduled. Executes enqueue, allocate, preempt, reclaim, and backfill actions in the order they are defined, and finds the most suitable node for each job. Binds the job to the node. The specific algorithm logic executed in the action depends on the implementation of each function in the registered plugins. -5. Closes this session. - -## Actions -### enqueue -The enqueue action is responsible for filtering out the tasks that meet the scheduling requirements based on a series of filtering algorithms and sending them to the to-be-scheduled queue. After the action is executed, the status of the task changes from `pending` to `inqueue`. -### allocate -The allocate action is responsible for selecting the most suitable node based on a series of prediction and optimization algorithms. -### preempt -The preempt action is responsible for preemptive scheduling of high priority tasks in the same queue according to priority rules. -### reclaim -The reclaim action is responsible for reclaiming the resources allocated to the cluster based on the queue weight when a new task enters the queue and the cluster resources cannot meet the needs of the queue. -### backfill -The backfill action is responsible for backfilling the tasks in the `pending` state into the cluster node to maximize the resource utilization of the node. - -## Plugins -### gang -The gang plugin considers that tasks not in the `Ready` state (including Binding, Bound, Running, Allocated, Succeed, and Pipelined) have a higher priority. It checks whether the resources allocated to the queue can meet the resources required by the task to run `minavailable` pods after trying to evict some pods and reclaim resources. If yes, the gang plugin will evict some pods. -### conformance -The conformance plugin considers that the tasks in namespace `kube-system` have a higher priority. These tasks will not be preempted. -### DRF -The DRF plugin considers that tasks with fewer resources have a higher priority. It attempts to calculate the total amount of resources allocated to the preemptor and preempted tasks, and triggers the preemption when the preemptor task has less resources. -### nodeorder -The nodeorder plugin scores all nodes for a task by using a series of scoring algorithms. The node with the highest score is considered to be the most suitable node for the task. -### predicates -The predicates plugin determines whether a task is bound to a node by using a series of evaluation algorithms. -### priority -The priority plugin compares the priorities of two jobs or tasks. For two jobs, it decides whose priority is higher by comparing `job.spec.priorityClassName`. For two tasks, it decides whose priority is higher by comparing `task.priorityClassName`, `task.createTime`, and `task.id` in order. -## Configuration -Volcano scheduler is highly scalable because of its composite pattern design. Users can decide which actions and plugins to use according to their needs, and they can also implement customization by calling the action or plugin interfaces. The scheduler configuration is located in the ConfigMap named **volcano-scheduler-configmap**, which is mounted as a volume into the "/volcano.scheduler" directory in the scheduler container. -### How to Get Configuration of Volcano Scheduler -* Get the ConfigMap named `volcano-scheduler-configmap`. - -```shell -# kubectl get configmap -nvolcano-system -NAME DATA AGE -volcano-scheduler-configmap 1 6d2h -``` - -* View the details of the data part in the ConfigMap. - -```shell -# kubectl get configmap volcano-scheduler-configmap -nvolcano-system -oyaml -apiVersion: v1 -data: - volcano-scheduler.conf: | - actions: "enqueue, allocate, backfill" - tiers: - - plugins: - - name: priority - - name: gang - - name: conformance - - plugins: - - name: drf - - name: predicates - - name: proportion - - name: nodeorder - - name: binpack -kind: ConfigMap -metadata: - annotations: - kubectl.kubernetes.io/last-applied-configuration: | - {"apiVersion":"v1","data":{"volcano-scheduler.conf":"actions: \"enqueue, allocate, backfill\"\ntiers:\n- plugins:\n - name: priority\n - name: gang\n - name: conformance\n- plugins:\n - name: drf\n - name: predicates\n - name: proportion\n - name: nodeorder\n - name: binpack\n"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"volcano-scheduler-configmap","namespace":"volcano-system"}} - creationTimestamp: "2020-08-15T04:01:02Z" - name: volcano-scheduler-configmap - namespace: volcano-system - resourceVersion: "266" - selfLink: /api/v1/namespaces/volcano-system/configmaps/volcano-scheduler-configmap - uid: 1effe4d6-126c-42d6-a3a4-b811075c30f5 -``` - -`actions` and `tiers` are included in `volcano-scheduler.conf`. In `actions`, the comma is used as a separator to configure the actions to be executed by the scheduler. Note that the scheduler will execute the actions in the order that they are configured. Volcano itself will not check the rationality of the order. The list of plugins configured in `tiers` is the plugins registered with the scheduler. The specific algorithms defined in plugins will be called in `actions`. diff --git a/content/en/docs/v1-11-0/spark_on_volcano.md b/content/en/docs/v1-11-0/spark_on_volcano.md deleted file mode 100644 index 47bb4b2f..00000000 --- a/content/en/docs/v1-11-0/spark_on_volcano.md +++ /dev/null @@ -1,101 +0,0 @@ -+++ -title = "Spark on Volcano" - -date = 2021-06-29 -lastmod = 2021-06-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Spark" -[menu.v1-11-0] - parent = "ecosystem" - weight = 8 - -+++ - - - -### Spark introduction - -Spark is a fast and versatile big data clustering computing system. It provides high-level APIs for Scala, Java, Python, and R, as well as an optimization engine that supports a generic computational graph for data analysis. It also supports a rich set of advanced tools, including Spark SQL for SQL and Dataframes, MLLib for machine learning, GraphX for graphics processing, and Spark Streaming for Streaming. - -### Spark on Volcano - -Currently, there are two ways to support the integration of Spark on Kubernetes and volcano. -- Spark on Kubernetes native support: maintained by the [Apache Spark community](https://github.com/apache/spark) and Volcano community -- Spark Operator support: maintained by the [GoogleCloudPlatform community](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) and Volcano community - -#### Spark on Kubernetes native support (spark-submit) - -Spark on Kubernetes with Volcano as a custom scheduler is supported since Spark v3.3.0 and Volcano v1.5.1. See more detail in [link](https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-volcano-as-customized-scheduler-for-spark-on-kubernetes). - -#### Spark Operator support (spark-operator) - -Install Spark-Operator through Helm. - -``` -$ helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator - -$ helm install my-release spark-operator/spark-operator --namespace spark-operator --create-namespace -``` - -To ensure that the Spark-Operator is up and running, check with the following directive. - -``` -$ kubectl get po -nspark-operator -``` - -Here's the official `spark.yaml`. - -``` -apiVersion: "sparkoperator.k8s.io/v1beta2" -kind: SparkApplication -metadata: - name: spark-pi - namespace: default -spec: - type: Scala - mode: cluster - image: "gcr.io/spark-operator/spark:v3.0.0" - imagePullPolicy: Always - mainClass: org.apache.spark.examples.SparkPi - mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar" - sparkVersion: "3.0.0" - batchScheduler: "volcano" #Note: the batch scheduler name must be specified with `volcano` - restartPolicy: - type: Never - volumes: - - name: "test-volume" - hostPath: - path: "/tmp" - type: Directory - driver: - cores: 1 - coreLimit: "1200m" - memory: "512m" - labels: - version: 3.0.0 - serviceAccount: spark - volumeMounts: - - name: "test-volume" - mountPath: "/tmp" - executor: - cores: 1 - instances: 1 - memory: "512m" - labels: - version: 3.0.0 - volumeMounts: - - name: "test-volume" - mountPath: "/tmp" -``` - -Deploy the Spark application and see the status. - -``` -$ kubectl apply -f spark-pi.yaml -$ kubectl get SparkApplication -``` diff --git a/content/en/docs/v1-11-0/tf_on_volcano.md b/content/en/docs/v1-11-0/tf_on_volcano.md deleted file mode 100644 index 4268ab04..00000000 --- a/content/en/docs/v1-11-0/tf_on_volcano.md +++ /dev/null @@ -1,111 +0,0 @@ -+++ -title = "TensorFlow on Volcano" - -date = 2021-04-07 -lastmod = 2021-04-07 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "TensorFlow" -[menu.v1-11-0] - parent = "ecosystem" - weight = 7 - -+++ - - - -### TensorFlow introduction - -TensorFlow is a symbolic mathematical system based on data flow programming, which is widely used in programming and realization of various machine learning algorithms. Its predecessor is DistBelief, a neural network algorithm library of Google. - -### TensorFlow on Volcano - -PS-worker model: Parameter Server performs model-related services, Work Server trains related services, inference calculation, gradient calculation, etc[1]. - -{{
}} - -TensorFlow on Kubernetes has many problems: - -- Resource isolation. -- Lack of GPU scheduling, Gang Schuler. -- Process Legacy. -- Training log is not convenient to save. - -Create `tftest.yaml`. - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: tensorflow-dist-mnist -spec: - minAvailable: 3 - schedulerName: volcano - plugins: - env: [] - svc: [] - policies: - - event: PodEvicted - action: RestartJob - queue: default - tasks: - - replicas: 1 - name: ps - template: - spec: - containers: - - command: - - sh - - -c - - | - PS_HOST=`cat /etc/volcano/ps.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - export TF_CONFIG={\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"ps\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"}; - python /var/tf_dist_mnist/dist_mnist.py - image: volcanosh/dist-mnist-tf-example:0.0.1 - name: tensorflow - ports: - - containerPort: 2222 - name: tfjob-port - resources: {} - restartPolicy: Never - - replicas: 2 - name: worker - policies: - - event: TaskCompleted - action: CompleteJob - template: - spec: - containers: - - command: - - sh - - -c - - | - PS_HOST=`cat /etc/volcano/ps.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - export TF_CONFIG={\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"worker\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"}; - python /var/tf_dist_mnist/dist_mnist.py - image: volcanosh/dist-mnist-tf-example:0.0.1 - name: tensorflow - ports: - - containerPort: 2222 - name: tfjob-port - resources: {} - restartPolicy: Never -``` - -Deploy `tftest.yaml`. - -``` -kubectl apply -f tftest.yaml -``` - -View job health. - -``` -kubectl get pod -``` \ No newline at end of file diff --git a/content/en/docs/v1-11-0/tutorials.md b/content/en/docs/v1-11-0/tutorials.md deleted file mode 100644 index b4c9c62e..00000000 --- a/content/en/docs/v1-11-0/tutorials.md +++ /dev/null @@ -1,277 +0,0 @@ -+++ -title = "Tutorials" - - -date = 2019-01-28 -lastmod = 2020-08-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Tutorials" -[menu.v1-11-0] - parent = "getting-started" - weight = 2 -+++ - -Here is a simple example of how to use Volcano with CRD resources. - -### Step: 1 -Create a queue named "test". -```shell -# kubectl create -f queue.yaml -// queue.yaml -apiVersion: scheduling.volcano.sh/v1beta1 -kind: Queue -metadata: - name: test -spec: - weight: 1 - reclaimable: false - capability: - cpu: 2 -``` - -### Step: 2 -Create a VolcanoJob named "job-1". -```shell -# kubectl create -f vcjob.yaml -// vcjob.yaml -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: job-1 -spec: - minAvailable: 1 - schedulerName: volcano - queue: test - policies: - - event: PodEvicted - action: RestartJob - tasks: - - replicas: 1 - name: nginx - policies: - - event: TaskCompleted - action: CompleteJob - template: - spec: - containers: - - command: - - sleep - - 10m - image: nginx:latest - name: nginx - resources: - requests: - cpu: 1 - limits: - cpu: 1 - restartPolicy: Never -``` - -### Step: 3 -Check the status of the custom job. -```shell -# kubectl get vcjob job-1 -oyaml -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - creationTimestamp: "2020-01-18T12:59:37Z" - generation: 1 - managedFields: - - apiVersion: batch.volcano.sh/v1alpha1 - fieldsType: FieldsV1 - fieldsV1: - f:spec: - .: {} - f:minAvailable: {} - f:policies: {} - f:queue: {} - f:schedulerName: {} - manager: kubectl - operation: Update - time: "2020-08-18T12:59:37Z" - - apiVersion: batch.volcano.sh/v1alpha1 - fieldsType: FieldsV1 - fieldsV1: - f:spec: - f:tasks: {} - f:status: - .: {} - f:minAvailable: {} - f:running: {} - f:state: - .: {} - f:lastTransitionTime: {} - f:phase: {} - manager: vc-controller-manager - operation: Update - time: "2020-08-18T12:59:45Z" - name: job-1 - namespace: default - resourceVersion: "850500" - selfLink: /apis/batch.volcano.sh/v1alpha1/namespaces/default/jobs/job-1 - uid: 215409ec-7337-4abf-8bea-e6419defd688 -spec: - minAvailable: 1 - policies: - - action: RestartJob - event: PodEvicted - queue: test - schedulerName: volcano - tasks: - - name: nginx - policies: - - action: CompleteJob - event: TaskCompleted - replicas: 1 - template: - spec: - containers: - - command: - - sleep - - 10m - image: nginx:latest - name: nginx - resources: - limits: - cpu: 1 - requests: - cpu: 1 -status: - minAvailable: 1 - running: 1 - state: - lastTransitionTime: "2020-08-18T12:59:45Z" - phase: Running -``` - -### Step: 4 -Check the status of PodGroup named "job-1". -```shell -# kubectl get podgroup job-1 -oyaml -apiVersion: scheduling.volcano.sh/v1beta1 -kind: PodGroup -metadata: - creationTimestamp: "2020-08-18T12:59:37Z" - generation: 5 - managedFields: - - apiVersion: scheduling.volcano.sh/v1beta1 - fieldsType: FieldsV1 - fieldsV1: - f:metadata: - f:ownerReferences: - .: {} - k:{"uid":"215409ec-7337-4abf-8bea-e6419defd688"}: - .: {} - f:apiVersion: {} - f:blockOwnerDeletion: {} - f:controller: {} - f:kind: {} - f:name: {} - f:uid: {} - f:spec: - .: {} - f:minMember: {} - f:minResources: - .: {} - f:cpu: {} - f:queue: {} - f:status: {} - manager: vc-controller-manager - operation: Update - time: "2020-08-18T12:59:37Z" - - apiVersion: scheduling.volcano.sh/v1beta1 - fieldsType: FieldsV1 - fieldsV1: - f:status: - f:conditions: {} - f:phase: {} - f:running: {} - manager: vc-scheduler - operation: Update - time: "2020-08-18T12:59:45Z" - name: job-1 - namespace: default - ownerReferences: - - apiVersion: batch.volcano.sh/v1alpha1 - blockOwnerDeletion: true - controller: true - kind: Job - name: job-1 - uid: 215409ec-7337-4abf-8bea-e6419defd688 - resourceVersion: "850501" - selfLink: /apis/scheduling.volcano.sh/v1beta1/namespaces/default/podgroups/job-1 - uid: ea5b4f87-b750-440b-a41a-5c9944a7ae43 -spec: - minMember: 1 - minResources: - cpu: "1" - queue: test -status: - conditions: - - lastTransitionTime: "2020-08-18T12:59:38Z" - message: '1/0 tasks in gang unschedulable: pod group is not ready, 1 minAvailable.' - reason: NotEnoughResources - status: "True" - transitionID: 606145d1-660f-4e01-850d-ed556cebc098 - type: Unschedulable - - lastTransitionTime: "2020-08-18T12:59:45Z" - reason: tasks in gang are ready to be scheduled - status: "True" - transitionID: 57e6ba9e-55cc-47ce-a37e-d8bddd99d54b - type: Scheduled - phase: Running - running: 1 -``` - -### Step: 5 -Check the status of queue "test". -```shell -# kubectl get queue test -oyaml -apiVersion: scheduling.volcano.sh/v1beta1 -kind: Queue -metadata: - creationTimestamp: "2020-08-18T12:59:30Z" - generation: 1 - managedFields: - - apiVersion: scheduling.volcano.sh/v1beta1 - fieldsType: FieldsV1 - fieldsV1: - f:spec: - .: {} - f:capability: {} - f:reclaimable: {} - f:weight: {} - manager: kubectl - operation: Update - time: "2020-08-18T12:59:30Z" - - apiVersion: scheduling.volcano.sh/v1beta1 - fieldsType: FieldsV1 - fieldsV1: - f:spec: - f:capability: - f:cpu: {} - f:status: - .: {} - f:running: {} - f:state: {} - manager: vc-controller-manager - operation: Update - time: "2020-08-18T12:59:39Z" - name: test - resourceVersion: "850474" - selfLink: /apis/scheduling.volcano.sh/v1beta1/queues/test - uid: b9c9ee54-5ef8-4784-9bec-7a665acb1fde -spec: - capability: - cpu: 2 - reclaimable: false - weight: 1 -status: - running: 1 - state: Open -``` \ No newline at end of file diff --git a/content/en/docs/v1-12-0/architecture.md b/content/en/docs/v1-12-0/architecture.md deleted file mode 100644 index ba369fb2..00000000 --- a/content/en/docs/v1-12-0/architecture.md +++ /dev/null @@ -1,43 +0,0 @@ -+++ -title = "Architecture" - - -date = 2019-01-28 -lastmod = 2020-08-28 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Architecture" -[menu.v1-12-0] - parent = "home" - weight = 2 -+++ - -## Overall Architecture - - -{{
}} - - -Volcano is designed for high-performance workloads running on Kubernetes. It follows the design and mechanisms of Kubernetes. - - -{{
}} - - -Volcano consists of **scheduler** / **controllermanager** / **admission** / **vcctl**: - -##### Scheduler -Volcano Scheduler schedules jobs to the most suitable node based on actions and plug-ins. Volcano supplements Kubernetes to support multiple scheduling algorithms for jobs. - -##### ControllerManager (CM) -Volcano CMs manage the lifecycle of Custom Resource Definitions (CRDs). You can use the **Queue CM**, **PodGroup CM**, and **VCJob CM**. - -##### Admission -Volcano Admission is responsible for the CRD API validation. - -##### vcctl -Volcano vcctl is the command line client for Volcano. diff --git a/content/en/docs/v1-12-0/cli.md b/content/en/docs/v1-12-0/cli.md deleted file mode 100644 index 99ac3458..00000000 --- a/content/en/docs/v1-12-0/cli.md +++ /dev/null @@ -1,74 +0,0 @@ -+++ -title = "CLI" - - -date = 2019-01-28 -lastmod = 2020-09-01 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Commandline" -[menu.v1-12-0] - parent = "cli" - weight = 1 -+++ - -## Introduction -A Command Line Interface (CLI) is provided for you to manage resources. -## Configuration - -1. You can obtain the latest executable file by cloning the code from GitHub and running the following command in the root directory of the project: -``` -# make vcctl -``` -2. Copy the executable file to $PATH. You then can execute it anywhere. - -## Command Line List -### Listing all jobs -vcctl job list - -```html -# vcctl job list -Name Creation Phase JobType Replicas Min Pending Running Succeeded Failed Unknown RetryCount -job-1 2020-09-01 Running Batch 1 1 0 1 0 0 0 0 -``` - -### Deleting a specific job -vcctl job delete --name job-name [--namespace job-namespace] - -```html -# vcctl delete job --name job-1 --namespaces default -delete job job-1 successfully -``` - -### Suspending a job -vcctl job suspend --name job-name [--namespace job-namespace] - -```html -# vcctl job suspend --name job-1 --namespace default -``` - -### Resuming a job (opposite to "vcctl job suspend") -vcctl job resume --name job-name [--namespace job-namespace] - -```html -# vcctl job resume --name job-1 --namespace default -``` - -### Running a job -vcctl job run --name job-name [--namespace job-namespace] - -```html -# vcctl job run --name job-1 --namespace default -``` - -## Note -For more information about Volcano command lines, run the following commands: - -```html -# vcctl -h -# vcctl [command] -h -``` diff --git a/content/en/docs/v1-12-0/flink_on_volcano.md b/content/en/docs/v1-12-0/flink_on_volcano.md deleted file mode 100644 index 90e994dc..00000000 --- a/content/en/docs/v1-12-0/flink_on_volcano.md +++ /dev/null @@ -1,279 +0,0 @@ -+++ -title = "Flink on Volcano" - -date = 2021-06-29 -lastmod = 2021-06-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Flink" -[menu.v1-12-0] - parent = "ecosystem" - weight = 2 - -+++ - - - -### Flink introduction - -Apache Flink is an open-source streaming framework developed by the Apache Software Foundation. At its core, Apache Flink is a distributed streaming data streaming engine written in Java and Scala. Flink executes any stream data program in data parallelism and pipelining. Flink's pipelined runtime system can execute both batch and stream programs. In addition, the Flink runtime itself supports the execution of iterative algorithms. - -### The premise condition - -Make sure the deployed Kubernetes, Kubectl, Volcano are installed correctly. - -### Deployment process - -##### 1. Download - -To run Flink, which requires a Java 8 or 11 environment, use the following instructions to determine the Java version. - -```bash -java -version -``` - -Download the package and go to the directory. - -``` -$ wget https://www.apache.org/dyn/closer.lua/flink/flink-1.12.2/flink-1.12.2-src.tgz -$ cd flink-1.12.2 -``` - -##### 2. Start a Cluster - -Running the script completes the deployment of Flink on the cluster. - -```bash -$ ./bin/start-cluster.sh -``` - -##### 3. Submit a job - -Submit the job using the following instructions. - -```bash -$ ./bin/flink run examples/streaming/WordCount.jar -$ tail log/flink-*-taskexecutor-*.out -``` - -### Flink on Volcano - -##### 1. The deployment of the component - -Deploying a Flink Cluster requires creating two deploys, a Service, and a ConfigMap. The scheduling strategy is Volcano.The contents of `flink-configuration-configmap.yaml` are shown below. - -``` -apiVersion: v1 -kind: ConfigMap -metadata: - name: flink-config - labels: - app: flink -data: - flink-conf.yaml: |+ - jobmanager.rpc.address: flink-jobmanager - taskmanager.numberOfTaskSlots: 2 - blob.server.port: 6124 - jobmanager.rpc.port: 6123 - taskmanager.rpc.port: 6122 - queryable-state.proxy.ports: 6125 - jobmanager.memory.process.size: 1600m - taskmanager.memory.process.size: 1728m - parallelism.default: 2 - log4j-console.properties: |+ - # This affects logging for both user code and Flink - rootLogger.level = INFO - rootLogger.appenderRef.console.ref = ConsoleAppender - rootLogger.appenderRef.rolling.ref = RollingFileAppender - - # Uncomment this if you want to _only_ change Flink's logging - #logger.flink.name = org.apache.flink - #logger.flink.level = INFO - - # The following lines keep the log level of common libraries/connectors on - # log level INFO. The root logger does not override this. You have to manually - # change the log levels here. - logger.akka.name = akka - logger.akka.level = INFO - logger.kafka.name= org.apache.kafka - logger.kafka.level = INFO - logger.hadoop.name = org.apache.hadoop - logger.hadoop.level = INFO - logger.zookeeper.name = org.apache.zookeeper - logger.zookeeper.level = INFO - - # Log all infos to the console - appender.console.name = ConsoleAppender - appender.console.type = CONSOLE - appender.console.layout.type = PatternLayout - appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n - - # Log all infos in the given rolling file - appender.rolling.name = RollingFileAppender - appender.rolling.type = RollingFile - appender.rolling.append = false - appender.rolling.fileName = ${sys:log.file} - appender.rolling.filePattern = ${sys:log.file}.%i - appender.rolling.layout.type = PatternLayout - appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n - appender.rolling.policies.type = Policies - appender.rolling.policies.size.type = SizeBasedTriggeringPolicy - appender.rolling.policies.size.size=100MB - appender.rolling.strategy.type = DefaultRolloverStrategy - appender.rolling.strategy.max = 10 - - # Suppress the irrelevant (wrong) warnings from the Netty channel handler - logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline - logger.netty.level = OFF -``` - -Service is used to provide services for the REST and UI ports of the JobManager.The contents of `jobManager-Service.yaml` are as follows. - -``` -apiVersion: v1 -kind: Service -metadata: - name: flink-jobmanager -spec: - type: ClusterIP - ports: - - name: rpc - port: 6123 - - name: blob-server - port: 6124 - - name: webui - port: 8081 - selector: - app: flink - component: jobmanager -``` - -The contents of `jobmanager-session-deployment.yaml` are as follows. - -``` -apiVersion: apps/v1 -kind: Deployment -metadata: - name: flink-jobmanager -spec: - replicas: 1 - selector: - matchLabels: - app: flink - component: jobmanager - template: - metadata: - labels: - app: flink - component: jobmanager - spec: - containers: - - name: jobmanager - image: flink:1.11.0-scala_2.11 - args: ["jobmanager"] - ports: - - containerPort: 6123 - name: rpc - - containerPort: 6124 - name: blob-server - - containerPort: 8081 - name: webui - livenessProbe: - tcpSocket: - port: 6123 - initialDelaySeconds: 30 - periodSeconds: 60 - volumeMounts: - - name: flink-config-volume - mountPath: /opt/flink/conf - securityContext: - runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary - volumes: - - name: flink-config-volume - configMap: - name: flink-config - items: - - key: flink-conf.yaml - path: flink-conf.yaml - - key: log4j-console.properties - path: log4j-console.properties -``` - -The contents of `taskmanager-session-deployment.yaml` are as follows. - -``` -apiVersion: apps/v1 -kind: Deployment -metadata: - name: flink-taskmanager -spec: - replicas: 2 - selector: - matchLabels: - app: flink - component: taskmanager - template: - metadata: - labels: - app: flink - component: taskmanager - spec: - containers: - - name: taskmanager - image: flink:1.11.0-scala_2.11 - args: ["taskmanager"] - ports: - - containerPort: 6122 - name: rpc - - containerPort: 6125 - name: query-state - livenessProbe: - tcpSocket: - port: 6122 - initialDelaySeconds: 30 - periodSeconds: 60 - volumeMounts: - - name: flink-config-volume - mountPath: /opt/flink/conf/ - securityContext: - runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary - volumes: - - name: flink-config-volume - configMap: - name: flink-config - items: - - key: flink-conf.yaml - path: flink-conf.yaml - - key: log4j-console.properties - path: log4j-console.properties -``` - -Create the above four YAML configuration files on the cluster node and deploy them using the following instructions. - -``` -kubectl create -f flink-configuration-configmap.yaml -kubectl create -f jobmanager-service.yaml -kubectl create -f jobmanager-session-deployment.yaml -kubectl create -f taskmanager-session-deployment.yaml -``` - -Query to see if the payload was successfully created： - -``` -kubectl get cm| grep flink -kubectl get svc | grep flink -kubectl get pod -owide | grep Flink -``` - -##### 2. Outward publishing service - -Once the Flink payload is created, you need to publish the service externally。 - -- If you use Huawei Cloud CCE for testing, go to the "Workloads - Stateless Loads" page of CCE. Select Flink-JobManager and click Access Mode. -- Click "Add Service", select node access, and enter container port bit 8081. -- Click Network Management in CCE, you can see the service we just added, and visit the link for external publication. -- Go to the Dashboard page of Flink and click Submit New Job to submit the task. Here you have the option to submit an officially-provided WordCount sample.The directory is `flink-1.12.2/examples/streaming/WordCount.jar` \ No newline at end of file diff --git a/content/en/docs/v1-12-0/kubeflow_on_volcano.md b/content/en/docs/v1-12-0/kubeflow_on_volcano.md deleted file mode 100644 index 8a7f4e74..00000000 --- a/content/en/docs/v1-12-0/kubeflow_on_volcano.md +++ /dev/null @@ -1,246 +0,0 @@ -+++ -title = "Kubeflow on Volcano" - -date = 2021-06-29 -lastmod = 2021-06-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Kubeflow" -[menu.v1-12-0] - parent = "ecosystem" - weight = 3 - -+++ - - - - - -### Kubeflow introduction - -Kubernetes has become the de facto standard for cloud native application choreography and management, and more and more applications are migrating to Kubernetes. The field of artificial intelligence and machine learning naturally contains a large number of computation-intensive tasks. Developers are very willing to build an AI platform based on Kubernetes and make full use of the resource management, application scheduling, operation and maintenance monitoring capabilities provided by Kubernetes. However, it is a very complicated and tedious process to build an end-to-end AI computing platform based on Kubernetes, which needs to deal with many links. In addition to the model training we are familiar with, it also includes data collection, preprocessing, resource management, feature extraction, data verification, model management, model release, monitoring and other links. For an AI algorithm engineer, if he wants to do model training, he has to build a set of AI computing platform. This process is time-consuming and laborious, and requires a lot of knowledge accumulation[1]. - -{{
}} - -Kubeflow was born in 2017. The Kubeflow project is built based on containers and Kubernetes, aiming to provide data scientists, machine learning engineers, system operation and maintenance personnel with an agile platform for machine learning business deployment, development, training, release and management. It takes advantage of cloud native technology to make it faster and easier for users to deploy, use and manage the most popular machine learning software. - -What scenarios can we use Kubeflow for： - -- You want to train the TensorFlow model and you can use the model interface to publish application services in the K8S environment. -- You want to debug your code using Jupyter Notebooks, a multi-user Notebook Server. -- In the training Job, the CPU or GPU resources need to be scheduled and choreographed. -- You want TensorFlow to be combined with other components to publish services. - - - -### Kubeflow on Volcano - -Volcano is an enhanced high performance computing task batch processing system built on Kubernetes. As a platform for high performance computing scenarios, it makes up for Kubernetes' lack of basic capabilities in machine learning, deep learning, HPC, and big data computing scenarios, including gang-schedule scheduling capability, computational task queue management, task-topology, and GPU affinity scheduling. In addition, Volcano has enhanced the batch creation and life cycle management of computing tasks, fair-share, binpack scheduling and other aspects on the basis of the native Kubernetes capability. Volcano has fully solved the problem of distributed training in Kubeflow mentioned above. - -#### download kfctl - -First of all, you need to download kfctl, you can choose the appropriate compressed package file according to the system [1]. - -``` -$ tar -xvf kfctl_v1.0.2-0-ga476281_linux.tar.gz -$ sudo mv ./kfctl /usr/local/bin/kfctl -``` - - - -#### Configure environment variables - -``` -$ export PATH= $PATH:"" -$ export KF_NAME= -$ export BASE_DIR= -$ export KF_DIR=${BASE_DIR}/${KF_NAME} -$ export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml" -``` - - - -#### Install Kubeflow - -``` -$ mkdir -p ${KF_DIR} -$ cd ${KF_DIR} -$ Kfctl apply -V -f ${CONFIG_URI} -``` - -Confirm the installation results with the following instructions. - -``` -$ kubectl -n kubeflow get all -``` - - - -#### deploy Mnist - -Download the official test set provided by Kubuflow. - -``` -git clone https://github.com/kubeflow/examples.git -``` - - - -#### Start using Notebook - -External interface service is provided, where the nodes under the cluster need to be bound to public network IP. If Notebook is not installed, please use pip3 to install it first. - -``` -$ pip3 install jupyter notebook -$ jupyter notebook --allow-root -[W 09:08:03.572 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended. -[I 09:08:03.575 NotebookApp] Serving notebooks from local directory: /root/examples -[I 09:08:03.575 NotebookApp] Jupyter Notebook 6.3.0 is running at: -[I 09:08:03.575 NotebookApp] http://mytest-87034:30200/ -[I 09:08:03.575 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). -``` - -Access your-IP：30200/,Enter the configuration password to enter the Notebook. - - - -#### Run the official instance on the Notebook[2] - -1.Open Notebook and deploy TFJob。Open the notebook `mnist/mnist_vanilla_k8s.ipynb` ,Follow the guidelines to deploy a distributed TF Job. - -2.Add a schedulerName: add `schedulerName: volcano` in mnist.yaml,Be sure to use volcano for scheduling. - -``` -train_spec = f"""apiVersion: kubeflow.org/v1 -kind: TFJob -metadata: - name: {train_name} -spec: - schedulerName: volcano - tfReplicaSpecs: - Ps: - replicas: {num_ps} - template: - metadata: - annotations: - sidecar.istio.io/inject: "false" - spec: - serviceAccount: default-editor - containers: - - name: tensorflow - command: - - python - - /opt/model.py - - --tf-model-dir={model_dir} - - --tf-export-dir={export_path} - - --tf-train-steps={train_steps} - - --tf-batch-size={batch_size} - - --tf-learning-rate={learning_rate} - env: - - name: S3_ENDPOINT - value: {s3_endpoint} - - name: AWS_ENDPOINT_URL - value: {minio_endpoint} - - name: AWS_REGION - value: {minio_region} - - name: BUCKET_NAME - value: {mnist_bucket} - - name: S3_USE_HTTPS - value: "0" - - name: S3_VERIFY_SSL - value: "0" - - name: AWS_ACCESS_KEY_ID - value: {minio_username} - - name: AWS_SECRET_ACCESS_KEY - value: {minio_key} - image: {image} - workingDir: /opt - restartPolicy: OnFailure - Chief: - replicas: 1 - template: - metadata: - annotations: - sidecar.istio.io/inject: "false" - spec: - serviceAccount: default-editor - containers: - - name: tensorflow - command: - - python - - /opt/model.py - - --tf-model-dir={model_dir} - - --tf-export-dir={export_path} - - --tf-train-steps={train_steps} - - --tf-batch-size={batch_size} - - --tf-learning-rate={learning_rate} - env: - - name: S3_ENDPOINT - value: {s3_endpoint} - - name: AWS_ENDPOINT_URL - value: {minio_endpoint} - - name: AWS_REGION - value: {minio_region} - - name: BUCKET_NAME - value: {mnist_bucket} - - name: S3_USE_HTTPS - value: "0" - - name: S3_VERIFY_SSL - value: "0" - - name: AWS_ACCESS_KEY_ID - value: {minio_username} - - name: AWS_SECRET_ACCESS_KEY - value: {minio_key} - image: {image} - workingDir: /opt - restartPolicy: OnFailure - Worker: - replicas: 1 - template: - metadata: - annotations: - sidecar.istio.io/inject: "false" - spec: - serviceAccount: default-editor - containers: - - name: tensorflow - command: - - python - - /opt/model.py - - --tf-model-dir={model_dir} - - --tf-export-dir={export_path} - - --tf-train-steps={train_steps} - - --tf-batch-size={batch_size} - - --tf-learning-rate={learning_rate} - env: - - name: S3_ENDPOINT - value: {s3_endpoint} - - name: AWS_ENDPOINT_URL - value: {minio_endpoint} - - name: AWS_REGION - value: {minio_region} - - name: BUCKET_NAME - value: {mnist_bucket} - - name: S3_USE_HTTPS - value: "0" - - name: S3_VERIFY_SSL - value: "0" - - name: AWS_ACCESS_KEY_ID - value: {minio_username} - - name: AWS_SECRET_ACCESS_KEY - value: {minio_key} - image: {image} - workingDir: /opt - restartPolicy: OnFailure -""" -``` - -3.submit a job - -``` -kubectl apply -f mnist.yaml -``` \ No newline at end of file diff --git a/content/en/docs/v1-12-0/membership.md b/content/en/docs/v1-12-0/membership.md deleted file mode 100644 index 8e64d843..00000000 --- a/content/en/docs/v1-12-0/membership.md +++ /dev/null @@ -1,136 +0,0 @@ -+++ -title = "Community Membership" - - -date = 2019-01-28 -lastmod = 2020-09-01 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Community Membership" -[menu.v1-12-0] - parent = "contribution" - weight = 2 -+++ - -# Volcano Community Membership - -**Note:** We regularly update this document with the latest information and incorporate feedback from the Volcano community. - -This document gives a brief overview of the Volcano community roles and the requirements and responsibilities associated with them. - -| Role | Requirements | Responsibilities | Privileges | -| -----| ---------------- | ------------ | -------| -| [Member](#member) | Sponsored by 2 approvers, active in the community, has contributed to Volcano | Gets new contributors on board| Member of the Volcano GitHub organization | -| [Approver](#approver) | Sponsored by 2 maintainers, has extensive experience and knowledge of a particular domain, has actively contributed to code and review | Reviews and approves contributions from community members | Has write access to specific packages in relevant repositories | -| [Maintainer](#maintainer) | Sponsored by 2 owners, has shown good technical judgment in feature design/development and PR review | Release planning and feature development/maintenance | Has top-level write access to relevant repositories; name entry in the Maintainers file of the repositories | -| [Owner](#owner) | Sponsored by 3 owners, has proven ability to lead the entire Volcano project | Drives the overall technical roadmap of the project and sets priorities of activities in release planning | Has admin access to the Volcano GitHub organization | - - -**Note:** All Volcano community members must follow the Volcano [Code of Conduct](https://github.com/volcano-sh/website/blob/master/CODE_OF_CONDUCT.md). - -## Member - -Members are active participants in the community. They contribute by authoring PRs, -reviewing issues/PRs or participating in community discussions on slack/mailing list. - - -### Requirements - -- Sponsored by 2 approvers -- Has enabled two-factor authentication for GitHub account -- Actively contributed to the community. Contributions include but are not limited to: - - Authoring PRs - - Reviewing issues/PRs authored by other community members - - Participating in community discussions on slack/mailing list - - Attending Volcano community meetings - - -### Responsibilities and privileges - -- Becomes a member of the Volcano GitHub organization -- Can be assigned issues and PRs and can work on review requests from community members -- Assigns issues and PRs -- Gets new contributors on board -- Guides new contributors on maintaining docs/files -- Helps/Motivates new members in contributing to Volcano - - -## Approver - -Approvers are active members who have extensive experience and knowledge of a particular domain. -They have actively participated in the issue/PR reviews and have identified noteworthy issues during the reviews. - - -### Requirements - -- Sponsored by 2 maintainers -- Has been a member for at least 2 months -- Has reviewed a large number of PRs -- Has good codebase knowledge - - -### Responsibilities and privileges - -- Reviews code to maintain/improve code quality -- Acknowledges and works on review requests from community members -- Approves code contributions for acceptance related to a particular domain -- Has write access to specific packages inside a repository, enforced via bot -- Continues to contribute and guide other community members to contribute to Volcano - -## Maintainer - -Maintainers are approvers who have shown good technical judgment in feature design/development. -They can grasp the big picture of the project and features in the project. - -### Requirements - -- Sponsored by 2 owners -- Has been an approver for at least 2 months -- Is nominated by a project owner -- Has good technical judgment in feature design/development - -### Responsibilities and privileges - -- Participates in release planning -- Maintains project code quality -- Ensures the forward/backword compatibility of APIs based on feature graduation criteria -- Analyzes and proposes new features/enhancements in the Volcano project -- Demonstrates sound technical judgment -- Mentors contributors and approvers -- Has top-level write access to relevant repositories (including the permission to click the Merge PR button when manual check-in is necessary) -- Name entry in the Maintainers file of the repositories -- Participates & drives design/development of multiple features - -## Owner - -Owners are maintainers who have helped lead the project. -They have deep understanding of Volcano and the related domains, and have facilitated major decision making in issues such as release planning. - -### Requirements - -- Sponsored by 3 owners -- Has been a maintainer for at least 2 months -- Is nominated by a project owner -- Is not opposed by any project owner -- Has helped lead the entire project - -### Responsibilities and privileges - -- Makes overall technical decisions of the project -- Makes the overall technical roadmap of the project -- Sets priorities for release planning -- Guides and mentors all other community members -- Ensures all community members follow the Code of Conduct -- Makes sure all PRs are properly reviewed and merged, despite the admin access to all repositories -- May obtain admin access to relevant repositories as required -- Participates & drives design/development of multiple features - - -**Note:** These roles are applicable only to the Volcano Github organization and repositories. We are currently developing a formal process for review and acceptance into these roles. - - -[two-factor authentication]: https://help.github.com/articles/about-two-factor-authentication diff --git a/content/en/docs/v1-12-0/mindspore_on_volcano.md b/content/en/docs/v1-12-0/mindspore_on_volcano.md deleted file mode 100644 index f153fb82..00000000 --- a/content/en/docs/v1-12-0/mindspore_on_volcano.md +++ /dev/null @@ -1,77 +0,0 @@ -+++ -title = "MindSpore on Volcano" - -date = 2021-06-29 -lastmod = 2021-06-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "MindSpore" -[menu.v1-12-0] - parent = "ecosystem" - weight = 4 - -+++ - - - -### MindSpore introduction - -MindSpore is a new generation of deep learning framework launched by Huawei. It is derived from the best practices of the whole industry, which best matches the computing power of the rise processor, supports the flexible deployment of the whole scene of terminal, edge and cloud, and creates a brand new AI programming paradigm and lowers the threshold of AI development. - -### MindSpore on Volcano - -Create a new `mindSpore-cpu.yaml` in the cluster as follows. - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: mindspore-cpu -spec: - minAvailable: 1 - schedulerName: volcano - policies: - - event: PodEvicted - action: RestartJob - plugins: - ssh: [] - env: [] - svc: [] - maxRetry: 5 - queue: default - tasks: - - replicas: 8 - name: "pod" - template: - spec: - containers: - - command: ["/bin/bash", "-c", "python /tmp/lenet.py"] - image: lyd911/mindspore-cpu-example:0.2.0 - imagePullPolicy: IfNotPresent - name: mindspore-cpu-job - resources: - limits: - cpu: "1" - requests: - cpu: "1" - restartPolicy: OnFailure -``` - -Deploy. - -``` -kubectl apply -f mindspore-cpu.yaml -``` - -Query the status of the jobs in the cluster. - -``` -kubectl get pods -``` - - - diff --git a/content/en/docs/v1-12-0/mpi_on_volcano.md b/content/en/docs/v1-12-0/mpi_on_volcano.md deleted file mode 100644 index 67580718..00000000 --- a/content/en/docs/v1-12-0/mpi_on_volcano.md +++ /dev/null @@ -1,112 +0,0 @@ -+++ -title = "MPI on Volcano" - -date = 2021-04-07 -lastmod = 2021-04-07 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "MPI" -[menu.v1-12-0] - parent = "ecosystem" - weight = 5 - -+++ - - - -### HPC introduction - -High Performance Computing (HPC) refers to the use of aggregated Computing power to handle data-intensive Computing tasks that cannot be performed by standard workstations. - -HPC = PBS + Maui + OpenMPI[1] - -- PBS：Resource manager, which is responsible for managing resources for all nodes in the cluster -- Maui：Third-party task scheduler, support resource reservation, support various complex priority policies, support preemption, etc. -- OpenMPI：The upper communication environment, taking into account the functions of communication library, compilation and distributed start task. - -### openMPI introduction - -The OpenMPI project is an open source messaging interface implementation developed and maintained by a coalition of academic, research, and industry partners. Through it we can carry on the parallelization program design. - -### How is opensMPI implemented - -Here is a simple 4-thread MPI program example. - -{{
}} - - - -### MPI on Volcano - -Create `mpi-example.yaml`. - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: lm-mpi-job -spec: - minAvailable: 3 - schedulerName: volcano - plugins: - ssh: [] - svc: [] - tasks: - - replicas: 1 - name: mpimaster - policies: - - event: TaskCompleted - action: CompleteJob - template: - spec: - containers: - - command: - - /bin/sh - - -c - - | - MPI_HOST=`cat /etc/volcano/mpiworker.host | tr "\n" ","`; - mkdir -p /var/run/sshd; /usr/sbin/sshd; - mpiexec --allow-run-as-root --host ${MPI_HOST} -np 2 mpi_hello_world > /home/re; - image: volcanosh/example-mpi:0.0.1 - name: mpimaster - ports: - - containerPort: 22 - name: mpijob-port - workingDir: /home - restartPolicy: OnFailure - - replicas: 2 - name: mpiworker - template: - spec: - containers: - - command: - - /bin/sh - - -c - - | - mkdir -p /var/run/sshd; /usr/sbin/sshd -D; - image: volcanosh/example-mpi:0.0.1 - name: mpiworker - ports: - - containerPort: 22 - name: mpijob-port - workingDir: /home - restartPolicy: OnFailure ---- - -``` - -Delopy `mpi-example.yaml`. - -``` -kubectl apply -f mpi-example.yaml -``` - -View job performance under the cluster. - -``` -kubectl get pod -``` \ No newline at end of file diff --git a/content/en/docs/v1-12-0/plugins.md b/content/en/docs/v1-12-0/plugins.md deleted file mode 100644 index 1b78643a..00000000 --- a/content/en/docs/v1-12-0/plugins.md +++ /dev/null @@ -1,176 +0,0 @@ -+++ -title = "Plugins" - -date = 2021-05-13 -lastmod = 2021-05-13 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Plugins" -[menu.v1-12-0] - parent = "scheduler" - weight = 3 -+++ - - - -### Gang - -{{
}} - -#### Overview - -The Gang scheduling strategy is one of the core scheduling algorithms of the Volcano-Scheduler. It meets the scheduling requirements of "All or nothing" in the scheduling process and avoids the waste of cluster resources caused by arbitrary scheduling of Pod. The Gang scheduler algorithm is to observe whether the scheduled number of Pods under Job meets the minimum number of runs. When the minimum number of runs of Job is satisfied, the scheduling action is executed for all Pods under Job; otherwise, it is not executed. - -#### Scenario - -The Gang scheduling algorithm based on the container group concept is well suited for scenarios that require multi-process collaboration. AI scenes often contain complex processes. Data Ingestion, Data Analysts, Data Splitting, trainers, Serving, Logging, etc., which require a group of containers to work together, are suitable for container-based Gang scheduling strategies. Multi-thread parallel computing communication scenarios under MPI computing framework are also suitable for Gang scheduling because master and slave processes need to work together. Containers under the container group are highly correlated, and there may be resource contention. The overall scheduling allocation can effectively solve the deadlock. - -In the case of insufficient cluster resources, the scheduling strategy of Gang can significantly improve the utilization of cluster resources. - -### Binpack - -#### Overview - -The goal of the BinPack scheduling algorithm is to fill as many existing nodes as possible (try not to allocate blank nodes). In the concrete implementation, BinPack scheduling algorithm scores the nodes that can be delivered, and the higher the score, the higher the resource utilization rate of nodes. Binpack algorithm can fill up the nodes as much as possible to close the application load to some nodes, which is very conducive to the automatic expansion capacity function of K8s cluster nodes. - -The BinPack algorithm is injected into the Volcano-Scheduler process as a plug-in and will be applied during the Pod stage of node selection. When calculating the Binpack algorithm, the Volcano-Scheduler considers the various resources requested by Pod and averages them according to the weights configured for each resource. The weight of each resource in the node score calculation is different, depending on the weight value configured by the administrator for each resource. Different plug-ins also need to assign different weights when calculating node scores, and the Scheduler also sets the score weights for BinPack plugins. - -#### Scenario - -The BinPack algorithm is good for small jobs that can fill as many nodes as possible. For example, the single query job in the big data scene, the order generation in the e-commerce seckill scene, the single identification job in the AI scene, and the high concurrency service scene on the Internet, etc. This scheduling algorithm can reduce the fragmentation in the node as much as possible, and reserve enough resource space on the idle machine for Pod which has applied for more resource requests, so as to maximize the utilization of idle resources under the cluster. - - - -### Priority - -{{
}} - -#### Overview - -The Priority Plugin provides the implementation of job, Task sorting, and PreempTablefn, a function that calculates sacrifice jobs. Job sorting according to priorityClassName, the task of sorting by priorityClassName, createTime, id in turn. - -#### Scenario - -When the cluster runs multiple jobs but is low on resources, and each Job has a different number of Pods waiting to be scheduled, if you use the Kubernetes default scheduler, the Job with more Pods will eventually get more of the cluster's resources. In this case, the Volcano-Scheduler provides algorithms that enable different jobs to share cluster resources in a fair-share. - -The Priority Plugin enables users to customize their job and task priorities, and to customize scheduling policies at different levels according to their own needs. Priority is arranged according to Job's PriorityClassName at the application level. For example, there are financial scenarios, Internet of Things monitoring scenarios and other applications requiring high real-time performance in the cluster, and the Priority Plugin can ensure that they are scheduled in Priority. - - - -### DRF - -{{
}} - -#### Overview - -The full name of DRF scheduling algorithm is Dominant Resource Fairness, which is a scheduling algorithm based on the container group Dominant Resource. Dominant Resource is the largest percentage of all required resources for a container group. The DRF algorithm selects the Dominant Resource that is the smallest in a series of container groups for priority scheduling. This can meet more job, not because a fat business, starve a large number of small business. DRF scheduling algorithm can ensure that many types of resources coexist in the environment, as far as possible to meet the fair principle of allocation. - -#### Scenario - -The DRF scheduling algorithm gives priority to the throughput of the business in the cluster and is suitable for batch small business scenarios such as a single AI training, a single big data calculation and a query. - - - -### Proportion - -#### Overview - -Proportion scheduling algorithm uses the concept of queue to control the Proportion of total resources allocated in the cluster. Each queue allocates a certain proportion of cluster resources. For example, there are three teams that share A pool of resources on A cluster: Team A uses up to 40% of the total cluster, Team B uses up to 30%, and Team C uses up to 30%. If the amount of work delivered exceeds the team's maximum available resources, there is a queue. - -#### Scenario - -Proportion scheduling algorithm improves the flexibility and elasticity of cluster scheduling. The most typical scenario is that when multiple development teams in a company share a cluster, this scheduling algorithm can handle the requirements of shared resource matching and isolation between different departments very well. In multi-service mixed scenarios, such as computation-intensive AI business, network IO-intensive MPI and HPC business, and storage-intensive big data business, Proportion scheduling algorithm can allocate shared resources according to demand through matching. - - - -### Task-topology - -#### Overview - -The task-topology algorithm is an algorithm that computes the priority of tasks and nodes based on the affinity and anti-affinity configuration between tasks within a Job. By configuring the affinity and anti-affinity policies between tasks within the Job and using the Task-Topology algorithm, tasks with affinity configurations can be scheduled to the same node first, and PODs with anti-affinity configurations to different nodes. - -#### Scenario - -node affinity： - -- Task-topology is important for improving computational efficiency in deep learning computing scenarios. Using the TensorFlow calculation as an example, configure the affinity between "ps" and "worker". Task-topology algorithm enables "ps" and "worker" to be scheduled to the same node as far as possible, so as to improve the efficiency of network and data interaction between "ps" and "worker", thus improving the computing efficiency. -- Tasks in HPC and MPI scenarios are highly synchronized and need high-speed network IO. - -Anti-affinity： - -- Take the TensorFlow calculation as an example, the anti-affinity between "ps" and "ps" - -- Master and slave backup of e-commerce service scene, data disaster tolerant, to ensure that there are spare jobs to continue to provide service after a job fails. - - - -### Predicates - -#### Overview - -The Predicate Plugin calls the PredicateGPU with pod and nodeInfo as parameters to evaluate and pre-select jobs based on the results. - -#### Scenario - -In AI scenarios where GPU resources are required, the Predicate Plugin can quickly filter out those that require the GPU for centralized scheduling. - - -### Nodeorder - -#### Overview - -The NodeOrder Plugin is a scheduling optimization strategy that scores nodes from various dimensions through simulated assignments to find the node that is best suited for the current job. The scoring parameters are configured by the user. The parameter contains the Affinity、reqResource、LeastReqResource、MostResource、balanceReqResouce. - -#### Scenario - -NodeOrder Plugin provides scoring criteria of multiple dimensions for scheduling, and the combination of different dimensions enables users to flexibly configure appropriate scheduling policies according to their own needs. - - - -### SLA - -#### Overview - -When users apply jobs to Volcano, they may need adding some particular constraints to job, for example, longest Pending time aiming to prevent job from starving. And these constraints can be regarded as Service Level Agreement (SLA) which are agreed between volcano and user. So sla plugin is provided to receive and realize SLA settings for both individual job and whole cluster. - -#### Scenario - -Users can customize SLA related parameters in their own cluster according to business needs. For example, for clusters with high real-time service requirements, JobWaitingTime can be set as small as possible. For clusters with bulk computing jobs, JobWaitingTime can be set to larger. The parameters of a specific SLA and the optimization of the parameters need to be combined with the specific business and related performance measurement results. - -### TDM - -#### Overview - -The full name of TDM is Time Division Multiplexing. In a co-located environment, some nodes are in both Kubernetes cluster and Yarn cluster. For these nodes, Kubernetes and Yarn cluster can use these resource by time-sharing multiplexing.The TDM Plugin marks these nodes as `revocable nodes`. TDM plugin will try to dispatch `preemptable task` to `revocable node` in node revocable time and evict the `preemptable task` from `revocable node` out of revocable time.. TDM Plugin improves the time-division multiplexing ability of node resources in the scheduling process of Volcano. - -#### Scenario - -In ToB business, cloud vendors provide cloud-based resources for merchants, and different merchants adopt different container arrangement frameworks (Kubernetes/YARN, etc.). TDM Plugin improves the time-sharing efficiency of common node resources and further improves the utilization rate of resources. - - - -### Numa-aware - -#### Overview - -When the node runs many CPU-bound pods, the workload can move to different CPU cores depending on whether the pod is throttled and which CPU cores are available at scheduling time. Many workloads are not sensitive to this migration and thus work fine without any intervention. However, in workloads where CPU cache affinity and scheduling latency significantly affect workload performance, the kubelet allows alternative CPU management policies to determine some placement preferences on the node. - -The CPU Manager and the Topology Manager are all Kubelet components, However There is the following limitation: - -- The scheduler is not topology-aware. so it is possible to be scheduled on a node and then fail on the node due to the Topology Manager. this is unacceptable for TensorFlow job. If any worker or ps failed on node, the job will fail. -- The managers are node-level that results in an inability to match the best node for NUMA topology in the whole cluster. - -The Numa-Aware Plugin aims to address these limitations. - -- Support cpu resource topology scheduling. -- Support pod-level topology policies. - -#### Scenario - -Common scenarios for NUMA-Aware are computation-intensive jobs that are sensitive to CPU parameters, scheduling delays. Such as scientific calculation, video decoding, animation rendering, big data offline processing and other specific scenes. - - diff --git a/content/en/docs/v1-12-0/pp_on_volcano.md b/content/en/docs/v1-12-0/pp_on_volcano.md deleted file mode 100644 index 7ce97536..00000000 --- a/content/en/docs/v1-12-0/pp_on_volcano.md +++ /dev/null @@ -1,242 +0,0 @@ -+++ -title = "PaddlePaddle on Volcano" - -date = 2021-06-29 -lastmod = 2021-06-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "PaddlePaddle" -[menu.v1-12-0] - parent = "ecosystem" - weight = 6 - -+++ - - - -### paddlepaddle introduction - -PaddlePaddle is a deep learning framework open source by Baidu in September 2016, aiming to provide a safe, efficient, flexible, easy to use and extensible deep learning platform. - -In October 2018, the Paddle Fluid 1.0 version was released by the Paddle Fluid team, which comprehensively upgraded core capabilities such as neural network description, large-scale distributed training, and high-performance inference engine. For example, in the latest Paddle Fluid 1.5.2, the Paddle fly supports data parallelism, model parallelism, pipeline parallelism, and other parallel modes. Parameter server architecture and point-to-point synchronous training architecture fully support large-scale training on hardware resource devices such as CPU and GPU [1]. - -### paddlepaddle on valcano - -Upload `ctr-Volcano.yaml` on the cluster node as follows - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: ctr-volcano -spec: - minAvailable: 4 - schedulerName: volcano - policies: - - event: PodEvicted - action: RestartJob - - event: PodFailed - action: RestartJob - tasks: - - replicas: 2 - name: pserver - template: - metadata: - labels: - paddle-job-pserver: fluid-ctr - spec: - imagePullSecrets: - - name: default-secret - volumes: - - hostPath: - path: /home/work/ - type: "" - name: seqdata - containers: - - image: volcanosh/edlctr:v1 - command: - - paddle_k8s - - start_fluid - imagePullPolicy: IfNotPresent - name: pserver - volumeMounts: - - mountPath: /mnt/seqdata - name: seqdata - resources: - limits: - cpu: 10 - memory: 30Gi - ephemeral-storage: 10Gi - requests: - cpu: 1 - memory: 100M - ephemeral-storage: 1Gi - env: - - name: GLOG_v - value: "0" - - name: GLOG_logtostderr - value: "1" - - name: TOPOLOGY - value: "" - - name: TRAINER_PACKAGE - value: /workspace - - name: NAMESPACE - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.namespace - - name: POD_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: POD_NAME - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.name - - name: PADDLE_CURRENT_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: PADDLE_JOB_NAME - value: fluid-ctr - - name: PADDLE_IS_LOCAL - value: "0" - - name: PADDLE_TRAINERS_NUM - value: "2" - - name: PADDLE_PSERVERS_NUM - value: "2" - - name: FLAGS_rpc_deadline - value: "36000000" - - name: ENTRY - value: cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1 - - name: PADDLE_PORT - value: "30236" - - name: LD_LIBRARY_PATH - value: /usr/local/lib:/usr/local/nvidia/lib64:/usr/local/rdma/lib64:/usr/lib64/mlnx_ofed/valgrind - - name: PADDLE_TRAINING_ROLE - value: PSERVER - - name: TRAINING_ROLE - value: PSERVER - restartPolicy: OnFailure - - replicas: 2 - policies: - - event: TaskCompleted - action: CompleteJob - name: trainer - template: - metadata: - labels: - paddle-job: fluid-ctr - spec: - imagePullSecrets: - - name: default-secret - volumes: - - hostPath: - path: /home/work/ - type: "" - name: seqdata - containers: - - image: volcanosh/edlctr:v1 - command: - - paddle_k8s - - start_fluid - imagePullPolicy: IfNotPresent - name: trainer - volumeMounts: - - mountPath: /mnt/seqdata - name: seqdata - resources: - limits: - cpu: 10 - memory: 30Gi - ephemeral-storage: 10Gi - requests: - cpu: 1 - memory: 100M - ephemeral-storage: 10Gi - env: - - name: GLOG_v - value: "0" - - name: GLOG_logtostderr - value: "1" - - name: TOPOLOGY - - name: TRAINER_PACKAGE - value: /workspace - - name: CPU_NUM - value: "2" - - name: NAMESPACE - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.namespace - - name: POD_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: POD_NAME - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.name - - name: PADDLE_CURRENT_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: PADDLE_JOB_NAME - value: fluid-ctr - - name: PADDLE_IS_LOCAL - value: "0" - - name: FLAGS_rpc_deadline - value: "36000000" - - name: PADDLE_PORT - value: "30236" - - name: PADDLE_PSERVERS_NUM - value: "2" - - name: PADDLE_TRAINERS_NUM - value: "2" - - name: PADDLE_TRAINING_ROLE - value: TRAINER - - name: TRAINING_ROLE - value: TRAINER - - name: LD_LIBRARY_PATH - value: /usr/local/lib:/usr/local/nvidia/lib64:/usr/local/rdma/lib64:/usr/lib64/mlnx_ofed/valgrind - - name: ENTRY - value: cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1 - restartPolicy: OnFailure - -``` - -Deploy under the cluster terminal. - -``` -kubectl apply -f ctr-volcano.yaml -``` - -Query if the job is running properly.If the PodGroup cannot meet the scheduling conditions, check that the cluster has sufficient resources available. - -``` -kubectl get podgroup -kubectl describe podgroup ctr-volcano -kubectl get pods | grep ctr-volcano -``` - -You can select a PServer task to view the log. - -``` -kubectl logs ctr-volcano-pserver-0 -``` - -Select a Tariner task to view the log. - -``` -kubectl logs ctr-volcano-trainer-0 -``` \ No newline at end of file diff --git a/content/en/docs/v1-12-0/referrals.md b/content/en/docs/v1-12-0/referrals.md deleted file mode 100644 index 33550bde..00000000 --- a/content/en/docs/v1-12-0/referrals.md +++ /dev/null @@ -1,88 +0,0 @@ -+++ -title = "Recommended Configurations for Common Scenarios" - -date = 2021-04-07 -lastmod = 2021-04-07 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Scenario & Configuration" -[menu.v1-12-0] - parent = "scheduler" - weight = 4 - -+++ - - - -### Background - -{{
}} - -Deep learning generally involves four stages: data acquisition and processing, model training and evolution, model deployment, and model evaluation [1]. At the beginning of an enterprise's involvement in deep learning, due to the lack of relevant experience and resources for deep learning, the team members usually choose to complete the above steps manually step by step in order to quickly meet the immediate needs of the project development period. However, no matter how domain-specific a deep learning expert may have, scaling up a project without the support of an infrastructure platform becomes extremely difficult. Large-scale operations in a production environment require more rapid replication and iterative upgrades than local operations. Production environment has higher requirements for reliability of data, reproducibility of model training, scalability of training model, reliability of automated deployment and operation and maintenance. Therefore, when the enterprise grows to a certain scale, the enterprise will want to build a stable scientific computing platform, so that the upper business personnel can focus on algorithm optimization and business development. - -In general, the construction of a deep learning platform will encounter many challenges, mainly reflected in the following aspects: - -- Data management automation -- The efficient use of resources -- Mask the complexity of the underlying technology - - - -##### Data management - -In a business scenario of deep learning, practitioners spend a significant amount of time capturing and transforming the data needed to build the model. As business scenarios expand, the automation of data transformation becomes a bottleneck in the efficiency of model building. In the process of data processing, new data will be generated, and these data will not only be used for this training, but also for the subsequent reasoning process. And the newly generated data does not need to be passed to the data source, but instead wants to be placed in a new storage space. This requires the underlying platform to provide a scalable storage system. Flexible, extensible and secure data storage system will greatly promote the improvement of data management ability. - -##### The efficient use of resources - -Deep learning-related applications are resource-intensive, and the use of resources fluctuates greatly, with peaks and valleys. Fast access to computing resources when the application starts running; At the end of the application, it is very important to recycle the unsuitable computing resources for the improvement of resource utilization. The types and time of computing resources used in data processing, model training and reasoning are different, which requires a flexible resource supply mechanism provided by the computing platform. Deep learning tasks require a large amount of computing resources, so it is impossible to build computing resources for each user separately. The computing platform should be able to guarantee the multi-lease of resource use, allowing multiple users to use the computing resource pool at the same time, rather than being monopolized by a few users. - -##### Shield the underlying technical complexity - -AI practitioners focus on model building and product development, ignoring the importance of infrastructure to AI development. Machine Learning/Deep Learning's own growing technology stack of users includes computing frameworks for machine learning such as TensorFlow, the Spark data processing engine, and underlying drivers such as CUDA. Manually managing these dependencies can consume significant resources and effort on the part of practitioners. - - - -Based on the above goals and challenges, deep learning and other AI computing choose to use containers and Kubernetes to build and manage deep learning platforms. Need to achieve user isolation, queuing mechanism, user rights management, resource management, container ceiling management and many other functions. - - - -### Common scenarios - -1. Priority issues:There are many roles of algorithm engineers in the company, such as interns, full-time employees and senior engineers. They all do the same thing and need GPU resources to do experiments or training. However, when resources are tight, they need to queue according to priority. Even in the same position, there is preemption for urgent assignments (such as paper deadlines, project launches). The priority level of the task needs to be reported by the engineer. -2. storage of data:The data sets of deep learning scenarios are very large, so it is necessary to optimize the design at the storage level to meet this requirement. When computing, if one person trains a model, then simple storage is OK. However, if there are 100 people and each person has different training models and data sets, and needs to run multiple tasks, what kind of storage can help you solve this problem? This is a realistic problem facing us. How storage ADAPTS to the ever-expanding computing needs of users is a matter of concern to architects -3. Parallel computing problem:A general requirement of deep learning platform based on K8S is to support multi-machine scheduling and make model training as fast as possible through multi-machine parallel training. -4. User isolation problem :For larger companies, there are many business departments, which require multi-user management and multi-tenant isolation. (User isolation problem) -5. Batch scheduling requirements for pod under deep learning scenarios. -6. Resource preemption problem:The upper limit on the amount of scarce resources (GPUs) used and the upper limit on how long they can be used. -7. Reservation:The details of the underlying application resources are shielded. It only cares about the total amount of available GPU, not the distribution of GPU. - - - -## referrals of the configuration - -- Priority issues: Customize the priority hierarchy. Focus on the `PriorityClassName` of the job itself. This field represents the priority of the job and is in effect for preemptive scheduling and prioritization. - -- Data storage: Recommends the `block storage` service in extremely performing businesses such as databases. Recommend the use of `object storage` service in the business scenarios of HPC, big data and AI. - -- High performance computing, model training parallelism :WRF, Kubegene, MPI high performance computing scenarios, multi-machine parallel training. - - 1. Job Plugin: `ssh` provides secret login, and `svc` provides network information needed for job operation, such as host file, headless service, etc, to provide automatic configuration for calculating cluster parameters. - - 2. `Policies field`: Once any `pod` has been killed, restart the entire job `restartJob`/`compeletedJob`. Because of this HPC scenario, Gang-based training job, if a worker fails, it is usually pointless for the entire job to run. - - 3. This scenario will generally define multiple `tasks`, 1 master + multiple workers. - -- Multi-tenancy, isolation issues :TensorFlow has no concept of tenants. The new resource splitting tool `Queue` introduced by Volcano is configured around Queue in various ways. To ensure the multi-lease of the cluster resources, the surplus of the cluster resources, and the flexibility of the tenants to use the cluster resources. When the cluster resources are redivided, the tenant resource quota can be guaranteed through Reclaim. Configure `preempt, reclaim` in `Volcano-Scheduler-ConfigMap`; configure `proportion` in plugins. - -- Batch scheduling: The `Gang Plugin` meets the requirements of pod batch scheduling for deep learning scenarios. The class field `minAvailable`, the main indicator of the Gang scheduling algorithm. The plugins field needs to use `env` and `svc`. The definition of Policies is similar to the high performance computing scenario where a `pod` ejection restarts the job. - - - -Reference: - -[1][Challenges facing AI infrastructure ](https://zhuanlan.zhihu.com/p/75634193) - diff --git a/content/en/docs/v1-12-0/schduler_introduction.md b/content/en/docs/v1-12-0/schduler_introduction.md deleted file mode 100644 index 90eba7e3..00000000 --- a/content/en/docs/v1-12-0/schduler_introduction.md +++ /dev/null @@ -1,103 +0,0 @@ -+++ -title = "Introduction" - - -date = 2019-01-28 -lastmod = 2020-09-01 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Overview" -[menu.v1-12-0] - parent = "scheduler" - weight = 1 - -+++ - -## Introduction -Volcano scheduler is the component responsible for pod scheduling. It consists of a series of actions and plugins. Actions define the action that should be executed in every step. Plugins provide the action algorithm details in different scenarios. -Volcano scheduler is highly scalable. You can specify and implement actions and plugins based on your requirements. -## Workflow -{{
}} - - -Volcano scheduler works as follows: - -1. Watches for and caches the jobs submitted by the client. -2. Opens a session periodically. A scheduling cycle begins. -3. Sends jobs that are not scheduled in the cache to the to-be-scheduled queue in the session. -4. Traverses all jobs to be scheduled. Executes enqueue, allocate, preempt, reclaim, and backfill actions in the order they are defined, and finds the most suitable node for each job. Binds the job to the node. The specific algorithm logic executed in the action depends on the implementation of each function in the registered plugins. -5. Closes this session. - -## Actions -### enqueue -The enqueue action is responsible for filtering out the tasks that meet the scheduling requirements based on a series of filtering algorithms and sending them to the to-be-scheduled queue. After the action is executed, the status of the task changes from `pending` to `inqueue`. -### allocate -The allocate action is responsible for selecting the most suitable node based on a series of prediction and optimization algorithms. -### preempt -The preempt action is responsible for preemptive scheduling of high priority tasks in the same queue according to priority rules. -### reclaim -The reclaim action is responsible for reclaiming the resources allocated to the cluster based on the queue weight when a new task enters the queue and the cluster resources cannot meet the needs of the queue. -### backfill -The backfill action is responsible for backfilling the tasks in the `pending` state into the cluster node to maximize the resource utilization of the node. - -## Plugins -### gang -The gang plugin considers that tasks not in the `Ready` state (including Binding, Bound, Running, Allocated, Succeed, and Pipelined) have a higher priority. It checks whether the resources allocated to the queue can meet the resources required by the task to run `minavailable` pods after trying to evict some pods and reclaim resources. If yes, the gang plugin will evict some pods. -### conformance -The conformance plugin considers that the tasks in namespace `kube-system` have a higher priority. These tasks will not be preempted. -### DRF -The DRF plugin considers that tasks with fewer resources have a higher priority. It attempts to calculate the total amount of resources allocated to the preemptor and preempted tasks, and triggers the preemption when the preemptor task has less resources. -### nodeorder -The nodeorder plugin scores all nodes for a task by using a series of scoring algorithms. The node with the highest score is considered to be the most suitable node for the task. -### predicates -The predicates plugin determines whether a task is bound to a node by using a series of evaluation algorithms. -### priority -The priority plugin compares the priorities of two jobs or tasks. For two jobs, it decides whose priority is higher by comparing `job.spec.priorityClassName`. For two tasks, it decides whose priority is higher by comparing `task.priorityClassName`, `task.createTime`, and `task.id` in order. -## Configuration -Volcano scheduler is highly scalable because of its composite pattern design. Users can decide which actions and plugins to use according to their needs, and they can also implement customization by calling the action or plugin interfaces. The scheduler configuration is located in the ConfigMap named **volcano-scheduler-configmap**, which is mounted as a volume into the "/volcano.scheduler" directory in the scheduler container. -### How to Get Configuration of Volcano Scheduler -* Get the ConfigMap named `volcano-scheduler-configmap`. - -```shell -# kubectl get configmap -nvolcano-system -NAME DATA AGE -volcano-scheduler-configmap 1 6d2h -``` - -* View the details of the data part in the ConfigMap. - -```shell -# kubectl get configmap volcano-scheduler-configmap -nvolcano-system -oyaml -apiVersion: v1 -data: - volcano-scheduler.conf: | - actions: "enqueue, allocate, backfill" - tiers: - - plugins: - - name: priority - - name: gang - - name: conformance - - plugins: - - name: drf - - name: predicates - - name: proportion - - name: nodeorder - - name: binpack -kind: ConfigMap -metadata: - annotations: - kubectl.kubernetes.io/last-applied-configuration: | - {"apiVersion":"v1","data":{"volcano-scheduler.conf":"actions: \"enqueue, allocate, backfill\"\ntiers:\n- plugins:\n - name: priority\n - name: gang\n - name: conformance\n- plugins:\n - name: drf\n - name: predicates\n - name: proportion\n - name: nodeorder\n - name: binpack\n"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"volcano-scheduler-configmap","namespace":"volcano-system"}} - creationTimestamp: "2020-08-15T04:01:02Z" - name: volcano-scheduler-configmap - namespace: volcano-system - resourceVersion: "266" - selfLink: /api/v1/namespaces/volcano-system/configmaps/volcano-scheduler-configmap - uid: 1effe4d6-126c-42d6-a3a4-b811075c30f5 -``` - -`actions` and `tiers` are included in `volcano-scheduler.conf`. In `actions`, the comma is used as a separator to configure the actions to be executed by the scheduler. Note that the scheduler will execute the actions in the order that they are configured. Volcano itself will not check the rationality of the order. The list of plugins configured in `tiers` is the plugins registered with the scheduler. The specific algorithms defined in plugins will be called in `actions`. diff --git a/content/en/docs/v1-12-0/spark_on_volcano.md b/content/en/docs/v1-12-0/spark_on_volcano.md deleted file mode 100644 index c2f304b1..00000000 --- a/content/en/docs/v1-12-0/spark_on_volcano.md +++ /dev/null @@ -1,101 +0,0 @@ -+++ -title = "Spark on Volcano" - -date = 2021-06-29 -lastmod = 2021-06-29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Spark" -[menu.v1-12-0] - parent = "ecosystem" - weight = 8 - -+++ - - - -### Spark introduction - -Spark is a fast and versatile big data clustering computing system. It provides high-level APIs for Scala, Java, Python, and R, as well as an optimization engine that supports a generic computational graph for data analysis. It also supports a rich set of advanced tools, including Spark SQL for SQL and Dataframes, MLLib for machine learning, GraphX for graphics processing, and Spark Streaming for Streaming. - -### Spark on Volcano - -Currently, there are two ways to support the integration of Spark on Kubernetes and volcano. -- Spark on Kubernetes native support: maintained by the [Apache Spark community](https://github.com/apache/spark) and Volcano community -- Spark Operator support: maintained by the [GoogleCloudPlatform community](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) and Volcano community - -#### Spark on Kubernetes native support (spark-submit) - -Spark on Kubernetes with Volcano as a custom scheduler is supported since Spark v3.3.0 and Volcano v1.5.1. See more detail in [link](https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-volcano-as-customized-scheduler-for-spark-on-kubernetes). - -#### Spark Operator support (spark-operator) - -Install Spark-Operator through Helm. - -``` -$ helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator - -$ helm install my-release spark-operator/spark-operator --namespace spark-operator --create-namespace -``` - -To ensure that the Spark-Operator is up and running, check with the following directive. - -``` -$ kubectl get po -nspark-operator -``` - -Here's the official `spark.yaml`. - -``` -apiVersion: "sparkoperator.k8s.io/v1beta2" -kind: SparkApplication -metadata: - name: spark-pi - namespace: default -spec: - type: Scala - mode: cluster - image: "gcr.io/spark-operator/spark:v3.0.0" - imagePullPolicy: Always - mainClass: org.apache.spark.examples.SparkPi - mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar" - sparkVersion: "3.0.0" - batchScheduler: "volcano" #Note: the batch scheduler name must be specified with `volcano` - restartPolicy: - type: Never - volumes: - - name: "test-volume" - hostPath: - path: "/tmp" - type: Directory - driver: - cores: 1 - coreLimit: "1200m" - memory: "512m" - labels: - version: 3.0.0 - serviceAccount: spark - volumeMounts: - - name: "test-volume" - mountPath: "/tmp" - executor: - cores: 1 - instances: 1 - memory: "512m" - labels: - version: 3.0.0 - volumeMounts: - - name: "test-volume" - mountPath: "/tmp" -``` - -Deploy the Spark application and see the status. - -``` -$ kubectl apply -f spark-pi.yaml -$ kubectl get SparkApplication -``` diff --git a/content/en/docs/v1-12-0/tf_on_volcano.md b/content/en/docs/v1-12-0/tf_on_volcano.md deleted file mode 100644 index cb2a405d..00000000 --- a/content/en/docs/v1-12-0/tf_on_volcano.md +++ /dev/null @@ -1,111 +0,0 @@ -+++ -title = "TensorFlow on Volcano" - -date = 2021-04-07 -lastmod = 2021-04-07 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "TensorFlow" -[menu.v1-12-0] - parent = "ecosystem" - weight = 7 - -+++ - - - -### TensorFlow introduction - -TensorFlow is a symbolic mathematical system based on data flow programming, which is widely used in programming and realization of various machine learning algorithms. Its predecessor is DistBelief, a neural network algorithm library of Google. - -### TensorFlow on Volcano - -PS-worker model: Parameter Server performs model-related services, Work Server trains related services, inference calculation, gradient calculation, etc[1]. - -{{
}} - -TensorFlow on Kubernetes has many problems: - -- Resource isolation. -- Lack of GPU scheduling, Gang Schuler. -- Process Legacy. -- Training log is not convenient to save. - -Create `tftest.yaml`. - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: tensorflow-dist-mnist -spec: - minAvailable: 3 - schedulerName: volcano - plugins: - env: [] - svc: [] - policies: - - event: PodEvicted - action: RestartJob - queue: default - tasks: - - replicas: 1 - name: ps - template: - spec: - containers: - - command: - - sh - - -c - - | - PS_HOST=`cat /etc/volcano/ps.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - export TF_CONFIG={\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"ps\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"}; - python /var/tf_dist_mnist/dist_mnist.py - image: volcanosh/dist-mnist-tf-example:0.0.1 - name: tensorflow - ports: - - containerPort: 2222 - name: tfjob-port - resources: {} - restartPolicy: Never - - replicas: 2 - name: worker - policies: - - event: TaskCompleted - action: CompleteJob - template: - spec: - containers: - - command: - - sh - - -c - - | - PS_HOST=`cat /etc/volcano/ps.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; - export TF_CONFIG={\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"worker\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"}; - python /var/tf_dist_mnist/dist_mnist.py - image: volcanosh/dist-mnist-tf-example:0.0.1 - name: tensorflow - ports: - - containerPort: 2222 - name: tfjob-port - resources: {} - restartPolicy: Never -``` - -Deploy `tftest.yaml`. - -``` -kubectl apply -f tftest.yaml -``` - -View job health. - -``` -kubectl get pod -``` \ No newline at end of file diff --git a/content/en/docs/v1-7-0/cli.md b/content/en/docs/v1-7-0/cli.md deleted file mode 100644 index d3170139..00000000 --- a/content/en/docs/v1-7-0/cli.md +++ /dev/null @@ -1,74 +0,0 @@ -+++ -title = "CLI" - - -date = 2024-01-16 -lastmod = 2024-01-16 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Commandline" -[menu.v1-7-0] - parent = "cli" - weight = 1 -+++ - -## Introduction -A Command Line Interface (CLI) is provided for you to manage resources. -## Configuration - -1. You can obtain the latest executable file by cloning the code from GitHub and running the following command in the root directory of the project: -``` -# make vcctl -``` -2. Copy the executable file to $PATH. You then can execute it anywhere. - -## Command Line List -### Listing all jobs -vcctl job list - -```html -# vcctl job list -Name Creation Phase JobType Replicas Min Pending Running Succeeded Failed Unknown RetryCount -job-1 2020-09-01 Running Batch 1 1 0 1 0 0 0 0 -``` - -### Deleting a specific job -vcctl job delete --name job-name [--namespace job-namespace] - -```html -# vcctl delete job --name job-1 --namespaces default -delete job job-1 successfully -``` - -### Suspending a job -vcctl job suspend --name job-name [--namespace job-namespace] - -```html -# vcctl job suspend --name job-1 --namespace default -``` - -### Resuming a job (opposite to "vcctl job suspend") -vcctl job resume --name job-name [--namespace job-namespace] - -```html -# vcctl job resume --name job-1 --namespace default -``` - -### Running a job -vcctl job run --name job-name [--namespace job-namespace] - -```html -# vcctl job run --name job-1 --namespace default -``` - -## Note -For more information about Volcano command lines, run the following commands: - -```html -# vcctl -h -# vcctl [command] -h -``` diff --git a/content/en/docs/v1-7-0/podgroup.md b/content/en/docs/v1-7-0/podgroup.md deleted file mode 100644 index 382f8213..00000000 --- a/content/en/docs/v1-7-0/podgroup.md +++ /dev/null @@ -1,104 +0,0 @@ -+++ -title = "PodGroup" - - -date = 2024-01-16 -lastmod = 2024-01-16 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "PodGroup" -[menu.v1-7-0] - parent = "concepts" - weight = 2 -+++ - -## Introduction -PodGroup is a group of pods with strong association and is mainly used in batch scheduling, for example, ps and worker tasks in TensorFlow. PodGroup is of a Custom Resource Definition (CRD) type. -## Example -```shell -apiVersion: scheduling.volcano.sh/v1beta1 -kind: PodGroup -metadata: - creationTimestamp: "2020-08-11T12:28:55Z" - generation: 5 - name: test - namespace: default - ownerReferences: - - apiVersion: batch.volcano.sh/v1alpha1 - blockOwnerDeletion: true - controller: true - kind: Job - name: test - uid: 028ecfe8-0ff9-477d-836c-ac5676491a38 - resourceVersion: "109074" - selfLink: /apis/scheduling.volcano.sh/v1beta1/namespaces/default/podgroups/job-1 - uid: eb2508f5-3349-439c-b94d-4ac23afd71ff -spec: - minMember: 1 - minResources: - cpu: "3" - memory: "2048Mi" - priorityClassName: high-priority - queue: default -status: - conditions: - - lastTransitionTime: "2020-08-11T12:28:57Z" - message: '1/0 tasks in gang unschedulable: pod group is not ready, 1 minAvailable.' - reason: NotEnoughResources - status: "True" - transitionID: 77d5be3f-6169-4f86-8e65-0bdc621ce983 - type: Unschedulable - - lastTransitionTime: "2020-08-11T12:29:02Z" - reason: tasks in gang are ready to be scheduled - status: "True" - transitionID: 54514401-5c90-4b11-840d-90c1cda93096 - type: Scheduled - phase: Running - running: 1 - -``` -## Key Fields -### minMember -`minMember` indicates the minimum number of pods or tasks running under the PodGroup. If the cluster resource cannot meet the demand of running the minimum number of pods or tasks, no pod or task in the PodGroup will be scheduled. -### queue -`queue` indicates the queue to which the PodGroup belongs. The queue must be in the Open state. -### priorityClassName -`priorityClassName` represents the priority of the PodGroup and is used by the scheduler to sort all the PodGroups in the queue during scheduling. Note that **system-node-critical** and **system-cluster-critical** are reserved values, which mean the highest priority. If `priorityClassName` is not specified, the default priority is used. -### minResources -`minResources` indicates the minimum resources for running the PodGroup. If available resources in the cluster cannot satisfy the requirement, no pod or task in the PodGroup will be scheduled. -### phase -`phase` indicates the current status of the PodGroup. -### conditions -`conditions` represents the status log of the PodGroup, including the key events that occurred in the lifecycle of the PodGroup. -### running -`running` indicates the number of running pods or tasks in the PodGroup. -### succeed -`succeed` indicates the number of successful pods or tasks in the PodGroup. -### failed -`failed` indicates the number of failed pods or tasks in the PodGroup. -## Status -{{
}} - -### pending - -`pending` indicates that the PodGroup has been accepted by Volcano but its resource requirement has not been satisfied yet. Once satisfied, the status will turn to running. -### running -`running` indicates that there are at least **minMember** pods or tasks running under the PodGroup. -### unknown -`unknown` indicates that among **minMember** pods or tasks, some are running while others are not scheduled. The reason could be due to the lack of resources. The scheduler will wait until ControllerManager starts these pods or tasks again. -### inqueue -`inqueue` indicates that the PodGroup has passed validation and is waiting to be bound to a node. It is a transient state between pending and running. -## Usage -### minMember -In some scenarios such as machine learning training, you do not need all tasks of a job to be completed. Instead, when a specified number of tasks are completed, the job can be achieved. In this case, the `minMember` field is suitable. -### priorityClassName -`priorityClassName` is used in preemptive priority scheduling. -### minResources -In some scenarios such as big data analytics, a job can run only when available resources meet the minimum requirement. `minResources` is suitable for such scenarios. -## Note -#### Automatic Creation -If no PodGroup is specified when a VolcanoJob is created, Volcano will create a PodGroup with the same name as the VolcanoJob. diff --git a/content/en/docs/v1-7-0/schduler_introduction.md b/content/en/docs/v1-7-0/schduler_introduction.md deleted file mode 100644 index 791b2626..00000000 --- a/content/en/docs/v1-7-0/schduler_introduction.md +++ /dev/null @@ -1,103 +0,0 @@ -+++ -title = "Introduction" - - -date = 2024-01-16 -lastmod = 2024-01-16 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Overview" -[menu.v1-7-0] - parent = "scheduler" - weight = 1 - -+++ - -## Introduction -Volcano scheduler is the component responsible for pod scheduling. It consists of a series of actions and plugins. Actions define the action that should be executed in every step. Plugins provide the action algorithm details in different scenarios. -Volcano scheduler is highly scalable. You can specify and implement actions and plugins based on your requirements. -## Workflow -{{
}} - - -Volcano scheduler works as follows: - -1. Watches for and caches the jobs submitted by the client. -2. Opens a session periodically. A scheduling cycle begins. -3. Sends jobs that are not scheduled in the cache to the to-be-scheduled queue in the session. -4. Traverses all jobs to be scheduled. Executes enqueue, allocate, preempt, reclaim, and backfill actions in the order they are defined, and finds the most suitable node for each job. Binds the job to the node. The specific algorithm logic executed in the action depends on the implementation of each function in the registered plugins. -5. Closes this session. - -## Actions -### enqueue -The enqueue action is responsible for filtering out the tasks that meet the scheduling requirements based on a series of filtering algorithms and sending them to the to-be-scheduled queue. After the action is executed, the status of the task changes from `pending` to `inqueue`. -### allocate -The allocate action is responsible for selecting the most suitable node based on a series of prediction and optimization algorithms. -### preempt -The preempt action is responsible for preemptive scheduling of high priority tasks in the same queue according to priority rules. -### reclaim -The reclaim action is responsible for reclaiming the resources allocated to the cluster based on the queue weight when a new task enters the queue and the cluster resources cannot meet the needs of the queue. -### backfill -The backfill action is responsible for backfilling the tasks in the `pending` state into the cluster node to maximize the resource utilization of the node. - -## Plugins -### gang -The gang plugin considers that tasks not in the `Ready` state (including Binding, Bound, Running, Allocated, Succeed, and Pipelined) have a higher priority. It checks whether the resources allocated to the queue can meet the resources required by the task to run `minavailable` pods after trying to evict some pods and reclaim resources. If yes, the gang plugin will evict some pods. -### conformance -The conformance plugin considers that the tasks in namespace `kube-system` have a higher priority. These tasks will not be preempted. -### DRF -The DRF plugin considers that tasks with fewer resources have a higher priority. It attempts to calculate the total amount of resources allocated to the preemptor and preempted tasks, and triggers the preemption when the preemptor task has less resources. -### nodeorder -The nodeorder plugin scores all nodes for a task by using a series of scoring algorithms. The node with the highest score is considered to be the most suitable node for the task. -### predicates -The predicates plugin determines whether a task is bound to a node by using a series of evaluation algorithms. -### priority -The priority plugin compares the priorities of two jobs or tasks. For two jobs, it decides whose priority is higher by comparing `job.spec.priorityClassName`. For two tasks, it decides whose priority is higher by comparing `task.priorityClassName`, `task.createTime`, and `task.id` in order. -## Configuration -Volcano scheduler is highly scalable because of its composite pattern design. Users can decide which actions and plugins to use according to their needs, and they can also implement customization by calling the action or plugin interfaces. The scheduler configuration is located in the ConfigMap named **volcano-scheduler-configmap**, which is mounted as a volume into the "/volcano.scheduler" directory in the scheduler container. -### How to Get Configuration of Volcano Scheduler -* Get the ConfigMap named `volcano-scheduler-configmap`. - -```shell -# kubectl get configmap -nvolcano-system -NAME DATA AGE -volcano-scheduler-configmap 1 6d2h -``` - -* View the details of the data part in the ConfigMap. - -```shell -# kubectl get configmap volcano-scheduler-configmap -nvolcano-system -oyaml -apiVersion: v1 -data: - volcano-scheduler.conf: | - actions: "enqueue, allocate, backfill" - tiers: - - plugins: - - name: priority - - name: gang - - name: conformance - - plugins: - - name: drf - - name: predicates - - name: proportion - - name: nodeorder - - name: binpack -kind: ConfigMap -metadata: - annotations: - kubectl.kubernetes.io/last-applied-configuration: | - {"apiVersion":"v1","data":{"volcano-scheduler.conf":"actions: \"enqueue, allocate, backfill\"\ntiers:\n- plugins:\n - name: priority\n - name: gang\n - name: conformance\n- plugins:\n - name: drf\n - name: predicates\n - name: proportion\n - name: nodeorder\n - name: binpack\n"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"volcano-scheduler-configmap","namespace":"volcano-system"}} - creationTimestamp: "2020-08-15T04:01:02Z" - name: volcano-scheduler-configmap - namespace: volcano-system - resourceVersion: "266" - selfLink: /api/v1/namespaces/volcano-system/configmaps/volcano-scheduler-configmap - uid: 1effe4d6-126c-42d6-a3a4-b811075c30f5 -``` - -`actions` and `tiers` are included in `volcano-scheduler.conf`. In `actions`, the comma is used as a separator to configure the actions to be executed by the scheduler. Note that the scheduler will execute the actions in the order that they are configured. Volcano itself will not check the rationality of the order. The list of plugins configured in `tiers` is the plugins registered with the scheduler. The specific algorithms defined in plugins will be called in `actions`. diff --git a/content/en/home/about.md b/content/en/home/about.md deleted file mode 100644 index bff522f9..00000000 --- a/content/en/home/about.md +++ /dev/null @@ -1,13 +0,0 @@ -+++ -# About/Biography widget. -widget = "about" # Do not modify this line! -active = true # Activate this widget? true/false -weight = 2 # Order that this section will appear in. -date = 2016-04-20T00:00:00 - -title = "About Volcano" - -# Choose the user profile to display -# This should be the username of a profile in your `content/author/` folder. -author = "Volcano" -+++ diff --git a/content/en/home/hero.md b/content/en/home/hero.md deleted file mode 100644 index 7bb0e2f3..00000000 --- a/content/en/home/hero.md +++ /dev/null @@ -1,36 +0,0 @@ -+++ -# Hero widget. -widget = "hero" -active = false -date = 2017-10-15T00:00:00 - -title = "Volcano" - -# Order that this section will appear in. -weight = 3 - -# Overlay a color or image (optional). -# Deactivate an option by commenting out the line, prefixing it with `#`. -[header] - overlay_color = "#666" # An HTML color value. - overlay_img = "headers/bubbles-wide.jpg" # Image path relative to your `static/img/` folder. - overlay_filter = 0.5 # Darken the image. Value in range 0-1. - -# Call to action button (optional). -# Activate the button by specifying a URL and button label below. -# Deactivate by commenting out parameters, prefixing lines with `#`. -[cta] - url = "/en/docs/setup/" - label = ' Install Now' -+++ - -An open platform to enable Edge computing :rocket: - -
- Star -
- diff --git a/content/en/home/home_slider.md b/content/en/home/home_slider.md deleted file mode 100644 index 48dbe124..00000000 --- a/content/en/home/home_slider.md +++ /dev/null @@ -1,127 +0,0 @@ -+++ -# Hero Carousel widget. -widget = "slider" -active = true -date = 2017-10-15T00:00:00 -lala = "headers/banner_02.png" - -# Order that this section will appear in. -weight = 1 - -# Slide interval. -# Use `false` to disable animation or enter a time in ms, e.g. `5000` (5s). -interval = 5000 - -# Minimum slide height. -# Specify a height to ensure a consistent height for each slide. -height = "500px" - -# Slides. -# Duplicate an `[[item]]` block to add more slides. -[[item]] - title = "Volcano" - #content = "Erupt with Computing Power
Star" - content = "Cloud native batch scheduling system for compute-intensive workloads" - content1 = "Star" - align = "left" # Choose `center`, `left`, or `right`. - - # Overlay a color or image (optional). - # Deactivate an option by commenting out the line, prefixing it with `#`. - #overlay_color = "#666" # An HTML color value. - #overlay_img = "headers/banner_02.png" # Image path relative to your `static/img/` folder. - #overlay_img = "headers/banner_02.png" - overlay_filter = 0.00 # Darken the image. Value in range 0-1. - - # Call to action button (optional). - # Activate the button by specifying a URL and button label below. - # Deactivate by commenting out parameters, prefixing lines with `#`. - cta_label = "Learn More About Volcano" - cta_url = "docs/" - cta_icon_pack = "fas" - cta_icon = "graduation-cap" - -[[item]] - title = "Network Topology Aware Scheduling" - content = "Supports network topology aware scheduling, significantly reducing application communication overhead between nodes, and greatly enhancing model training efficiency in AI distributed training scenarios" - align = "left" - overlay_filter = 0.15 - -[[item]] - title = "Online and Offline Workloads Colocation" - content = "Supports online and offline workloads colocation, enhancing resource utilization while ensuring online business QoS through unified scheduling, dynamic resource overcommitment, CPU burst capabilities, and resource isolation" - align = "left" - overlay_filter = 0.20 - -[[item]] - title = "Multi Cluster Job Scheduling" - content = "Supports cross cluster job scheduling for larger scale resource pool management and load balancing" - content1 = "Star" - align = "left" - overlay_filter = 0.25 - - # Call to action button for multi-cluster scheduling - cta_label = "Learn more about volcano multi-cluster scheduling" - cta_url = "https://github.com/volcano-sh/volcano-global" - cta_icon_pack = "fas" - cta_icon = "graduation-cap" - -[[item]] - title = "Hierarchical Queue Management" - content = "Supports multi-level queue management for fine-grained resource quota control" - align = "left" - overlay_filter = 0.3 - -[[item]] - title = "Descheduling" - content = "Supports dynamic descheduling to optimize cluster load distribution and improve system stability" - content1 = "Star" - align = "left" - overlay_filter = 0.35 - - # Call to action button for descheduling - cta_label = "Learn more about volcano descheduling" - cta_url = "https://github.com/volcano-sh/descheduler" - cta_icon_pack = "fas" - cta_icon = "graduation-cap" - -[[item]] - title = "Multiple Scheduling Policies" - content = "Supports various scheduling strategies including Gang, Fair-Share, Binpack, DeviceShare, Capacity, Proportion, NUMA aware, and Task Topology, optimizing resource utilization" - align = "left" - overlay_filter = 0.4 - -[[item]] - title = "Heterogeneous Device Support" - content = "Supports scheduling for various heterogeneous devices like GPU and NPU, unleashing hardware computing power" - align = "left" - overlay_filter = 0.45 - -[[item]] - title = "High Performance Unified Scheduling" - content = "Supports native Kubernetes workload scheduling while providing complete support for frameworks like Ray, PyTorch, TensorFlow, Spark, and Flink through VolcanoJob, achieving unified job scheduling with excellent performance" - align = "left" - overlay_filter = 0.5 - -[[item]] - title = "Powerful Monitoring" - content = "Provides rich logging, monitoring metrics, and dashboards" - content1 = "Star" - align = "left" - - # overlay_color = "#333" # An HTML color value. - # overlay_img = "headers/volcano-slide-2.png" # Image path relative to your `static/img/` folder. - overlay_filter = 0.55 # Darken the image. Value in range 0-1. - - # Call to action button for observability - cta_label = "Learn more about volcano dashboard" - cta_url = "https://github.com/volcano-sh/dashboard" - cta_icon_pack = "fas" - cta_icon = "graduation-cap" - -+++ - - \ No newline at end of file diff --git a/content/en/home/posts.md b/content/en/home/posts.md deleted file mode 100644 index dcbb041b..00000000 --- a/content/en/home/posts.md +++ /dev/null @@ -1,66 +0,0 @@ -+++ -# A Recent Blog Posts section created with the Pages widget. -# This section displays recent blog posts from `content/post/`. - -widget = "pages" # Do not modify this line! -active = true # Activate this widget? true/false -weight = 5 # Order that this section will appear. - -title = "Recent Posts" -subtitle = "[View all ->](blog/)" - -[content] - # Page type to display. E.g. post, talk, or publication. - page_type = "posts" - - # Choose how much pages you would like to display (0 = all pages) - count = 5 - - # Choose how many pages you would like to offset by - offset = 0 - - # Page order. Descending (desc) or ascending (asc) date. - order = "desc" - - # Filter posts by a taxonomy term. - [content.filters] - tag = "" - category = "" - publication_type = "" - exclude_featured = false - -[design] - # Toggle between the various page layout types. - # 1 = List - # 2 = Compact - # 3 = Card - # 4 = Citation (publication only) - view = 2 - -[design.background] - # Apply a background color, gradient, or image. - # Uncomment (by removing `#`) an option to apply it. - # Choose a light or dark text color by setting `text_color_light`. - # Any HTML color name or Hex value is valid. - - # Background color. - # color = "navy" - - # Background gradient. - # gradient_start = "DeepSkyBlue" - # gradient_end = "SkyBlue" - - # Background image. - # image = "background.jpg" # Name of image in `static/img/`. - # image_darken = 0.6 # Darken the image? Range 0-1 where 0 is transparent and 1 is opaque. - - # Text color (true=light or false=dark). - # text_color_light = true - -[advanced] - # Custom CSS. - css_style = "" - - # CSS class. - css_class = "" -+++ diff --git a/content/en/home/required.md b/content/en/home/required.md deleted file mode 100644 index 46e2a579..00000000 --- a/content/en/home/required.md +++ /dev/null @@ -1,119 +0,0 @@ -+++ -# Custom widget. -# An example of using the custom widget to create your own homepage section. -# To create more sections, duplicate this file and edit the values below as desired. -widget = "featured" -active = true -date = 2016-04-20T00:00:00 - -title = "Rich Framework Support" -subtitle = "Seamlessly integrate with mainstream computing frameworks for AI, big data, and scientific computing" - -# Order that this section will appear in. -weight = 4 - -# Showcase personal skills or business features. -# -# Add/remove as many `[[feature]]` blocks below as you like. -# -# Available icon packs and icons: -# * fas - Font Awesome standard icons (see https://fontawesome.com/icons) -# * fab - Font Awesome brand icons (see https://fontawesome.com/icons) -# * ai - academic icons (see https://jpswalsh.github.io/academicons/) - -[[featured]] - img_src = "spark-logo-hd.png" - img_width = "100px" - img_height = "60px" - name = "Spark" - url = "https://spark.apache.org/ " - description = "Apache Spark™ is a unified analytics engine for large-scale data processing." - -[[featured]] - img_src = "volcano_flink.PNG" - img_width = "100px" - img_height = "60px" - name = "Flink" - url = "https://flink.apache.org/ " - description = "Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams." - -[[featured]] - img_src = "volcano_tensorflow.PNG" - img_width = "100px" - img_height = "60px" - name = "TensorFlow" - url = "https://www.tensorflow.org/ " - description = "An end-to-end open source machine learning platform." - -[[featured]] - img_src = "volcano_pytorch.PNG" - img_width = "100px" - img_height = "60px" - name = "PyTorch" - url = "https://pytorch.org/ " - description = "An open source machine learning framework that accelerates the path from research prototyping to production deployment." - -[[featured]] - img_src = "volcano_argo-horizontal-color.png" - img_width = "100px" - img_height = "60px" - name = "argo" - url = "https://argoproj.github.io/ " - description = "Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD." - -[[featured]] - img_src = "volcano_mindspore.PNG" - img_width = "100px" - img_height = "60px" - name = "MindSpore" - url = "https://www.mindspore.cn/en " - description = "The all-scenario deep learning framework developed by Huawei." - -[[featured]] - img_src = "ray_logo.png" - img_width = "300px" - img_height = "80px" - name = "Ray" - url = "https://github.com/ray-project/ray" - description = "Ray is a high-performance distributed computing framework that supports machine learning, deep learning, and distributed applications." - -[[featured]] - img_src = "kubeflow.png" - img_width = "100px" - img_height = "60px" - name = "Kubeflow" - url = "https://www.kubeflow.org/ " - description = "The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable, and scalable." - -[[featured]] - img_src = "volcano_openMPI.jpg" - img_width = "100px" - img_height = "60px" - name = "Open MPI" - url = "https://www.open-mpi.org/software/ompi/v4.0/ " - description = "The Open MPI Project is an open source Message Passing Interface implementation that is developed and maintained by a consortium of academic, research, and industry partners." - -[[featured]] - img_src = "volcano_horovod.PNG" - img_width = "100px" - img_height = "60px" - name = "Horovod" - url = "https://github.com/horovod/horovod " - description = "Horovod is a distributed deep learning training framework for TensorFlow, Keras, PyTorch, and Apache MXNet." - -[[featured]] - img_src = "volcano_mxnet.PNG" - img_width = "100px" - img_height = "60px" - name = "MXNet" - url = "https://mxnet.apache.org/versions/1.6/ " - description = "A truly open source deep learning framework suited for flexible research prototyping and production." - -[[featured]] - img_src = "volcano_paddle.PNG" - img_width = "100px" - img_height = "60px" - name = "PaddlePaddle" - url = "https://www.paddlepaddle.org.cn/ " - description = "PaddlePaddle is an open source deep learning platform derived from industrial practice initiated by Baidu." -+++ diff --git a/content/en/home/skills.md b/content/en/home/skills.md deleted file mode 100644 index 742b61cb..00000000 --- a/content/en/home/skills.md +++ /dev/null @@ -1,53 +0,0 @@ -+++ -# Feature/Skill widget. -widget = "featurette" # Do not modify this line! -active = false # Activate this widget? true/false - -title = "Why Volcano" -subtitle = "Are you planning to Deploy any of the below workloads on Kubernetes?" - -# Order that this section will appear in. -weight = 6 - -# Showcase personal skills or business features. -# -# Add/remove as many `[[feature]]` blocks below as you like. -# -# For available icons, see: https://sourcethemes.com/academic/docs/widgets/#icons - -[[feature]] - icon = "r-project" - icon_pack = "fab" - name = "R" - description = "90%" - -[[feature]] - icon = "chart-line" - icon_pack = "fas" - name = "Statistics" - description = "100%" - -[[feature]] - icon = "camera-retro" - icon_pack = "fas" - name = "Photography" - description = "10%" -[[featured]] - img_src = "kubernetes-200.png" - img_width = "100px" - name = "Kubernetes" - description = "Gaps for High Performance Workloads" - -[[featured]] - img_src = "mosquitto-200.png" - img_width = "100px" - name = "Mosquitto" - description = "Eclipse Mosquitto is an open source (EPL/EDL licensed) message broker that implements the MQTT protocol versions 3.1 and 3.1.1. Mosquitto is lightweight and is suitable for use on all devices from low power single board computers to full servers." - -[[featured]] - img_src = "docker-200.png" - img_width = "100px" - name = "Docker" - description = "Building and deploying new applications is faster with containers. Docker containers wrap up software and its dependencies into a standardized unit for software development that includes everything it needs to run: code, runtime, system tools and libraries." - -+++ diff --git a/content/en/home/supporter.md b/content/en/home/supporter.md deleted file mode 100644 index 531da667..00000000 --- a/content/en/home/supporter.md +++ /dev/null @@ -1,165 +0,0 @@ -+++ -# Custom widget. -# An example of using the custom widget to create your own homepage section. -# To create more sections, duplicate this file and edit the values below as desired. -widget = "supporter" -active = true - -title = "Supporters" - -# Order that this section will appear in. -weight =7 - -# Showcase personal skills or business features. -# -# Add/remove as many `[[feature]]` blocks below as you like. -# -# Available icon packs and icons: -# * fas - Font Awesome standard icons (see https://fontawesome.com/icons) -# * fab - Font Awesome brand icons (see https://fontawesome.com/icons) -# * ai - academic icons (see https://jpswalsh.github.io/academicons/) - -[[featured]] -img_src = "logo_huawei.png" -img_width = "100px" -img_height = "60px" -url = "https://www.huawei.com " - -[[featured]] -img_src = "logo_bilibili.png" -img_width = "100px" -img_height = "60px" -url = "https://www.bilibili.com/" - -[[featured]] -img_src = "logo_360.png" -img_width = "100px" -img_height = "60px" -url = "https://www.360.cn/" - -[[featured]] -img_src = "logo_tencent.png" -img_width = "100px" -img_height = "60px" -url = "https://www.tencent.com/ " - -[[featured]] -img_src = "logo_baidu.png" -img_width = "100px" -img_height = "60px" -url = "https://www.baidu.com/ " - -[[featured]] -img_src = "logo_4paradigm.png" -img_width = "100px" -img_height = "60px" -url = "https://www.4paradigm.com/ " - -[[featured]] -img_src = "logo_iqiyi.png" -img_width = "100px" -img_height = "60px" -url = "https://www.iqiyi.com/ " - -[[featured]] -img_src = "logo_redbook.png" -img_width = "100px" -img_height = "60px" -url = "https://www.xiaohongshu.com/ " - -[[featured]] -img_src = "logo_vips.png" -img_width = "100px" -img_height = "60px" -url = "https://www.vip.com/ " - -[[featured]] -img_src = "logo_didi.png" -img_width = "100px" -img_height = "60px" -url = "https://es.didichuxing.com/ " - -[[featured]] -img_src = "logo_jd.png" -img_width = "100px" -img_height = "60px" -url = "https://www.jd.com/ " - -[[featured]] -img_src = "logo_bosszhipin.png" -img_width = "100px" -img_height = "60px" -url = "https://www.zhipin.com/?sid=sem_pz_bdpc_dasou_title " - -[[featured]] -img_src = "logo_jianxinjinke.png" -img_width = "100px" -img_height = "60px" -url = "http://www.ccb.com/cn/home/indexv3.html " - -[[featured]] -img_src = "logo_qvtoutiao.png" -img_width = "100px" -img_height = "60px" -url = "https://www.qutoutiao.net/ " - -[[featured]] -img_src = "logo_ruitian.png" -img_width = "100px" -img_height = "60px" -url = "https://www.ruitiancapital.com/#/ " - -[[featured]] -img_src = "logo_boyun.png" -img_width = "100px" -img_height = "60px" -url = "https://www.bocloud.com.cn/ " - -[[featured]] -img_src = "logo_zhongkeleinao.png" -img_width = "100px" -img_height = "60px" -url = "http://www.leinao.ai/ " - -[[featured]] -img_src = "logo_bibdr.png" -img_width = "100px" -img_height = "60px" -url = "http://www.bibdr.org/nd.jsp?id=53 " - -[[featured]] -img_src = "logo_yunzhisheng.png" -img_width = "100px" -img_height = "60px" -url = "https://dev.hivoice.cn/ " - -[[featured]] -img_src = "logo_qiezi.png" -img_width = "100px" -img_height = "60px" -url = "https://shareit.one/ " - -[[featured]] -img_src = "logo_vivo.png" -img_width = "100px" -img_height = "60px" -url = "https://www.vivo.com.cn/ " - -[[featured]] -img_src = "logo_xiwangzu.png" -img_width = "100px" -img_height = "60px" -url = "https://www.grandomics.com/ " - -[[featured]] -img_src = "logo_ktnexr.png" -img_width = "100px" -img_height = "60px" -url = "https://github.com/nexr/ " - -[[featured]] -img_src = "logo_openinnovation.png" -img_width = "100px" -img_height = "60px" -url = "https://www.openinnovation.ai/" -+++ diff --git a/content/en/home/talks.md b/content/en/home/talks.md deleted file mode 100644 index d6ebb1b7..00000000 --- a/content/en/home/talks.md +++ /dev/null @@ -1,68 +0,0 @@ -+++ -# A Recent and Upcoming Talks section created with the Pages widget. -# This section displays recent talks from `content/talk/`. - -widget = "pages" # Do not modify this line! -active = false # Activate this widget? true/false -weight = 6 # Order that this section will appear. - -title = "Recent & Upcoming Talks" -subtitle = "[View all ->](talk/)" - -[content] - # Page type to display. E.g. post, talk, or publication. - page_type = "talk" - - # Choose how much pages you would like to display (0 = all pages) - count = 5 - - # Choose how many pages you would like to offset by - offset = 0 - - # Page order. Descending (desc) or ascending (asc) date. - order = "desc" - - # Filter posts by a taxonomy term. - [content.filters] - tag = "" - category = "" - publication_type = "" - exclude_featured = false - exclude_past = false - exclude_future = false - -[design] - # Toggle between the various page layout types. - # 1 = List - # 2 = Compact - # 3 = Card - # 4 = Citation (publication only) - view = 2 - -[design.background] - # Apply a background color, gradient, or image. - # Uncomment (by removing `#`) an option to apply it. - # Choose a light or dark text color by setting `text_color_light`. - # Any HTML color name or Hex value is valid. - - # Background color. - # color = "navy" - - # Background gradient. - # gradient_start = "DeepSkyBlue" - # gradient_end = "SkyBlue" - - # Background image. - # image = "background.jpg" # Name of image in `static/img/`. - # image_darken = 0.6 # Darken the image? Range 0-1 where 0 is transparent and 1 is opaque. - - # Text color (true=light or false=dark). - # text_color_light = true - -[advanced] - # Custom CSS. - css_style = "" - - # CSS class. - css_class = "" -+++ diff --git a/content/en/publication/_index.md b/content/en/publication/_index.md deleted file mode 100644 index 22984353..00000000 --- a/content/en/publication/_index.md +++ /dev/null @@ -1,17 +0,0 @@ -+++ -title = "Publications" -date = 2017-01-01T00:00:00 - -# List format. -# 0 = Simple -# 1 = Detailed -# 2 = APA -# 3 = MLA -# 4 = Stream -list_format = 3 - -# Optional featured image (relative to `static/img/` folder). -[header] -image = "" -caption = "" -+++ diff --git a/content/en/talk/_index.md b/content/en/talk/_index.md deleted file mode 100644 index 875bcbb6..00000000 --- a/content/en/talk/_index.md +++ /dev/null @@ -1,15 +0,0 @@ -+++ -title = "RECENT & UPCOMING TALKS" -date = 2017-01-01T00:00:00 - -# List format. -# 0 = Simple -# 1 = Detailed -# 2 = Stream -list_format = 2 - -# Optional featured image (relative to `static/img/` folder). -[header] -image = "" -caption = "" -+++ diff --git a/content/en/talk/kubecon-barcelona-2019/featured.png b/content/en/talk/kubecon-barcelona-2019/featured.png deleted file mode 100644 index b99cb8d3abf6ec545a783a8fa1208080fc642b3e..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 273373 zcmW)nWn2_(7shFK=@O8RMFi=TM!G>lV(E@unx&*`GdC=} z=Or4-FO34R?TA09{<}y$dHmmr>UYeRulj4`e|PGtZ{>Hpx)rTvZUFz<`2n5QH&gPuuU%6Gq0%*V z9!*&2Iu&8B9=^#RzZ@7!4Lv^){APiYdhGusjdA*Uk-rvH66sV=3RYHK+9FI2a1~KP z6*+u5mNxpzD+6BPf3zRDYs!~3P@SZAH}u)+O5LII3CDL~>q#wB=?=1qnh#;0xz)47 zfZutQoh=P5Y(oXF5~#wX9DYuTMj|hDcLw}^p>A3;8|^4xys_-C@fs`b78ms$IlK|w z4$9j`S+^ad30<9n&&5Y3d0L<}ZmLI-n;IX3gtb$Bmp?h-!Y7sn{MKD=8J%9f;pahH zb6e}=0l0ANvVKH>UV8jV8eF&4{<-#LpFr)|Bkzoq7_RFrimV~$&Y+Pf zq7Rm2*w|@@}s zAedrff;i^WVq$%cSwguyrdlT>`+aU0M~+x%Wj-D?`AmkI&~B1Jhy^~yAsPRN2va*dJ6L|q;7BoI-v!`6m; zS31rmff9>5?~>Y?-KopkbuxIbqOGKHbpOSVyZI%$lj>p{tT89|EQvj>wdqh~ zWS|BqHl8S!GNbwIYu{8Xc4`N@9CH<8N(Ex&&g71B3Abe++xXq)O%J^--I9f|ZzK<0 zal$~dL(W#_IVam%R+sf#vyQh@7Hxd`Y^*^D)7dKM`(IYF66u|D8KqpK#71T>E3xoW zD{UEL#X;J$cnTZTyUV#Teuxj5e0Q*n#{rr_GbRl{GK14o>(vXAO zya*YgrMLb3h#o0$`gp5y+WODhotW5k`gogYU{F)%Bfx^tt6&V(KXuv@HQs;VzC&=q z;#Ww|^6lCwQM)dNu&*Fm-fz9EC55eR*VyYRoGS22fB-VF*@$17r8QUKr)66*J{I?oJO4N)n~+0y;EbpAq3N34uxOpf0yHry9G zLUtS$z9660hZ-yyu3xzwx%ZM`BrTLLhYVmvcKRrFQNI*5FyOL|EIJZ~*kS1(fh7Q0 zZ#(%(M`>SwCRcwH{@U?$Ht_c=*xeQFuO*;DW&RC=E(tvtWuLA{ZTq)slT)yod*Z6>#`~TM1lJmwIcS~aycP_uX#Ac9|6cpaC-{1FH-z~Dphu;}dyTA$J zYAmWVu;SJXBcibNyNoa3HmL!R^NV1Gk1it5bv#B~Z{)GCJH0~9*ms2#f!*DuYlgS; zSqSw{tOv0DBl+Ulo6BBh{qI}0=_B{1|6~peW0k(^KZ_$}#VcmRCmwKdhhG6xMZW*S z45c}=pjYI>7)QK&!lm}kk7c6pB@?r#SW|6#SF)~0P#@Y0pDJKKIaFi5Q@XSm(L2_5 zC;6wD`;qIuNK~I(Pp~y-S}rfp|rI6@sUHuXzQ+=jZ-u;S{LNirv=sXg*V+$xr18;VA@OD; z=|I)cAmz>65`bTF*RP~*Q{D0io`Hfjc8xfwFR+S(!%YSIb|>XdR%&>!rQNxpvWhe6L6vSH>v zl227+?UDE{z1DV6Cn}1DVThzNk_T(|4_wRV-0qM7yA#H55)Ee+THLxJir@YRv2L`; zyjpu{PrdwSqO<(sUv2pHqwtIQ@Z0&qyCW7CpeAnOPflkah1(BVd@pwz;%7{(l0j>H zjrgg5r;ef)nJ-wdaB6xu6oT!P!v)c^vR&j~Dm9vA8DsBF#lojNd{k5WgXs0s?f+mLtu4keBcl><{OQ%*7fz2XzNt2^1UF)xVs@9x6h z-29D~k(J#iJ9j<*y;m+~C8$#^q!+2{bT9{xz&V4}z7H4Al*m-gY(Hqk*Agb#cySX; zugd)ei`;k28Qip;BS+R@uj5;#X@`;>g$bjfbz&U>z;6i4PcSYRwW#ygH;c5iE9TyY zQL_8-;1R?-1w6mcKH}dHHx@&aUw_(p{v>SlHxBK0YWRkVKXCj)%9AW{4sb2!PgVg6;<*;TRmqP!*pc!s z&>Eg;({GuQmVkH(83~3_zUfKTejSYU`C`Ew=R!=q%SceptQ{`TocYp0SQ*on_ME1_ zLJ{_i0>HpPun7eVI8RD4_OJdbQ_gdHZSwQ&XXdEMx7)5B_BzIooybjqYA(>$_hKU( z6M+Hq!`D(JluT(-+KViI4i1V)MzlvPR)469D+2h>YuP?hEfNJz3Ua*s`9{8w-=`|^ zHyYYZ@KS%KngkQ(gefD!YP>0Y9PQzZqZL>2V();NSx#mt=Jr9IzewsZXkzJYQ{s^0 zTg=L5@*P2|^@?lTs6^hINPX23O?KB{lv&KXuh-n!Uv z>JeWca4_>zyA#N+L;i07P+ZaOuiJ7gFB3AxIO#h6soJ+1D)K=HV{eKN@4F<7wXFv(&?546365@3x>8DJp~k{5S^&Nnb- zt>U`=(h05Xm#&m&tCB{lWV6qiqe&D$99!roM=U`GB6NG+9N2|W;owsk#6RxU_6ODD zZ@1p?1$ABEKMP#C2WOpKhvq@?*|L1kGhf!FZivb%FpRX}Og(n2lwvF|{9EW9&>}y9 zMSVr%4b+k^)659Ydd@^Y+Am0|+@lkO^W`^5amr9j9d?Cfw?%^-MPN~hm&5_{!QmmS zTi055a!@v!l6$_y_P@%m|k&W1}EI3n6!n~plQ=mD@QJ(j!p%UL$&Xk}RbJlE@_ zuKVQ7;H;>d*OWRdg)3=YHmDN9%0vWbC=U6dV$W)Fwkd8YF5LDalviG&FYU}d?PFOb z+S%tNgR;h3Q3jKgpt7-t)(s~?BOA(pjel8l zktrD8mfZbz&EdiU1i~mE$3I&%QJ`QC;52C5j_ih>E!t8Kmyh)-bFO*gn6H9>T zB1;yVghjdVl15oKN)J4tn9ejW2mIeL>iy^OAC?(8o&$d5*8gDn{aWEuHi}+T@hi24 zyUr6om(*g-O74`nNcMe)Mk?X?lmVIMO2T*GH)KdHzGGS#8qIy) z5XmWh@@HAkechWfDoIYN<-a%SB6bL({l1A?+jmp2*7_~@0G*Vc=AZ?YEXO_aR#pxM zVnREFRuLGZb1S8x^Ksm-5c}na8OnC25)22$SkEf`7km4cgCQKP9IX zzM_>vkxxG-#0FeS#$}ZbHneNqJyuj;rR&3LS}!*-O&sug!FZ;p7Xbja`7wm~$i`^Y zaO7_aIMwu{C+XfgtOTdBngKsjWIc}!a!Z4{bQA$=Zp`aAp*7lK(}wt=Lakrp8h|jv zw3F_#-+W(^Oa?}Ncrn4s9VBXoq)XK_@GN}E+q0tp4Z&>r>d={%{Mv8% zrJoaxTP6Gd{*{Uu+MMGfpkHIvolUk8WP8<@S^jkXo|?b^oGR2Ud2(h-k~79?bTZlM z2+wDHm}ouYFH;=?96lKL`zuVS_waGsB>Jlr4qIyE-X{YM9Kn**3|kk(W?){GkCaZJ zT=LS-83O36(L#yw<57~7*w|PyGBV$}Oo76OgTq-J@HS>5J67C?5TMC={6>3DFu~2m zMy+Nm43E~G$Ey6c3gpnm_f~fCRFQRCV{-YYl=yqv+%HS!hG1Vh0+mQ&rP=pX-l?Ed zq7t5C87^!^E3Z2|I5|MMSQP#6;(D}5Gm}p1=~ZwZ78FHyj{e;+_v(l^i#Bncj(hDG zzbr`BW#-cC!;ebKlBM{0;d@`@O&(zHj$0ruq0uL3Qa~14zP-I6`u-I=ye>6}vs%_V za*2~(!kVB4HfAY%yEkW}SbIpVi>V)(t?Y*IQ_$j#jS_<9VFkmYNLag zeZ>pPuiYP@euFDkqh&bS>|y&vlDXurR(&K21}h zM3F~57bepoj!d;c*(ql1cKo1|;59Pa=e6W3+bR_^z`~YT=FP8_;{;&^hZVeLV<-<* z4)s3Cb4#Md@opZX(Y|RvZ&;$*k!hM=5W7~Rsv<@lXfRx2l*+=da|%3v-@x__C@ z<H3;EUI&CA+MGS2Z7&|^a~OP1O$9N^11=yc)TaSz^<)bpQ_l9k!Sf(v7ET^ zf><0MUp?!2#iiif-6dCPS=&UgLdWm3_7&^X^BRsIGT-5SZ_I&qak0K9^gV3Ws0=H) zbmcd=TSB1vUU5}n4JXp(B}Tq0?6b|W*W=W=;;?dB*Wi6}e<_aSPHe+l%bXAW(u%9U z9$!Z0`X7)vUFYt#D0Gtdumd$(--SCLlKxm4&aX`hpH_!`x(+$KSH15TM2c?f+y|GwY5>5ev z91Sgh`=txHNvA(-Keqb-%bJd66Up?m@GJ zZ?4cmqAoN1E$_+L?JQ@9Jt;l#KVjM9Ww-I_up7O!`Ae3E9H!-13OiMa=G06y9~aGs z;j4#OW}rea&akDhPF~a=NHz_^B1bU0)KJ~~29`CoHL#94YY53ICLTuOYW%2JU0p7| zxW2!eDnN1yz4<7a1_j0XJpx>Q(lA(&_z-l&I?|& z&n^A!xQ11SHcY?=1&PCZO)+Yk$S7u7v#a8&sh$zx9`kDxSzpWfxE;%#}mU{{HubX9?H|jX@uRc>F6livv8tYy9qZE3+$S#^)-qU#lV-hSQ0q z0$T2q&B&kvca`t!USFC|R_Fa!kXzag&IZ$Rp;ub3P>#?p>HYbTTOrgEp}d)h(dg(` zRHL67Nrghz$*kCYj4fTvy2*!?(1;;=VF9+DeL(4vG`MH8oPq7{BQEsI^ALL__GEZ> zH52H%5;wPykWF!Pxa_Vo6ZS zDR0yDFJN-(5s`0$1@FNb+ zk%0h0>%8VGAN^&O+KbfvtapJc2}X2L*l0yg9Ls#|?mLC^@aTAe1~OOz3Bo-?Cx9x_ z|MlG89yVELG&egPQ%6Sk#&L}=48}WL%O(|$seAsw`Ew~wC6!CPt=&?Yc;Iir<#bl^XEnsX=ATDz}zyH@Q~y9+TOLd=razfycsH7cvJ+HGk+yWJA&;1j zLV|bS1A1&5{*AkvUCTbU{m#JrL1F9qGJ1rgcWY^0jcL_y;%p?UZH@Pr8wgF-Jxb%; z?#!duF2|Rb-Q~>fy;R~MFOA!e8k1r1@RVPSKqJeJ*%mL1_1@Tb1h6DwMBBu)HWw{; zY@vGhma36Kizb?=gmi&*CFK5aJ zJKYh5MA}Tc6!XcL_B@9rngPA)PdlM$i920HWd`ESP54svmlA0E$0vfsEFPT0fA10q z#P8NAY(L-oId3`7wq*SHAcUsy&{^toMvyrtycqX8l+-a)jy(JEP_q_oTvIH?o-f6(;O65KoU>U2lZ+GyXjcH3oYiWY_3!aI>iQ`#!LTfYGO|Pd z&0)GN-7sBEYhCgpiM>bL>bI6pE#W;|OvcH=f~os|5&r%J$wur88YN?mr0J9qz0-O? zL92|?bAl#R-Q(1uY+~`+KFS5xGzYoG<;^P_PWY>5nplW{#>PYrBTf9^!|;EM8Y}l7 z6|`slUQBR)MM6jH!Ke(4XRIWOH}p6f>aL9spS74Eev0j>_Aci`hpjlH+M9`r!<!!zs_xvZ+9;5G8@-~%l|xjg!81Td!)r}MokwPh2{KI-z_Lm!wr2> zz*S1EAwFGCQH^f*KmncEc`@w@HJ7&S^ej9SavYjRy1_neU+{JLB`QjtwvdPm;@-^7 zzO;RmxHXyZeKzFd7qP9NrID6trDjY9ETta{NH^XvBarQ2qrim}?qW3gGHY~qg{g(s zCU(}45U$x?)R0c4UeEvwu_*}g|Ip<=2;}+XPp&(}nuyHkr=&~=2djQ{4ooPW_?q6Y z$xGY^kpvVs7wpFA<1H>Op2#k#rki6>l)g^M3VH%U5<7}n+6?alOXDd&*c3Q(ALQO9 z@78wTAocgo4u`wjw-+3z)8r**>2WI3gIExWl-Ho&NDn@d7sr`Hh%m?9_cqQU?un%X zGCKYR&bg~A9pU|u?3b)l@Z|3Yvi6mw@ZhD0OP2>V#G{il8f1#Gols@Q6F$-QQm!aH z$*>7Dl?vJAXm5TT-ocaS{jBU(yzAWG60Em7*Pg8G6(X(YZ-w%A2g4Z9{5;d78u65z z#nqS`APpVwD@_XmE)lA?^4X$uKEA#pI{qtH4JWg5ROEC86YuUG*V6z6ZE#nS@S(HG z$$i)HA8Xi5^;)#gCNdjkWnRdV;K1tI_0#KV@}6SQ^(O#RuIZe)cl`;vnmYZ?(y(A=GcmuD z*l|baTSPnc_?kb)dI4Kgo^z3OIB6*(hIJN=WF|dTHt%CqDx3GucdKa5TX2Uq8zNwD@od%(Yj(2TGZRZNHHsNnQQ&b$C}LXqUNPSr6tn-P_XLq*v!PYZ z*L?8abDls-Xf!O=eF4#8W(ZoLC7WFr)HB>)%-Hm}|0b7{h<~+|g}5mzJ)W4w@vdzS z3kcvEDo5*uhh8-?WfYl4#1U7{aLod^BSPTZLjfiClx{He68}8X6xgfibvC@jbpxZvUrN}d?=jmHNX7n z1YW^&-l!m@)HR+moV&KE#^#G!W!{bRc=6Qs5*3IoZE|Hyq$B(E8 zFqi8Uh53+2zD{@^h|^WAybY@@C)@>(E>8p#sYFXlOZa{0y!5A9;@Rxc->r;bd3S_t zu(D=^P}mvu<#|XSB}K}jNa7SWL0&fOZGu8Ovu4Ao7i~d#MS25c62c#Clh%LyCd&TUuA@`280%H~RucF@>_*#$fmC(3~xEElj>B!Os4k z*Nn_`K)}c<6`r26LgXtDWEpCxa|N7&04S$4jQ8)5h1sQI)$Tn8`<5JL80MoEP(5&Ar1sv=Y5vOQy z-M3(t^r)C^Bh$wu7TWyXW*Vhk9c;-!kRTO-LmnX#f@ff% zcM%__S19)paCg+spDKvFZ$6pmJ*79#1>}2zfB6j{#Ck2{^bp=l!P8I-xn6?a-5mr> zaozD`vz4*MAHzgt%F^W0If+7cblhNOtz&a@5zSEO*7lQNy)qwmF$noXv;!e~(5hOo z@(Wp#VFAlGY%yfUFWm%9UC>J!h@7%iWIKZVTgQB}%8votRdBz!J(henzh4&$qzTU| z^YH}>_*e34DEYu^+ilpN2p2CW9mq9};AyExK&>5}WaU?@(#)qvAR%aR=cT+|NP#=| zx24Rnv!o|&r`!Fn$`Q_MD8%XD+tBO4iK*<74Oo_5-`Q4^CqjC1TKELD@EHLT{4Yl7 zeJG~N3nTBkwG@ZEno0>HI_gVskdA9~nZ&dg{8K|1rvRL?fL7SlPE^lO3)Od+xE(2V*NM@})nZ?yTrN(2*#I z)`#(6EmzC=>u$NY7Zo41G#~Kxob;_gH^))Pzgw92oYeHAiFQ7ObyD4~lEl^2#D))V zsrIbQ9MnuAEQ4(pS^9!4@iO{9xbEs~s0z|jr8o2~EV2%R{5oGxf!v$oly*U(1tHw8 zs)Ad1Ilx_qRH@Tt2DvRx7bdNzr={P8U1=q++xf%qS4vGKoS$CE~ z0mh;5vbjtj%Nw>j(SLJfOu4=EY-bV=9oXEthtz{O%!;-k9br@KY5psk@j*2$5~ow{ z?x6m#r&fgadrb&}p!wvy25bm!X18=IBuWnzj`g#uI~BmyrIx_^Orc0T za*Cje1dI7guLX1InyUT6za4QL5h!I~hV< z9|bPEOCugg@%-74Q=$0H`6e#KsENiPrMRA0I#nt**yo|3%h5pvVDS8A%{eJ&@kvvO zfgke-dI?z?YH6@uAqP88I9*qBXGxD6m@HCug}Nga0hjLeuRG;w^fbZA*=V_r2m7c6 zwdZt9Wq`5)6LOz;Xr}&`0Z&ssR4YhQVBy}xMzj%GKkvUed&UvAfx_SBJ4iFLcT z!dyX>&1VI$GtrFw(3m<;Pt3X(ZVW7y4)8x#mNEhQgDxZ}6>2HIqZNgRLes>gwVz~BJyc4AfJZ5`4Ik9{E4m=4OLyUud=axSdx^QbauX_eyeaWw zpDy@p!{=u0{%%cXdc569JVwGP&6SlMln{O4_+KeTJ&_xhCb}EKwBtElT#UiUZ4zt` zj~baskJt4@T2SJ|JnGq28Lz)eO_zIt_wiaXo+b(k*yV}T_ zMK2NH!fFkd3+(qH?w0eLt*0LPOUay~)7v=)NImQQ74ir;lw+n)5Ne?R$IPh)y%aLA zCE?hJz^snDRW>7$k#wu32Vc#g2nMalCl8^;tXnp&p2Kkx9j+}2u;~Qd_%i*g#5_E< zLg6Ccfuzy-*OZIKgQA5-f{?E5cuwB#BoOETxF-sb4Pe%a;Dia&p=p$Nwzj;aY&1qT z8fw6pn7U2Ks|RhScO%_B2yCh0YgE7@u)kycZlo7phHOr2y6q^hc9gsmo17J8N>b}3 z5bDeym-#zjTUlD23XSBPc+lI2>Tmv#u^MVpk^-@IA}8QJXCJ#hS=mj0bl(O>qTsra zX4HpqH&MM~ZavLlbky>{um=A6+9SHVYF;Oe|56{lI3n^sBI)K|mFX;Wkak%kx9w<#nqBQf z5fAFLCky2O?ps4s+3i*%GaNlT~Pa8e^Z# z{D7?of^Qp6N=+V42}J*F2%z78U?ImPPjdGmTH>U43$X>y%xti_JSJ5&6v_mKS{{Ii ziUF9L&x<`QTx;)DCL%(ulA`%vR&6g_6S~D3KyYl!(5*0mgp(6tS$QFkx8aac)1N7j zSc%V7WIsAY49&MMopQ|QRF%;U9wUwz+5KcZby;?@*xj=&o9&}O2x9S>1DAkJ=&McDDgvRqdgmSt*<|4D{zMkT*ZKWxR0H6P-^AkXWoH?43 zC_Y4$=E6}P@7Vjv3{@d@4_mhLg`=U|t!2RH+BNk~KKlap!RK|SWlSkI-)rgbdtGX} zQR@acvVZVOs^<5uewDidd8exJaEDX7x4~2btPEY{4L^qiAFPa{n{q!JmL!JsfpxEt z5URfRn(S08NG(`;ahzg$miisHuiJySR3HePkx-*7Y)z+^bpSQa0v*bJKO-EPEJD$j zriM!LqrLYRIgJ4dUqhS3Jv^P`J32ZxyeFx7kbPIaIS_g_ZMNm%R@h@4nhyMJfcz==o=q^xJ693BShtj9D z+Sr7?V4pmV9#3m{`2~iomnEV3YD;G*)~Tn-xq596xGW#Yl>uLnee8kD5s8h8k4S?=sZuE@y%$}Pl`Bq21}DLntdGuMhYY>kfEhk`|fq|?s2 zedb~ZEHv}CmqLmA5Ma?&Q=pnqyL7fKx54r~=1?Tmnqv3ddL=pR~!S<#R5 z(w1G*w1UmZd1tH4j*S#VX+EoimGvR`5Rp#Lb~l+yOvfAb`<^<60KLedO<@2Jp|yLc zmq0z@UwR@twjsiC3=_KK?5qmkHwA2x2c6}#6hL$aSt9nn8E2m#ciHFf|24GSo?q)de=vYtm;g=pXyW)y z2=h))JSFA#xe#YcI?=hhji$_#q(kKD?PWXHkvZh|#A--xnVB?mO7dj(-|ZE$CllG; zPLDDvDSSi} z4NH4Cy9V270hgyRi#q&;_5;HV*cNC}2-6*tKZ7Q}6o2t+IhyX;L-Jm(<6F3-F#4X~ z&ju2Fn8~v^-k@Qd+xWAY_^MN6rV;%CpiyK)C)bz~zTY(Xu-I?F&GGkjo72|%5!#W- z?~2ta=BKxR6+p|W>W<8ofEiSrAV0_k#uQV??EYZMI^gyP#g;()XMmweIdTg(e zpRU1KAcbP{3?|oCjY0B4(d~S#5Xx|7wkdYSqE5Wf{wxza27qg2dt1IIsUc8buJPY+ zuqo`D`_9?fC;h7_+1ZABbjbAZy}8g_NMqxVxOFY zhjfy&;sZ`x#(0r@Ks;==k8#0EXJnesK#_}=>rvDT!@`jw`K=os0n)Z&mJvco?Bm(Q zhfd|uL04fv-`-I2HjmlAQ0z~l4rMR7%faOV^X6wmiHGjb-`275r3#`Uu|X!=b}Sd;?(Hgyx}a# zsEI!o{;v@d;RBcx%3O&pAlKUYVUctMY>~#$czmppo7^F@ z(^iH0V~e`w{R7BUhVHjI1PK4ml_~tcw>PWiv*qD!;8X@W$i{=Glo5$oGx7TOL^wJ5 z1sY}U^1dt9ZBR~2@|SR}DIx#OvkN+SbiD|8+9JvQ;B#NhDiu*aoS2oe4V6{C>*cHa zz3UV@9$Mq`mbH5a%3#{2{8aWIJ)=_%EOOKX;qEW_Y@a7SOi@9?vj3}3Iyh+X z7WV5p>GrSnvjZ;XC~buWtn7%yw85lD*3s-!xtjVEI4Uu--iZ@WN`hZ6$9 z%wuO90k|nl4<@K5TE<#xF@wt0Y~BNRAqy|KCAEfl2nnUgRC~{^$OWiKZ_S}+L$|F- zi%%7-JPn~=#40w8$m1^3k9)RR+asrfz3pw*f;ito%~BHFs5l7eQ6^mk&^LZ0SB|I? zdP}}LFX`Foxda|Y1@4f!v(5O*gRvJ_iAytHOdNZg)t0y4%#>>A)zVHAb0@R1FvU;& z&3S*!Y43@6==j`?c{OoW{||LucFbgu<6F>CcF*U{B>Agn`4_#sO7uiUbwf&tBgr_) zzl6`};6gt=0u+#S-b-{pixQrH+=-bE_@eNX?TeR}$Lj%{sOZGLP73TT&OLdj2^bs< zW`%V%R^K|Hmqb4fte549=t!%5op_+H^2B=hch#wa6Vg;1JV3F!1?!kJR>+z{|63bp zh%{WKLTJu6gQpzIgMeac&YMnKsZ+4S12%*Nd#EC#Z;ZrbdZr)_+$F9#im8btsClUY zbDX+R%SHDbH$1m)W>*P$h5O2$SO}XfETcBJy6egp6`1$~Jng4UJq8l&-+zEbPt+e7 zh=xE%V-ll>qa6SHD%!>N98)d|fP(rv{d7Pixdn;z*w2G>sGug+*yDf%4f4@pwv^Fl zCER%IA7P4b1+LHiVc>AP2Btg@heNPCPT8(kh_AFxTMgJ9agN&UK6iX+=WRQx?VteI zWeYUp0z}qMg-CPGnV)OElE^Gz9}d62J>R^~9lo@me{m!&!g@r5^RNNl_F7~9kRy(E zj50?P3JofK7xb&3KCwhVu@+z4LOk9?kPdql$gY@;yj1x=OwN2>jX_d4uRo(GV%LnP z{Gza+qAjMGK7le1IHY_EKGcz=Qevfgwy8+-JeigE`&(@;BsN^J0r71b7U&6g-03=0 zNxhZ{&ndgyDm&?Ro>;uNnEpeHk&I7J5>iv7CJbd;QOZT{^HMZh)D(H5+I6gddw`wFuk-2d zsBoWQgX{0HMgC9g9e|byg=ekA)2iRPxo{q5X)aI9-pj4zopm8F!F!?a^_{Gd{woXS zX!AoLp|noR03m=n!=PV09!k8!woIV;B+21gj<~^pu?5xFC4gu{v3tLm@EYBFu(8m> zjm@y;rOqfU-a%b18FPQv-696J{4(=*`?Y@B;V`zXXPwmH`6`0rvl-@;sHLg(3E}3q7+P!q=0>>x9{=I(Bysr@U!qPJwFx9 z!MG?cd$RwdQbK&7Wk$sk|d{b)<+ z;w_KKW5OA^%{ya}8XgUV7ZXE1WMQIhwcVr=#d|6=lzN`tINE zZU!3fXCJ1+!|$)H!(ZqYt>=j(odG1>A6ytliSlDT`L$tWIj_DRrt^bi0&k*&nXkiR z-hud+s+}Ni!PmCl>#_~ElzR?%HhVb2MD@Jx9XnkuK?NM5IwI*++Kx1|{aHlZ3TPVO z`fC42aPl%|qR`a#@q%#Saj{i2DsuDO{98{NHEY-{e<#!>|?)WKY*}!(% zhv5dtzXMX_G#z>+7Dx_7yB2iU_`_nPi6V|>RmVLCkvI&pSrrEF!=%~G(@1A;wjXkG zcz@Aye|6q-8<R}$N&TQXL8 z!~2QgbOz*moFej!=)bg73kNCty6E28J9>#|TP8_W{;H{MU%Xjr3S}3D2#^BeRDxzi zGs1gQUC9KEvvj<#!iF~HtRN(Z1NbMT_la3&Zd(eG3+ZHD&b&!30Vup$RS6s+d;nYVNtY*my`?&;2!&=3mkI=rB3AqrMLJ zV@;wBH>ndPt4Xu0*Mm82TXYrvfz;!Frv);4?vID=?>4LjJTeC3A)h0Jv6o_cP$lf) zZ4YbyloZJPE-{%;aCCTgf~}r3St->8!507CMd^mnWYd&WUwgL6i666cZA8kZMYkKa z$#Uy9LR&BL6PgYjBQzYA*#VWQA(Ns%Z*ekY<_~4SDgD^s6s$A$D}g6oXHD-MCwWYdvg`Vp3Onou7CKd?GW@f=5V3MGIYrK<^{#ByzWqHUWU#TS z>iz<4eScNXQnQ7s*n(6tb0+f<(o!QG&Det96GLbZD137p?X>AXRKJ*5M6(s&Zdl9r zyvWb=A5Gy0tH09)ex){D7P_E{tqjWi%|xfsbO}T?NUP zlg#$1m$G9;t?)Tvymt4m9J3)B%`lf|xmfXO51vX$n2_I%eqNMHf8fTAc;>P$p4<@#FBBuR`IRv=2kdUJL_ic(`1@DPk<@U5i+iZ}(VyXw!n5-MlSM0(xAa$W zy%AE&ruU|e-yH`JX19TK;8wyo6nnq1xD6YuW95Kq@aU@fFQf&%yRIk@S+csc?^0AT zLS_;xFW(U?-lAk=BG$mo@TBIUJEz9S7q^M6@3#Stec^s<%C@J_G$Zq5^EDQ%+I>rK zRG2Oq0*vgvmO`)0=TmSDg5gl2gb~w5;e-Cv`WX5SGLBva$tpYwHZ}+Z{A=UMSK9G* zX4LBE78)$^Ur@#KD6}wU^t2|DDvQgWmzUdI*?p?IfugXF8NIZMG=6NLoPG-gOYOrt zLRS)#XS)7T_Ik8XMVz{sJo}^qQR*79{WCwZ`$EkmW@X_m*x5ytQTG+$acTlI0ZXZC z**t=^x?z(Wl;ga4I@ex}X)xM!HJjy6whDU&7aAq#R6|;K=dYf_>?3f9tc&v}R?FjJ zXv1RjqX!;lb6x*XeAO0Y(IW7*WpGVsmqZj#9{*scPK{n|FXKNj5{j8mk=S-Li?!4% z*f!*+=o0H|uSOzdTd&?gEat8d8OC(I*G>CsOZzZNaXM@`uLcDrlG{Gl^Xt?_d^_rp zPz7Gh{kCFcPKIgQl@OoQ#89jaU*}uDEfEpNaVilcIYAk;lUW1>H{i7U<(0A2)MT)r zIw%{BIkhOtk_A&CY#X6dXH`vV$h*m3aB-xFZ$T0Tw%z%9LV|)ZOj3~|PB^3BqQJkN zS4j>%G5aG}`BN;5(=#lkYL02G)w~QxvqUJdx?b0(@o(m@5%&1@KSdA%=tci3JwxR* z%I`ZOcT)k3GwH~_u-n&oeC8cDB)H;&K!@pnA!ATpo?Q#>b z6=K;I?1%V=e zW+q-E)wRw`80 z0llylsO6@&Z)1%+-}q={oS7VX7JT;XRn% z;l8junHG@l2F54Tc&#}vZKS@>9z44r2hf1vsn=JxR6?HFvk8ux?!nf$B8e4W3@)SR zWGtD2$$avx6B>Ph!zDTGHDt3%^CrGYGzMud#a8n1UjrEc8Vz-h_DD8KuQ5W`f~C15 zFPRC$B!9rTdPZC_G`n_~dGX*x-B)ZaQSB>Hv{%{7Z;FZ+0Y;N6PsSUbWTzc|(6cGX znHYK&Vm3d~t&BmeeLyMHvV)!HMpKM2e|Vng)0xR|GY~zerpQ+EJS8T_i6pV9dl|Q) zmS%PC%fsr&)tj|)P_a8Fc)nwqf@n()?-GlNUJNZ_T%Ie-E?aG96vpAzc*agm2xzS4 zr~<(!wysG$&S{zp>o+SdamM00Z1wnJ!CU%!QAVN_27KCnW8oC4`CT8 zuYLOM3XI^Uu2SavysswrU&AKmIj-Gl4%==~8oFNAxVWH#+7Gd965EpVRARwg<^iR? zdD9r5VDI3&S<~t8YmBvryr3eh18=;-v*fQO+}op!9BQvPJHGVd!~{iSl!1$>5$Z|AHn{ViY`Ytne(h#94~51|_qZ5YO2sYj+(Qmx1$LkUJ6pkR0Vm%)Z%=Onj5?Jj^*5Kj=FenvtF1A=|CKjZ!yGOUNXf-_S{@VhF5+Dl z4v+8|{VJ6yswbwspQkS$TVn;;R@-#sm6q#_1dgx&GQK#LQ=l;PCBiwq9@8-&x05p? zUqI1I5)c#-W_(On#3C302BdU;NyZwWuWp)eaq)#uq(aOFujZBY-1uqIh327P6@RIB z%85R<43w6!1wT8MNji+{dMR)KoIg}XLrkD%0)HjmmD$6dAFBjXE|I8knV>qf_1v_7 z(2SyQ%Ra=i)aCm`LBdM)bVwH|fYPDc7J z$>-7tiDxo0tm6}%<#Gb(V1B0Hl6aX(vtRukl9X2T2}ay?e=%!+F$Fr&v|C6{$bLOq z)`+x}p)vAJ$q8zt#qQ~`>TXl0b`hG9s<;)aj#yLy9-%9-)v@d!2{nl~S&%}mTzsV? zB4?{F%#t#t?M-Cn%i8#J-j0{%@7TxY<-g@orW&_TkO!SY&2#nd zbwbkL*+qeTyP`5-s^_7(xGb#(xd4sArls2*;j=viy^V49o6M`Hz63T^hHgp1E-*wV zD)lElyH?QV=_h(q-8g2C&pQE5+T?she2gxjh#3_VVI7j$F_CP25&@h>!G!Dga|xP9 z9Wjv?&$yK^`QM?OWoKjB-!ih#XUrPIB5i)emPRx+l(FUV@XqWVl+6Ri7VK3c@i6f) z_^`EOR-5eQ#;PZYs0SvhCHYiUOF-OSm+e5Blbr3D$z$c2Y6DU1sP>v2pJL&*;C1S; z9}A7cJ=w(OABE`al2UEEw~?NbYZh~x8VrE?6ZVtJ`H)j5CSM`4CR2n zjrq6)$}=x8-vlnjOgEtOaJ|R!a9Id1c{(_fUL)7^%t0^T^eT8X7?T+j( z)IqG~-29jt0vusTV_jwbbs_D!!O9I$e4oyMlawy65FZD*hp>YL7Pzv}@hgvU(TL$(Wzw z|Mq!~VDBC1W#7%>UcznXy^D2>2e$d4j=anb28O|;D0&-0O~d-H^it~i4DQb+!pLt^ z!ps&R#w9GS-&cBNT=fTxsPyFo6J8dUR+tT5-}a;Js-{KHR%W;PqBvAU*=Z;{=Wr13 z^hvaOVyNcBP7CZNGYpTvNu$ppmwGiQDH>~ZDvWhnk=i%HlFUWzN0R`zc(s^DQNkPb z^&&X;PaeofbLH^(qAN6}c$*MHyqa9nf7Dwzw>SXj{&pZXm)ojy=q(pESsQZUJA!~K z>pD9Xm8)ZOX1D4n=4>wq(x09J0C1^OjSQiM+%Ew{@m|Cl`xgAtf z5Q(9!ydH%t<-@@Gpu*;X+BhZDkaD4kGu4Z+fw<$un9##LkHJ+3civ?;r{ffP3;O80 z0rWd7mneKcc2atw&<#aStY6ehYg~Fn8S24+)7>_isMG#|w(G1AY#a^2Ij%EAUaE@h z^|YpHI#eJLpU9?;biK%5ru4BT0rqb{gC_&WyCxIL11YtfLOQ)HMhk<-9%mXXw^^%K z!~-%QkW3Q>7^{6iuRa{g-(SOeXJiaK!9R;&%z0%F@#3(`ja!z7bBM5kUPnLX)|;b~ z5DX1n+gSS=SD%bnTvCuLc>QoWCqKY7DP!rGaU&}DeIQRsS1)=NI`qnL8PLE_`cVXb z3mjh0l3Tucw{TCbsdpiasI1&r6Vk6SR(ARtOR*j)RJ)M5NlVvpk^0SiC9ibP0tZTV z894`vFlkH$i8nVm8AE&$N3BPiQ>Ql4_<_DDsjIX0XUm}6C%FUhZxj}rX@YYK=I0Z| z1{;~lL%v+ovpX+6(97Sqi*>H@TK=Jsk_`BB!P0&AC&1w(FNtF@+a<)b#*HbrBsG=! zcy9;BoGz}~{xf^N!%YK#PA{&)>F58|BTgr>%7i*8>yrpte!aUTKhUni;@>`L3Mm{_ ztz|a_^z8a{0}#k%QjTa#m7gRLcGg6lRM$=1#1Wm@k3Pl{`w^I7iU6m+JDo|!rNYCXq>P-7bFsRx0JAiWF+;3@aAQ&s`r*=T}XzbPB%U%LNV15lb`po)BF{4b` zEtxj2Okl_8QG^T0ZpXqCo28|t&*M99XHA}0jd;olhB!?J=c=ghpplu|VfX0ssKET% znnI?Ox%krE>EHv|Z+?{-C4Bv=)C1xlp#YLw_h1<`{_YO?%0H*JwuT4YKStIk?g9Y` z^b9dmX|pMH=ukQ3uSoyV-u}S@H}lP^+_$#n1;rFQRD<81RLTszIPd-0P^pS&ORf(o zfj*>eAHpl1N(Q=74X?5+9N~$~Tg(Yhfb(_&W?BUB@E+(e zSR9V_SH7c%(Wak!%D{CB)wbcRscGteV=pG=L(=fZN1fK<*wzh2R5rmI5OWDNgIS4b zBkS7Qdu`ghYhVvcx?DV*SqM?9MrcsQ3d!L^2cWebePqc_8FT5CQ=rac2z zovb0|+4=c&^7og$XJ@lgo}>}X0(5D`sgIBNO%jlvv%5I-Hq}*W>!rc zL8*}2WUrb~N=ek4AAp@*A!G5e={&S}HKgb^Ae&6fdvEfl9>$!kS@gzP<&AneHWF_r zcO$sXID6I@EyEY7Y+N(9~h+4LAKdQ)Ms^qB`0;`AouEvYt?mrWoGP*XENzk}A*r~GZwpmL^>B>Zu*1Cp7^$gM?rkVZfZW=m@@pF?o=={rcdh`qUU=e9+_I) zvp43pZLe$k#VUT2v)cN?kJIdgbdQB~V2tm!G(mjEI8-=hrCcA9pDI72;a{}Q+mYL; z_E^9wO8utNh`=NqrafHIAopl;^b(tEZ=2{;ncWzA&s(mLH&y{d4_k-j2RvuWG~SsG z*oTOPV-9IWeKNL~PRE(J4xtL5$%hHmW*?Xs+(d-v%I4t%4+0mdeyfXu`xp{a9btf&Z=vS(=+`U`UW;JPqi=UA+rR-?T|s)tM?@cY)x<)>qoY z8Zs?uVqc5sdC$H=;tDpW-WX>ep(n`0wc^naq z>dni(Be+dsQxZbJV_P()U?!fBHgppE@lza0bLfuTSoMY6+3wnqre=l1K8ey>V#^jMo-t`-?0HJJoN~^&vYY3gU~9>EkKSw zNk8>(;qMGIEFoyPvE4K@j56sm%Kmv|qcOBqU!&CszUmVmDVtu{QtL5^xay_9RrN3i zf9EM%*)g5=_L2h6egXJ1Nr(80|I@p#lh(NWR$&5&d3VE>yvzyoSgWc&wQBbGOqsU*D^*!AN=A znI^uN>r8t~6=d5j3C=s;D3(R7+I?WWZF5$)9}7wEJF;28!hyE)B`86@`3>^Ym7y16 z6a%Fx7J@ATZH=ps#Bz?R<>75|uh4BZm`$J9*dJq&M8RQQ#p!t3+3<6|<_S7DadJ74 zB#sk`WU!$G7Nu?UC&6r!Tg_bGzGpvxi=Q`7*km4maQn0j8FtIYZ^k#GvzF)jWL~?H zNqPKJ{g$WU#(rUYIu&m|`*aXt);n*|cXrkwf48SI*!#9dq`BS&mqepaJQ}cJGWZWY zA%BOqlt(Th;@Udf?yve2Mo3rJ(iIlihlYg*sLiRSTp;CD!k$c$hzdT4?TGI}&HooE zN9Z;G!?=9(zm%{$RM`F1g-qF{xR1Cv%4kIL%$HI2oYu||S`{Ey2+pF4hz^*b18v+$ zQz);gT~9BDx|1Y%*cCYcYOh2{uf`MIT(yzgu;#~-uJ<0^{uNO+h$wy&x9YY-sKbzv zJ972Z2@@{QQ^ue*Suu*&-{($EQ=+$Ue$JF3%^kQmc+Ow# zED`5*z8BT?VY%3|YwrJ%NrBQ})1B$O-q%qc=5Ibdrngy$Oii4Cv9u?rzLwZQa-FvG z!T25TPs>(YCeZ|)=sy#Pn z=<(Z}>Y4m5`;mMF%YMG3gQ|$oS*L!He4<^z{W>O`Xf6|-mdM`t`WfQ>1nX@RANsn< z?#z~ca@g_U&Vm`x0!h=SbvvZ;Wl`mdLcn8yt+U7bCs4t&G#kSY8h)B9M8F7A6N z9o}Cw|6N?XFshN(=V8CHi)=nM{egpZk>SLuk^Z}vXaMc>EdqNo71sU z;=u*eDur01tE8^)D z`@t5ay&&S$Xd!`(3g{~=2V_QR`JH>dRIpE$@s6E&_Q?jt#Q2Vn?Q5^Xvd|og*xskC zUn@K_Ez0_A-x9zps-Shau?pmTJTEhpqE|u%!lc^@YCdo8T`!zj%el_sXQQEsX;e%x zc%ksl1hAJHZFo^43nC%?^x6iGb{-MLgohuPYVP9lTdp9Rh=qKRpE*^>Iaze*eA>Kl zVZmY$BQlK}wY`ug4t7gWY#AO2_!;0M29sNM<#>&J%s~utL90RevDIB-hRa(0+_ZZT zX{AL%*GWs~6RsbEkng3vUN)`-A)N9twA;*E3gL-ZL|Iy^Ej5{j8-kn~h;M~HTVz1e zyL1M_mp_n@NAoElODjUJ_sSmT9&R4)Si)Cdpb{|ZUXeT13lFqt2=l4Lah5IB)Q@V}~fbWz5jW1h8#?O!C-f{=tdK z@e!3wTEQV0jP{=M*<;7aD(I%*9QcW4!^gg-2WP4_SLxep%={T1`swXy?3u*MqNYx4 zFKe-IwyEhce?C@+8VY!#Hm`w|yI_(+#A@RUFVoTTaSq9AvmP6UwhmrgoH~tg)}Fmj zls{=bu#g=+@U;w%;n66(G7|(H&`C%9uXse zxE>69Q_-~VIxU*gYStD9{Z9I4&LdAjAq$L!w4=iil(nchaq^I(Y6vfH2qAiGuLFic zVX~RJ2($?eXLq2a^XaTr{Vx%i_?UFB1KF;$hJ(u!MM z1LNPmq4PAdZP6U8U_AiGT1B<9Hf^(Q3=S#O&W2He#MJKh>!5c6C)C~%>Fb%%NC%H_lz)0@^=kklLH$~brZAp6nV%#N~-zqdA5X0 z-^Kyf2gF{m>QjNzSG2X_I`$tvJZrgl_Ex`$ASb2i5j|l0%3WGSFr&zQZ_4w4a&Cz#JYUgZmapaQnqp8l=lozNjF+am$g0Qx>_5*lF&qcr>L_N>zW8 zU1QBW_pA)&a)%$<87D?xa!Gr<&E>6GF^#T>7fc7gR#dWhx-1hBN)8q-lni=(? zU>qevlNH6!FIiJd&hD(`1|JiCQaudA5lV&(C08p;TQ zPT%rWj{NhOb>)4&j@yMpUVH=zbR^K_BK!kRc>}vL0IPh=n8S409pX3F7*Mt)m z8hiboEuN6K-{EfZ=70^A9U!E`D<0%r>?TCWL>e=~XwiVTn*f;I!fYhl-VQ{ z&nuu6X4z!M91O%mKC*?7d}Lw^U73$L`-~~f**0ck7oPUBo0m< zY^}JSf!OheefE(K{oa?^E0jL%7$c{B=blq3GA#~IsZHgptK5CTV79wRkHWIL)wA6PT1U&yPi#w+3)*%B+1)7*W zOQVQIG?$&X2_T1FVTrM`0u;7W8iQ_4lCf;{wki@;@ia}_2}4Y~iUt`@`3>Zs3;vSn z>FWBKa%@iMo*;K-9Pi^i47M?)j$4*kwqA8h3Mz`49Q3Z4wPCRtRo#7#@D#%72(txT z9{L-|JPwKSX_8>E1S6NGY+y2*oG!O~w}?-oF(fG!$TWcm>H-5Lh2&rluluW=e4oV9 z)I4IE4rw;kZTW|@=>X%J+S+D~XQ0Qv3-j4UlqC3qT{YZXL^F$`EXh;b2+m%qfrR{N z5mg2hlQln1pJ7EUYJRwWkz*2`koGdVZ+7;irXm%iVWi7te_lC!ZU<;qNU-10u3k#2 zQu9K{>6)Mt+FFj8;^Tj{aIJpm(B^0GvhToUO6RwuIpm&acTR;s&E$9sxX+;w+VpFIHl9xLhWu)yRZWIWFBa`#C2m%VA%W=~SJ6V4=tsgju ztMrYBy)o-^^2W(L+xUp5)H2_QQ({OhRjN?vrb&W-KTyLuh!^BDZ88eUZZkw%x8;w^ z7Q7y5``V-MH?Bg_cqttjNC|%!%$KyBgnV1l*FTHBVUb6`c>67@%Q$t#tzLA<}xi>i~6*D%t1`qh)9OiHK)|Ot~oD6Vj9F!wg5#6e15(B;(Au@cJkr4 zW7uY?K&{ioW?NI?9I04C#OiGgDPC!dL{^(m2Vdps6%KFx^OOQRPHgz%zRLndbrUJL z2_#ZkHW!p@e$_EIsS6;1FZ+r*Q3(qmDwKZmhSCRM+J+V!9@o;xk~N9xcSy?0Tl!oO z@lkq4zj9WXiGjV+07Yp38V$({d^xFvJ^zA6iD%>+@Q@E_4If7Rz!0+8iup68z3npc z1@{!*I3Qdm z@?&XKRiD>)(f>|9_?`DS&qAMAf{1cgv8E1eRhf{T3$D^TuD*!N+0m*)Y3pCP&ijOA zwH6zuRTsy#AXS}N$I!N5pB8)r~Bz$(9TLs%S*`^T5rc1CJFf zXx!>A*!x{!Hv$^r3h8DrM7@BGIVOV^&{QYsp0BA8{%0_ISO*sJj4oX%b4;d$Mr&tb za$<{xY9?8pL_4+yA0wKi47Vib~s|In&V!B-s;ebqbF?`S+e5?~wuYz3I= zXyp=pD9nTQEzWLIIW31FEDFsusfXszfOU`Agoxg_k!_ASGPKL&)UXS>MDabQXOL}C zF!o0=dTs&ReWX6U>Cye}FniaJ8TF0Q@D_}MN#b!Lw z-v0{DA>^~>Deet!>FmsYU#W4|K1{$j$4FNly)tvzrHmg0zvs^Z2(ux&l4L= zg3v!uh)=;>$%VM~^w4u!_^O6q?>W8XL@bda_ofJ(qEgQ7r_5Qk1fRstbi%Y_qBm8# z?SVufh+d&K%G94>Rf(*=D&gfz=UNkoR-WQGS}Ka{pDRg_M{UOE^u|b*?g6?Bx@(L; zW8FA7%lh!O5>Z_m=*y@p-IVK~5u{3lj{3fC@i%<=b?}z2Y9KwgCYDl<618!wYk8no z0?FfOA%h=h2$QGeR&dfziLCtj&aiE}g>lckNS=?f2d{?H`k~4BXitl|w~n?8Z>)Y< z-Hyp0wb@E5cbz-gvIr(P*Sfe$<1VW<+ij>N_#(HHDX%+ZYxz|ky|FP53%$Q>lbM*F zo%NY5v3iWdzBkuXS5f6-&PE>F`#E?|fGfS@^rZ92;DEy| zzm_i$yK)iH@>SHi%VbnGKk#UXo^@2*#l>)|8ehW-OGCj~o7!dLF{67{l6mJU_c)2{ z?z&aN4XJoIkY;|Jx30m`@)1MqM#<&-&KFq;DGkUoN2lpbebflCsaMftZTo;$okZt7 z?TD)Sdl(^1x~>nGAaN`v$@#{{aF(l5h0;}_vHT2yk!X?vf7QT`K(Jw7$x4@fJ0ncP ztV4>7Ek0xBJ+##8k?cp8QlC_1HvCV zS9`Z4e*Ekc)`3a3ep}k^^xA%W)f4u!`O{kc^R{H)sWQhWy>?m=V@J3Q#1YRc%?{aI zd|A51Y0V4+IZJtmISZ`RqCJW<$J)D_F2HM12p$<_URgZ1u9eB8oX5}FV zu|N6g1_o$DZv*%vt^yBSf?Sm7E<=y$K^XFZD26%n<$%knyKzR>@}r9aW_kXVLtoYO z`feiP7?VcR^aE&RRf4}H8;%A6>n{O(OzV5*?+1KyT!xa`>wHidfhazRGn0?SoZaRR zKE@GhD6z7lbrklfM1}m9-ffIV!W6TC13bKJ9>rfcbcNO`{X4cd-B#*`lJ{R}9=g9s zd@J}*Wg!rhyKqh)AYHn5-fLqlGS}08bU``v?74YU6aPWkG>;f%7x;raU<@5B!|YKY zXnj~GEFvD=9XLOVTa?n-!b`lW_0(W!3vx4{9tShJG~e5^%r6ige=0fPYu7TTJJ1C$ z>2aM4Dl8?Yx0&D8hGWbPkLIoFXfTGTnV^h!Tcadq{2&JCun`e^5s{0QlQ~KG4^W-{ zW?e(`iD#^c@sjqufGM+QHulQ}pe=iErFHMcqlG7l?LLqWtzC1GN8W^W{Ckc_4&=5R zppIL5w5_TLjTrG~-Z-z07^|f;Sg=W~SS$tK&}%Iy9mpkOv9lt#@@|&cKr0jD^8?eO zvhYsVBKQ|6{Z5G!$L_y%uQM)R=>APl>{DR?v393y5!e)D+aRV{LxuBO7_Rma2M-skx{6N2CnWisRR!&Ng@ z_PtNplj`5fvHePl0_9m!3}F=(iNXgkuRXj7XIuUePoLR902fl%<|2r}Fbk}r4h^6P zdS!|MwR+EH^%GtQA|peQM10a%tO;e?5^(VD;q0;apX1G7Z%z$l=|!l;I7^;>19E|Y zS9C(}<;bXyc_mNroe&g>fdr@S3gTKug1yN&IR=ahSN{Xwt2N?Hb=$8WnI1vKb1gji zY?%CJvql_0v>k}3ESU~Ji&`+U|96+>m9*CR_Wj_X`*_AEC0 zVouEG#?ESb4@J*Lu|?c6suvMvhIT0Mmzyh@OL}QN{jbvEDT}RGwu-33I{sQF)C<9p zN)*@%01yhN($W0goAukT@92h|_#LByF4wEv<)bJ==|w0{;&8a~-o|#g znFbTn@h92d?^k+Nz0=aG|7MNFSc3d-ew(y z3Pud57a6Mfu{1mncc3CMU&Mf9K1{GXDB_h3GDk;^0-{ZFYuDeH-kN$9>|0SW%DQCw zs=B(@ssSnc+wY!Ac2QJYKkGYBLhS(*@ zjtq#cWJO2Ye4-VuAg*f1XKZaq+$10>GB+CeJ4(fscBT39CcmM&Vg+d~yE%$OocOQ{ zF~K!%<=eqnipxT!U+2yf5dE#KY99Ms)U;x1Dsn{>WQcqfRs4tJI19x>eR=i;qo$wX z%(pXE%LcPE{@u|2Vh6$tL|s&=owNL_td`sVb)L!+3f}u(8U}Lt@*{le2*!Ic(p=0F zP8*-vO}qYBKGC_8d;2Zzw$e=m>GI8>42W-ut#k#qI+uOA6E?Y~q|^3346{U}*BcgW zlp@;_>Q=XPcHc>X!4a}&p*cHoUsF?QsK|Lo-YwXuXd7ei%BXCZe-C&NN}=Swg;?gUAJ3jDZXhLFv{NQU;CZwF+6I= zL_gVKsJkqf9?VMvE5^eP-I0Z7oEkJfW6jT>F27(d8V!_boc3dU0IWk>JSFPzO44LX5=KmA zN}~zTBCjxMWEQ6S@HrGM@tzWg?{xWPvwvm455tC6F1K5wAN9A|EY>T1Sq-9&&aP-` z87#TjK&%@b3_Hg;Ez{ygYDsAr6eYl9?opiOftMQrLmdYBkY}a{*&n&mWdt@Py4;SW z^#`6wx}{NF&=W^vhh1AX+}6wyN7c*O_aKz$++(qU_p2b>31YR1`h6nX61$8@`KcB|9um|fcg%0IsA>Axzj}Gy2 z1f%hK+7&N<0(PPR#k3^lYNONM(iQ|I+TWX6#6(7n1=v+yDh~^*lD{D`gNaNVg$pTR zdxD^kRWcXp-U;O~yTv?PtjIKd(ZbtS1Y^kgnK1>}t7*KrVmS-bLJcaKxh(#JZ{s+{&d7%U3 z-yl2ROb|RgoIgD#ML-Kz+a9_NL>nK75MJsty+DWhN-MwW=xa;%Nra}(H+;op>lbtv zn(M3`A#h4+TBqu}zXuDJ_qRK8V(!i)yfmx}FX?-bl<`EMjXcjP8*@6do+5h(s(4EJ zBH)wZ;@a6^uVZjt?c=hyoFlqG1p*-&BOF`Y`vG=S9E$M`plBVO_>s_^iqEQ zhT^lKzyt0jo>sNsN`Io!Wu=cf0^j%kvtd5*y$MJ$=dIJab?vV){9zIDU$doxq z2AarnqQFm*sa}WwzQ{nIwZv>ra-b!PvSd)KC1||0M%3knORnzyEW-={Pm7zo%p45+ zT(GHY3Yfh2x3(#P`_3Buv0pCBEJo6d8W{n%aA4s^M*bA(%B*!2P6@0*d!9T&QP9%b z$<8jCPFvdSoWmZRUw#Ruii80NLGWYNn~$;=%u6?d4+#$#QuPG+yMx=&W;63-m_Aam zq?-fF9*9Lu%`wDjJR%%%Zg%mT^ee6sZH%z}h;FsP#eHL7q`!{*t=)9Nr}WR1aopC^ zgZLm2PFv-0cDfe`FJtq_bGhv{y0yh;$Zm?jT{d#bJe7e^Zkw_lI0WNTji$6$DW(Or zzI$J_*eni3x?Fcc0N?8RK(*DnFM};;i`O|mI@ojJ8#dI@0B5h1sOuuEaG;QbBY(#C z*17TkfZu91-(Js0nSjEURTH4%Z+zlNmm_-7!%4MK>B%o0crzG>IBvvD#2!R*7?+rxzhpu;5zH^4%4j$h(7R7X78U(hU z)___!>A&J`78ht7M#22V?HQHhgac19p(A5G{IX00pvM=(&6L5AKkkH9qW^62X#_LZ zd)ZhAS_(i%_KMVd7mevU@xR8JqA z8{jgn5!I}%m*FHyleIe}rA)aU2@oqJMwsp2kP)%7MmH}bk~{@N=C;g)6m)5b{%&w~ zbhZn4_2|dZ$$zW+E1C931;|e?NG(fC>oHQA* z`eLl-12LqkxN+#?9eyxJOPcnwOyKEXOgr5<(I%jKmoPWgqHOZZx@Es%72J78;m262 zPc)Q;8Wf(VvEfar3lX?L+8mGNTOKY?Hey$APiFHU)+aD*KiY62Go4|J4G&jgh=Y@L z*|=9_#(UvQaay=0H+@S|?sKWru;Ed4&77PR(bPe_gEm+B(`8d2(BTQ-4NBw{zqVtRy&^2kKLb914-A;A^GX?9>V9}to3*q!3#F?rrc*9&lF5_6+`jc z8y}XNp0nl~a~o4zVReZhnX&0vKPl+XRW{={E_jFVPzv7c$Qv!KPjX!cozFhQdvi<+ zlTb?yg}r!))c7p$R4}#=-+YUls$Q9vZ$KR*{b9VL24Eh)OpWa%iO8=Ej?qc%xGw~n zS93sr!m-F&Xq5c_U7i%)pN1gC1q;tZRyABjBiVtaX$mCKLk{lGe=Xr|MPlW&=JHLJ zQX53adEg_Bxhr{G#Xof2-fm3Y4feu(%p=@?sTU;J&#U~I;Mi#)XvD{Hmw4YwwWG^y z-)L2`dG!g!uY#+GQ_Zkc{}-wJn@RGf0<1*D9Wq-Mg_9Vk?~*?d>9f4;(!t%24M%_M zV2ndLtJHqwemYjNHSyu2D)texRkXd^dXHfoRtNfZKb1zSjs?k2Y@+{-jOK{$hyVm` zPXcuJ+A$wCYAi_AiM5kL&*8_N+T37ql=fVS*6 z;fmcl_w}`|s>sL5X0cSucK>WefCKVl<2pV_ST;T9{bO>e&6#j5eCqL&)}KdONeehQ zgbM3giaoTEEwqui4aKIBlfTb)=0+CT6w+h5`BinV15vD|RR&?TafUQ7vi$~n zuDa0N9vGw)i=-?}!DmU$z!H<(ZoTZKdy$XdaI|E4-6_cNXokAl9J5CoD6tejvD?#y z+8o#&@@b$I5SsKrZeq$`gcrwmBVD)Xlas0GpHFUYulDxJe=P%+aXB1n+td|abLCKZ zLjb8?%lj!Diaiou?78K!(ol6f&4%8LZA^Y{U!j4-Qobh*C)nmJq5g!Orq>RkCP_-g z(pG$rjnm2)oF;6l5$^j35D{G4^(DMUgT?$(EH!%aq61!QQ0V~-w=})~xEVe=-#k`V z+}`m7;uV3l2+nQ-Xc6maa(pPSK$U8>(oh-J)xDYZ(Vsn9xd<6&5GDEF8j+JLP%1pJ z#v;Z|YQzq2=+SZOpBHuiAume#O*Zg!_vZM4Xm@smF6?2=P5%CLV0Az_g4PfwMO|+_bx5RX|U$&&$9ZT zu&~+jcY_u1@gU0Vshjk3k;?eD)LDnoo!yo7PmI&gRodXeCPFj2pbvZ8`IQ;!*|#f-L7%)QwHvq%AlTo$de zijKEDl2&wU3s$uDB~IMLlRnGJSgCMpV%Ga!`QV4YYf;nF5@jFC^3N8v(w4ToIc2HS ze?*|=w{ci;Kd6)tN&WfLz4T6b(g8cN5;U=(rEjRFc41t0e!07luv2n+lxy$VYHJ_P zKy+kD1a4POeVg8kkK>J%sdcnl%l+uPpyC3_B%OB)$ZCY}a3`i1d#EIlUl>-L`Kwl- zS3DZCP)1@T=CD0IUot+Z@0ynT(-uu`$+8#Qu?;i|Mn*3{iNlEHo0`=o;SJrzvurcV zdn?-Po*_UVGWDzdgMpquE%h%-B~WYE2Q$>yKQ-R-*;Cjp0QVEYRn-s6{g3hYTu1weaogEh=r$#!#%%(UCn)h_<1n=m#=-n+kA@UYel&NA&K$z zeHChaNw01gpNN`cZcX^*I$w<=$ZB2XvEGNGOgs&8;#0w%hbI>T`+H;89jE-Y@2Dgk z41?Yf#z2@~MTVimaCJa^-u|qVhN34S z_HOn2Z$X5|r)^q~docc_*s%MZuqsqrIVxl0duBS=dd{cRO8;wGsum5-t4L7KSAM-g zm0T(ik>XeDVj7(q?eYA~l`4s`sb9T=DS7%g>u2S-K1_&w3AdH@OwGEA7QDaf^J~-^ zSefM0(i#mNr;oW6j(|&ARCh{VZ({>dzEm89n-)vPn!-_3pa{pSkeZkKOsh{AlvRp~ zng2OL(=q?+wZj^sS7Kt<&=7!FiSh-1CSe?N)UH#YwtUgHFf$W}Us+M-gmkK^_ze~> zp`i=q&ufaB5rdn-V<2ajAMVt@^a`GD`clEZQ5D0dtKwOG-1i>!vwMHQdBi3hoX~h@F397yR_wr?+BXMu9O8q=Do>Y4%yo(F$A`+q=w1IFa<_io&}uIwA+ zD&OB?)I*jJw>ubye9Q*Nd=BPeM>-jaw!UzMT$8Rslo;SgC9O70=N ztXR|CCxqTk#$*%z(DC@C+SNasK|~aut1;Xvs!hs@bsuzNLA2Ypjb}?li`7EcP27w9 z4J#f?P_9Zb_|@4C{%fx{a_@f<6xWuuW8o~g3U=#D?kFP2o$0Vb8=>2wqp%d}HqGJv z=V2gRlYj7|S1HdR@qlz~Cl!Byfb_nzSCeP9HDe`-f@V2QEY|&fo@0a?GT=p%tmQpt zSEKt;&yL}rCvGwaw)tHBX1j%3TW7{Zo(>fN0AtV~r+VTZ3(ug9 zJmbb~7PT^&f19NKB>yfGS<6jJH`~(_S?rw2@4?_&&p$;!isN_$EY~1lwm6?{v=ZFG z+XWtNh$;wez*a-h6Az~;#rr%6;myw<*>oogyDoR+@7FdjZN1J*j~`W9{d7n>0+IHG zD`?O-e#I`v&-;|U49}c1m(n7pO*txD3c8$p@9%S6`Xfge(pk}acQc5xw5Xa`$xI(A zcLhD|DK`1!#nq~?Zq|>}2ikRv)>E~DMAH@H{L)`akkKN~M12FofD^tw%N+R6pKQV4 z*>CG~;|1&)Js3WT;;7)fe165*($^{!IM-2;2s)Bub5sQ+3b$cW!2B`$ot?g(9*8P; zO!TI`=yUZyD;X-`Mwi1tVq$h-JS;e=DRyyMUk~+#sZmS+D~oc{+^S(D^1uiKm22)DQ%@(?_1$7fQaa0ZbVbg=D zH!$@(2b_T@Z!CS=4yZn<@EkxIK^y-@bMIzm=V|4&{|}wBUF#~$1jsyWElhvEx!^Mz z{`Iy(O0xt7U+<^y4mePWUDv}C7N7=>I{ncT3Fj$h&dVeGBi- zl4wPRRLL3B#H(V0L}bNVTBZG)%yFGdTC%#o`1QwqbT*(g-HM7K{$~~i#LYy@wRjq- zd_o^z01bZyq{IJ9PGC$L$5XJSYSLn3dZ9&&F&s9s)JEo)@2G`1v&Iu+=) zzn!ugBEk2e)-r&5*)gb?H=ysu#eE)aTgXl%bnMjrksXh@NXyG-C-VB&T z%l9(1JX2uHI%46w8pW}!|GVl%?cuPSo7IQ1hr_VDt^S5soi8)m(>qGf(*-R$gYT}z zG&KIQ(XYPV-z>r_+3aqOEQl%PrWX|yrNYWpqy|dH;Sjl}!^&KYK3g8T3x=#p_$-T8 zLz3XG%KQ_K<0M$0C~?L#xkEJbN`<<$j%gE3gTKC@mE-or7J9DW*Y{L~Xhy$egwLqs zjMXq?IKQ6oBVJ@zDNAqUcJ=s$1{0w0`A^+c&2fEFkg}D4z~dfu+>^&ph2Nt}i8z|C z`=epEFz~*9P_fzgYfHb7_eoN!09prDdMrAtXLULO9eZ9L zMnCqqS58hStE&ywY9|n3Hz$J+S78tG_n7`%l2Vd^r<)oLQYKx&VlH}O+WR*(C=h4w z{a%N3K*46?Lg-fw)ODklDB6Qi0m%9&9?$XKeFeu(x~N|MgSp}0pqk{)Sm}e=gpR}y zFIHXqqMAzj?>p%e87_yvm7kG49m5&%?cMYwW|@e@BJMX=N?aR%=TEt<_QGSe1L4x- zWC%8u@ymD3Z0f1aYyyBlX*9q?xDj74Ezx`?72acNOx4-&uDu0vAfQNJghkymH!`O$ zlvGtw;rK*D@`GP&U0t1f{-0VZm=+bcM);Ka#!wb_n{Mvww|MVMiN!Fj9O*iAIm|_A zL>9jjQWRop^X;=rI>9dm1)W!@)d%0e=4VmT79EmzsMXLGKkr+?hrf4su_^ECtQ%@3 zrq-vXD&nV>6Fx@*P}&#&W>IoCzdCL%SY%w1lPi3;8d<)`-C*QT4-BG*DP6ZvjjQ}1 z%*#MsSf!0xkoOWfH4KG7`TXj~`9A#wW(a{KQqJ!__aI?3BeRN^0%743;+JB! zU#_$jz{?U|gGsV}NjeMGk0>AFMxWre1q+QZ=2yxrYY%gm$aw{gl!opJeSpdHjRVnGV zJtvY^dYE<4_Qq$0eh(VeVnXtZ0Eo2jL`8t)piu{^syaFv5@1jM_P4*LoU)e~Ln`Iv zo7113J{uBokG}op_}+t*b9Z(sW2o!lAzh!=XQ#tDyM~q3%^$vf_~5~#*fgp5(A4Me zKmYQtfBws#|MtV1%{ohU(;gq6td57@^zh`tLkscE>H6ujr<-mgh@3Z=e8*f8z{77f z0NS)CC>@-wnUORKf^?N<_V&bV1)${8b$zgE+h4fJDP?#6oj&3Z8={PaV0C5koO7D0 za?d8Kb~hJF2%%}45aVY2oR&WQOIT{q#Tds_+a$oIz^CaS*p9mw8^%!d+x;Fhh)CxaE}k)VaGPfRF@%#Ba#L!h*i04oLd^7Z^E$E7079 zcj(=fU|3JwC`Y-RfR03i3%g*;{4- zi31=BgO6)!0JnpPIfBdG1+uQ{uIqCtaQ#lQ_TcXdZe~r>n1!zEOU{sh%afHf5)2kd zmMxA?Eo&FtOG>G(su033An@hNT5#5~kszThUd7n7jRjk8w&jqEJ7Bu%Ai>=uM?OBp zrFWl%{tk$rteY+=iU9OEKcW|i9iy3yavh1?cn2+G%iceVuOw&j!f(_$>#rrohKcoA5Be0SX%Q>bKv1 z`%nM;&yOEIID7s2)vFgL_fA3xpwYJLO`m>x_AJD3|Ngy)j~^c0TUiV*pFREgm#1fE zo2G6~PEHFdEC6Vi@s&nzpCgoCeO@1RuvkAVPE!Mn2%EL5=qrVQKVqSL~{!d<&+zJyYIez^5pUHlW$KRKkEOFerRLcrxIf=8Ze5_KL7mu#h-up@!_LKfBf#7_V`|` z>S}eeUT>w+4a;NH)tP2H==txHKcUcX1zj@OZo2$RmVM+&7yI$+lg^ z7)!}H`@H7ZZvE*awAXg}5r`^Q+xHvn`>ou5$L}Fv#$82kFUcLBN_H&}01cB(&Ao&W z%&ers5Fvf3N}c=3vn3kawr-ocl-&25oQt5@to%3@G!K<8d_IP>pQbiI1V{>ibZ7@~ z03lf0wgU3$TExQLhQJiC&1ENR7Zw&4K2d|&TT@L+E;kw^R{{8z43mU88gB=Xpy1e{ znYJKE5UgU{%Gw7H07tA+O3$7>8-sFw@q9fuc~&v z>Hhq$fBv^W|M~gLSHt*h(`_qX0993oSb-ExB+Ue;tZtgej~@>T*+P)3@B63Ew&a}6 zBA|DwP3~6*8%VQ<2=ZCLn*!u613=?|k#aZ6b!!4V@om{cLdZFHT^D0@a&qimq`njk z>}Bm|>`o3x-PkcG-2TrAV3lU8oO@OG*t-E z%z%<}jv;)xFAI{UV#rNXAGI~0P2c5|1VOm=@VCg_g|bWIa_av3#0|Jx}lk11(YZd1SJuqJ$Ufo`#=3@FfhZ6@PWaK-1ohev|gVMMYo3ciR7@y<)6RYndGt<+P00c z>bic@_3i?2b;W9e=T)XENrMK?r2zI@sPUzux+Hgxqp~F#NPL-aZ^;}m)>YNi6@;$e zq`s3oV1Ohn^>miPvtMYvH>2$!{oUCg`;WsAf(7%UWpdtOc3D#Hp3@|2@1m|>SXfvf zXM%RVzJvsTH?H9&2@Vg&SVs`)?6e9o_37F37eD>{v`aaJK+r)*1PLa`>$A-emHqkI z3l&d!cML89jDgW;Mw*NQ2ehvQGeDNnPutYZP&&Da6(rD7N~fo%r4$<<3jrXE5+xA; zuj`FW=paz7$%#DQnDbJrphE_`D7*#UHa9Y713@UoH=8bma6H(?kB;26>3RsCnGizT ztlFk6UQW->N-opxzWf#m1c5kaNQPK()5Q=03Qh=sgpXl?Gg&KP_kTs&fP^m{<5t$L z+Ep;?y3KklZAV6cix&Ij5Rk~G`12D-z zn53La&b{0vfa3qNp9BlY1))lt{`ITZUDrs;+1R)U=Ig)_cmm7oSy}`t>YKvtGFrcL!-k z68ils&1gO#pc{o@v|G5FJXIBA^pd+SrR;!#2JknF>+mo_!u@j10Z8MJfWCNH3-X5l zkPC$LE9W0EAkGR#fut~^w>G=@I*!Z?prqe0EUIh%k-0b-J{Vp8K&2q?nos69%31;` zrCeO%fG!<(rj@X;aDm=2gBKPgeNIn|bY(FV?s$i|{ned^jog1vXyDw9{NO%Hk5Cu) zUxtdu>y23ur~%qOL?i+bjG+o%N^zHj5z(ARa6s7i*O6n=zeEVu)b)@tTE)tDC*xV$ z{*>#DTp<`?E?vkp;t(FRGCEMR5CmPfuLzLf`BrjF5EKdF1L4i55P2i-aD+tiS`kxp zc6zp2t&ZB(NXm0AIXeI|6F>=DgpU=soGTL&wvE=dL%3MNW_YhQodoI14_JXBAd>b^V0f? zIRCT+3vE-kO(Wk{g-xjt5i|%O-ziEVkiWXa=mnfF1Sg-gX3rB}dJ$$ppa9OKJP8WI z!n-Tu%^)ZOkYR2|;g#)`2|-SY^CK(-fO<>T3X;Y(%_QyGRtx;RGEVqJ5R^pnbDqQT zQ|7JlFnk{_>-FGKQDeE>qxYy$}-Ac!IeXy8W2V$u%IGj-)FiD7JJd;dP%5!846 zpa1;lp`??AAixDo50=2`04qpl5FiJ@foY}rix5ce59leSQZmU`uU`KT|G)pi%>MYt z?|=U3=fC{5Uvfs_Upar^6#WM(0RoHKB14&1yp7IazM z@;uaE*G(;Hy;*l%FM$l83Hsn`dmqto{jNtbrHejr1c7O=@DYQe!+tStppKh+9B)peih@{>8Wx3?D?}dr>8DC zAeRKt07&BeDS|G5Mw@c|<`E8XQrQPO-OP?w?I5JvwlT9I0BPj&XU|{0cwuH$Wss(D zpl+mJq{{(Y@^VPrZ~#AB9qgO`6UAV3;T^a=&@`2ieM&jG0Le)M?0R4E z4G~62gb-Kls%=*!*Jo#&t}D5`Zz9sjrm2TppSpp*?}1&B=gn5K)NNhGs)}Q`-aPiB zFB06fWKsS)G?H^3BKmb*4}$|E{w?J|+)5&4X5+-JgK^V!sZWy5HLYBb@M&1kVMLuG zep8m-eIp^*uz(UEkkDNDh#(HS1CVrAN?8!@C<`%$SVagaCAmJ~Dfq&|!h)pF|K|i9 z25$ih0?ChQzfbZ|>Fuuwixdb54mEZ4?Kh84PL5s-@{;!E&FPDmFH=qcwBJ+!s7XPJ zf+UCp43Hp+14c6rK*Xg>)}mQBJ~}}<(wWqeDLbk%a^ZSk{A$N0>FPoZU45@RAK0AA+VC#Hy<9-MeSB zl1rcZ^?F@Oc~2mVq#hvKJbyd}M zyeO^+@Y1H_?tVH*n#*%3((*5GryvPQQ*}lNxeT|AEu7CJUR6E*Iu;fd79`#M(_#VC zMUn<#BmjpKT5V&t_aJ}u@8@sMPpspZQG>r zQb#F8_>x#yAP7Jd07($cj1Zuzs@k>=!Hjg@&hQvQh{4E1aZ62OAQZ=D+hVQD#eMM} z-jQRv&+@PtY#>O=DZjU!FK6BRzf=|$78dpywl_g=H~@o*0zhyoVQ`LtMgs0GxYscp zwXHl4i$jB~efjbw1~gCJB&iW-cIFE}q>-FN*Ys6@L(@Y6Ba6#jLtEBC)_(WhcgM#^ z!;*7({`~ptH?ONLjXTU#9C&Yz>+~$(lItd2>@U=!M1E``21z&G;6@U9o6ahs@00tx zGrS4NrPP<#I&JE@X;-Uuwe|Oo$*W`hb}IcW0WP6vx@0he;DAQr#8%Q)70pu0gQnev zQ*)7S=&Fi?AvT1VQrdKV&MqfFV0tbDwZ%R6DaT;jKUT%M-s3almw}&;tXJ0yUoCQP$jQkzZz0)b4qpH`sM0;2UVPQ-;Jvga^A1N4hmt6 zHpv*nqel;iQp~$R;4ogVhj{HP38cBpT@r4H1YNcH%~bB88=0R*(@Oe?T$>mMaZ9r6 z`cj-1Jx+L~W)Rf(*$*y#bvfsvgKN{5-n`d{Jj%bg1eWJy|B2AlW2M)4mKpNg%H52 zvkf&X5K0IX$-1uEww176pY>^ywM&rdwzG7D(LoUl36M&`?$Amb%`a6P)l0Wor_|@% zOGPqwWLK4PRmJVbFb+VIF{gChW~{WA0RxCHgoSysG_O1#5daaa$E&)MQYr2N=C#W2 zG7Ad}OTx~LLKg(Vd5l+(G!rsYuMqBgqmgaf4slcfLr!JgtOjNK`pp}cu)l)5m&$y* z%>hLNaSkbBg!@e<=Q6j%y51_=A;35~RXejVr1d_3{vxF;!M8}DJ|Ov!faZesl8f_; zakj^<>#MpRqOTIVuA6=x-+5|fV}Q0#In{Nn#*pjyCDx_#;%O|lD9PR5D#-ykzQaw*sc%6YOUawfy6-z_iVEX845AnBOD;Y`t%wzr*>C0SRM8AFU_h7oOndy!o7f!WlY))OS(TANv2)v=Cd zD>CGSKJ~kJ!VY)s*sIhVm$TMe@h%b^8M}cERGam-7_v*E%?~z$ZY(uK5DN<*39omH zNQcu{cGxz`j3LDBrMl|;AG04weNMgPt^vr^Bjr+hv+Xf^AZ`UN2#m5ekG7YTGKmml zE=3@|C>Dr;E5)q{lCXgg`mUdJEdfps6z&8I3kwSzxj3i$2&UfyQWOd8(_bkO99Xpj zjl0uWN*TJ#l8iAzhaG!J8wVl;Py+ykDQruOOe5mr3qhIz3L)ORe}9PCrqqwl)AVfM z`SY&pg_({PgliErt_G+9moyW%B)&}I-~zw@;6c;0M_9>~a&i|gGn4P6N$y^DqSTBr zjxpF>>y)YvF$&VO{bf%nNl9t+F>hnVE{M|G&4RC&wGJwnwe8jtO>&6V_NjL{0l_Hg zJu}oK9|d^HImZ~M{M#DfFxYH6nkJV6BsjlDbGN;Y1ICL4aT^ICR8>7*M5w(ujqaXv z9s@fVvag4+M%*2cgb#AwldNL4U@=4s!NztJP1BGl#m7R{c^LG4Uwq3oy~G4wQIeg| zwCR3IDumFCM$KX7Y&IJ}Amo(P|+;s zyeolF+$Bf^0vRk|Kw>F!PkonsqU-m>4Y9#e5jDdPob z0h9z@$Xel+H@sU|SP(u8;x|LO?Vl1e7t4nu;4sUB!rPD*%ro3Qgm`*(`fvZwpC3GU z^!&y1l#-cAAULjg*$wNE$^ufI|p08>89XK8QN3CAl2Z z;iM$_p&483Q$_;NlQg4k2{DGYU5#h#R!XU=ILhgibCGx$#l3l&m_`$I-8OA5+obOC zg+To96oG?7fbbi!u(Yk!Ta#OGIb=kPq1oC_st|&JJW%qeX-n~WTXPC9f}rcdi!xr? z+gp~5CJCHfH5R^DmX5Z1``EPeN&zT2dvRUk(b8S)XS=Yl$l8UQYBNET``2g}*}0Mk zUS+Z}v#O~{O4x;b2755c3;Xk4O7QSGNPzYSYyzCCpNp`oAOS4J<)vwx$B!SkZQFM^ zet1$h&5&e!dUl2_{FMP|1D%pMj}ytmOf>7 z$GaeDyDtJ-v2KnctN5D5!527s(-8Z#qJ!e(=p zbICco;8P^%N`vuD&Xg`83%1+E?vhvvNTxI_eH@>R8DkiR^=j4Rl)F9+l;YX~f;X&Z zZf4Dvw)nYFmjLk6BHRuZ7Kk|_L}M5){Zw)EQ-qLmPNNN`c)6}kw_pONu}Fb|W|9LU zK@pI?z(d>uuj$M#To6Xwz3aPDB-~jR78VxBD{^Z;sDcy*Bp9)SbEBRFxZG#}LX0OT zC!wm&wzVyiVBHvUo3&HDEOA7D5O(Mtxov`^*@^mt2lxN*hwseHT}{(&HvLaO{XA%C zfcsy37*QuKF*8UDa!msAy#egG8~5YYtfcp9U5&RJ5&;TGDQ~#h_NtAMt9Gm>)n5{~ zy!@?M#E2OrkYIoS_iG5fv(oonRmEmoJh=`bjH1rZr4#_&iR)n(ZK(?2B+a;59qmH3X6B`&ExDAk^Bep5 z%(uzw50W2$l~Dp*+!6o{0NT-QXJSucDp*xDyG+7hRTZWXZkBWrt|#^o(`(dZd#d7yKVMF8IVi+HuCSqNCQdOY;zHJ zge6&f?j|P@LZn&Q6?zmwVqsxn;j7|?B8ebqv_mmMP?-5lxdraBF&~p;-)8|i<l+HCyyTu6{I$s%@C@sn`XV<{POf^*YzYJ z>Vx(KK>@Rxy#x=onV=xdnC$Q(ua^3m|D%*^=K{ z3pazrA(9;2a#?if{|lyuQPIRE8pAxR|7D3uVnZQ`^#l2D50iJkFymSRnSl6ZBeVtx`U0ttYUX73Jj zE@nQEQmLvqFlyAMl$84Zyy6ssah|a4*hB90z|nSkSLc16iU$jl0D_CAy>%=Q;`qfS z$Rn*9sAE;tAynHS?et#?$Ta}zL((uM=)&eCfVfjEdA1`0xjSrTFMUe>PV>>i!otFW z1}K8pX*dE#8d)SSKDL-CP1Bl%oP8+oRM&OiCunyQcIZ$u&f4oy01p$Gsok9hBSE4_ z!EA^RuU2i&y{ois^L!-btn-KR!(Jxz4uOBAASQX3g?O84BV&sJK9J;R>$Rk&X@<9? z>Z|p7JvOS<6iG9Z@9q%u>mZ3MA+*J5sH?FLQp`u>z52CYR-XzDq1DEL_ToBR&Fxa)~n7BylByyePLPL7GYy3tgX?%4XfUP>pn5 z5(I(N`2$D{$af5joy>LJ$hF>VQc3{GuxQ#_#e(4q#^U5#Sv#zyl2Y#bP0lG7f>AuI zD*u7v7My&9_P5S)kKG!8XS(B$;fc_fPSMwg_S5du`#)!;>59UV)bJi%G)kX?izf_b<>2nRdMK$Qr5ub z56x+yk1EJUfCp-sAY~+CXzHeI>tO8DHa9XQ&xM3RaDeX)4}PrJHpx>eId4+R^~kP% zatvdhXU^yOw*)l*kK!kaY4fA&;|b8glOyXonprcJRdwaA>xcNfy8!Av7Hj^hH}~cU z;QO>R1(y%xQc9mvUDsm^s8y(%^|rHdU$S)mNFQ^vt*#@f^YmKKED9RH;qgNdaK6Tk z4{ZFX^UgUpIjHDP)paC3ou1+i`jLM=h)W2ZYW(4CL}GtGdA_Exsz$p(2msdWt)_J^ z+II%f+#>W*<{t`QE(`Oi!pvwLI4FLDTx4Rl4JiU2^64J z+dh2wu&!%@M@L5~<-h*rZ!ccFBn;byv$I*^0$!PJJgqg?x)cu{M_zwIb-5BhabUf? zzNWE$lmD!!!)Gaw7E@6^JL^;02D!#th}Lb(HKkH~SO7B%AmzS)NxG~?J^%y3bH!53 zXn`?=s@lq0+iJ)(#*Vnl}v zX6r8ziNbIDugeeFMxX!ehSKbA&5Ps_a4(Q7gkZzUsyfz9?QS{itY4Q>ykrf)ek&dN zWg3zfbXXQ~HWFzhir4y3<(lQ zOmNcj`lR+_-q4M#%e{O6N$f@?*Y|y0S8dY_7Y5q4Y(rjsD!E8SIMDIrTTO9<8^Qr$ zizug|s=_vyxFxuI-*=-?w)X)6B=mbE!3pj@MvU{QZ{w(ML)*4hRYr62uy`ph7~uGn zyxV7}cWOeJ5fI5SXSf+lA2a9NbzL6Of&ZVs_jO0!d;^Vu^0D7ZDS3?FHcj2G*InP8 z5k7n~8pw0E=}U1ay!CIdk4}HruiRgHpUTo~nFmnrPSBR0Z${~=i*EY8Qy@j`Q3z(V zo&7jgk%lrJgmYFdMG_8d1|Sv|CN7Ct38rWV! zyBkGtF5)jAV|N<_3r1_3_R-_Vj~+Z~+M|E_x4-062qQQz2gGSi$Xo^n&}h+2Fb9iN zB%l4=DF&+IMbOdl>fZf(M(g`NR@LiQZ(clqnNtR)?Reu%Kpl>B(3OE#HA*LV`N5p- zKJ-9J7|~}veT~;g%mkr$anJ5a%5_zR7}~a~>#ax_gc~@~%j42xeBrXnnuiF)qi10mSFvFxE<9-=fyI zYnij#u>iWwrXOnx10?6qkY8jUM?p}@xs!aHv~}GawN({cqpGTL*H*k(kH+*gMPkOZgfA|Z8&u|M*pT_|juWRu-FYTO*m-RM1aqD7jS9M*D)hMblL|F#h9pJc@Fd)J(p=)A} z`)Lu3bzPfT*s@*coQKGMDY@_aQi=q#Kyu%B_V^M)3>bt2rNZix5>Fk zlS4qigQs_4fwS)K7oR5-2h8fG9xyXWHD^>*JYDZ9tKBkYlP8Ue$nRxFd%?|16qan z&GE_O`wx!lrhu={)=%~7l?z?~_+FWl0!k@uyL$5E@x6N|0D@Ug>8GE5dhz^uDFx*8 z1mO@q65h1b%!Cf~BL%(tQT(lAwj38+DQ*D`WY^9XSJ!n2VKkI)8;TNi$z5Z^nB?wz zSF?l>2~##Vxp!Bag&4zmHKrIal5^f{HdENsu_f%|!METHkK;0lR;hRy$fMJ14By7F zsx)&R{mENgTU;Q(X^IlR1tg?dR=A7^!FC0frmWg+8Lp{^)9gCUmnUv{I9&y~LvVmf zeTGY3RdpbYq_)6}+E1j<%ff;%uXyNyK;PAfTD2<+!Hf6(zDcc$OCkso0uDM$FD$$( zdr1IypVZQJQc1$X!otGB4GSY*YCs9_vgXt7F-9{BqkEH0vq68{uC~e+3PuzP12WKv z2n4bMn!3JUw~tpx539PdkX^9Q#i|bl1mH;y2!IHUgSxJprj;wD6wJJozy0;EKmYX8 zFbJ^{J{bfD@L?_=y`>=u3Ytm3&q2JQa*%vn!`=II7Q?wH-6ins+M?F=aVh6vOn{d> z+tYrc83`W9l2S?)jr92gy5;dcniJV30e2qU2>Dmr|Edj~pMf>vAc6 z(N0X5AnrH|nhz%Q8S{vB{*AT(qS44|*Lf>c0(My*W3s*rfT)WOcNkaKv9O@G#vxdY z(a4-jE=9tZ!otGB!mpG=dds*dk~ja%=d8HX#Xy(OnF&b*L?euZ>+J08^v$V{XZ&(M zUL7@c?ZqwZduUR+h;9Y5CRA-zo0K*iIkHe(k!)hfxn#*{E*G321d{jf-~aym?@vxn zD5&_^>H5X<7wfb2Bu%vB+{No_5%Ewph4}`#c+Mq+yk5hMF){>en#O2mln_DyczJg2 z)`SF=Ek3Z*g;SJ51c3utdpkmGygrO>ayThr4B`5)j5X4>l0Ss1ltT7-_wro7!Sms6 zB?*+#ycC+o0P<1OG^Lcymd;&(-4$6ohx2522@;$vDd&>ib+p2^aa0KFb)WOp@RpQI z^Njwx$fbi*=ms!va3W^B;DUiWJtcF{NF>qJTt?yGVF35cSF<8xUxvq<*v6 ztk=8R$bnW>v92meUfdzhT0LrjBWCX2Z#tllg3}TxxsgUvgr-6=?tskf-pRcuPrhmE zCimMau)qBL%c~bJOUWcJ48bG?L5s50eEg0au_@fOiz~<-?f}_!+01y3k-Y4hljL&j zy}Yiehbs03qP~!7M#KS>Ai3ajmZF?X+!A7}LWoC4A!qOVKBerEt_-*8BXlOgsUfB- z3pvKE1G%nicVDkJ1Ddh~_KB&NyNMEe_(j(Jo3{iNG~nuxjMlcXs%pSSahu&80?3&? z;J0T%ATe*P4C$>8jM>ixPP3cWzvuHuUs1}Bl!KZcMtBbut(6_GMRq9h755)vS(K*`Hma)ug= zpo0arfjTc)5;46Thf7opVd!Lg|K5E`B!Yz(FJAoe%hQ}j{iWGao`8s1HwC@#is;3P z=QltEZ``3%w@umVQezAOx@vCe!1M|MlnAi^rC`LzMZZoW0>rJ;S9*`naN2;NgfifH zVDxF{7^}8vV_U}b9t1_nwbpgr&X_F;kN^&Vg790iz(cu4h%}72 z9D?NF00@qNHr_}aq5%lH7<2k=ypb?~z~LK9@DqQI3jnYAfk;4d2@14W$FW3t1=78o zZMF#lUW&^BZh`4wKS7JEUAO@pmkGugLJYZ-&FBRXL=gNiaVuOvEG&E#Xk2_RKcs*! z6%rqWL_oLe0t0`2u*1ySrcHekL~siz>86Ir%T0r(Caol}8(_=%l|-oTJGqakh{eb% zgh0wYFle{2I|!13iu*=60UKJR1wo(~GTFH&c!tD__O=x2s(SR`0i|Dl`Q>DFT-VK; zH?N;Re=ac4BwaK{ksyo%kcVY>=537zAWdK9`;faE0r19FR^9TL?PYx>T)@MZVE{hv zO=FjGE>edxzOa8)VpRZPHKwl`tG#A|2!B5oHJbSaa>NYH{n+ya(jPB%jUGy>Jl za58*+dU{Io=Ftn{;37FW4rXSmaUcQ&C;+07(q_Fmef`?K1Xh7)tE#q8`XnKS7|ck5 zfM5`TB6m22L>fS0{A!{VK<6`1K_UdeOFpWOAD!Ggd-eK%|DXSthYuco^UXIeo<2W2 zJq@&Ap|~4y5FpbmTTht2el|ydZcm=AKE5FI2SR{%(ibL2fx9c0l5(!ATn*HVLP}|yS=;r+WoDc*_r4xeo}hV~Ye^|3NhD*8 zO;ZCdxm@CW<`bWX1%iYPaKyOFOeeX!Z+b^uavVU3Bw{2PNlI`pa^=x1;UiMf`!BQ+ zpAt>W77|9r7^<@J~5TIz|S`Zc%e)Va&8DoqfN-hJENa~u|zOb+$ zzJ2YJju05kDk)>(*87krNNNBIAvAS^5u*vZp%9aF5v>(mPLL4HKoF#0Mgrp^157BT z^qb8fGE>*pp;Ga-s_z{goz#sNuVW1egn$Ac00GjaUGNqHCBjI9IGrAma`PMmDRiaW ztD8SQdiT|BmO4mR8`R6yUUXvgoIY1toZ#fH2a<=|$;1w~^&rZZ|$O1tD5F{6r zJNNOtwvRE?b*$UQ>w5TpD?t1HLe^?rUz1{Y=|>#D$cVa*Tf=Qt_kF)!Z+1nwK!XSx z_DJ|@-~d2Ea(B5qNuU`^SXb^@8LrQeP_rO%|FI#GW?`VND;q~$<6(PmyF6 zNW%pLs7p!3ONXk34{f(F{zqV(Y*`n{>%_vs!gZ!s?gykqW1vNv!LjhWAbdEYnJ5S$ z1ed>j^)lxyh$IApH;5ANn2NCMALC97Mi4MC8V!)U+yPhdoKoNQUDs1@&Q8w)@clR6 z{)d13Clew=H|XBYy32i+Q&-#pBgVB=F+>^#xs1}rHvC2SuokN^bv8kp*gLN)-HM2SEknow6EF>o&?0UZ zJ_<9d>l#4LWk3MQX}z%UwR2sRfC7zxj23JQNgpvu_@d$a6uxe-y)bd}Vz|`16y%(% z&T!`yHf|0j5U65wFQ=z(a?a3Hd4*G;<1##~2rEdUK!Ac7Z5*3KIHkbYS{y`avAA}1 zIoEab==cOFcW3LiZ9HcqDhp44{%QT{^vS&k5AWT3usTv1Gm>cX1h9_PaoZfPR!xX> zZ`-;#s)l&)>&@oK213dypPn`MPKICJr|$8?$A9?lyXNFxT{Yi-^WE!Tp0i8e9^Zcu z+fvlFE5=ZSKBe_m3G`h`XWeGg_gP*b0Rr9@w=)x)G;5ng-k=5zulP;^0cmj~e=SG^ z-~uVhOWCT>ZTzaLLPTAMs){LjFYNjcjjJb0N0}F$C67N z#~hbHnrOgoA!K0X!-CcV>wvD4cN^LkLEcl3%`lnY-Rosx8zEsq4)#gnVm~U>jm>q0rj8xp(hA7+$?OH38zA z=P&b{&4YTy*rr?rMNTVGJP0U#?rgvKTgIb_uLa&{8)1e7YKs?nE^d8aRbi~5x;+UeV@Ub@ykC2L zOHQF$Kp0>#3snql+ekVu8C6`EWUUTLmEU1>`Qp;`HIF3~_pa|ju2xN5x22%#dbvmm zBz92v?cnA0;c}UISWoY6F!rh8Y@nO`8O~B!<93Qx77b8w^nKT-fgYf?05lAcA|-)0 zE`kV{83ak^x)_j{iYpgmp`{Q*-PB>AiZmmcQeS*3i%;>!J+vuJ&7&oVgb_3gG1g;@ z7*uhUJlc|NL@cM=@y!Peu zvZk8idRqhoC>lYs3PFSqcxYBJjL{Uu%J;zziE z(=OHJU@mbjng?k$E23zG#<3ZskC^nD$y^cAz6#VKn#ZmsXgAgX!2tsa5il|cfuLZ3 z$#b{K!B%yB6oS?D5RYZfKm6sdFP=XA5C8Gsw{06^b^rc@2lwusy?7mB7!Z(BUs8^= zHbe^{((?AMP2X?t4z57@$r*yzl}}(%d=<4b({PC`O~NU z>1oKhVepcF{N?G>SFadjyIO@R)^)QQYC|{8YcqKf1fu~Wc$MlTK~m!RCY+zfNyL?N zoPa3KW)=a0fwW*@z^qSw-*-2U`hO(x^T^so+)9^3u0RkpK}8z7Gk8c|RjR5uhJVb9 zkDZ_UQXJrHItg&lqp7azwyoWLjMkPVnsml?4C;5>f5TWs<=98V+y0PP9dKD;rX4#FV9AqzJVaRNc^y?&qb^hgjv&E9;1cn>7!BjU24BtZdi zP$0nhjGPbNYZ-6|S2h}#NM3p^kU$cUbkT%`*S{_bys{x6r#_!hFxyqIhZ(e`NZH*b z2gD0k>ZEUlODU<-6FJaHmv`)EM#2nU5pWdfB@;;k>5+w zQ~Q>%91t|Nfz1SiS#jS_14) zs%;*$_xqCjl#06uRD$qTg&%x7oy(&z@}F3=ePapFH~Zo0A9k%d@A9 zp}=3hc%Ihlm_1l*NzZAsJ}qXao3m8P(b3WO_PrasZj(~)fk2=c0Z5!>SKAaFT`4Z`(%;b6Qy^Y9zUm+lP$}Mb zy^(EGw{5#~@c|%ApW%A?-RyuR0GvAdY?b6{WrmcxoJ*H{SIz0daCF#2Ss&Qu`liUTc?UR*Z}K^sYoCD<9XDpj2{7UIx6#wFQ765mRQZAcDB% z#oRde9xdEXbXiS}zJUm-47f{|`uOqPzh&WL;LQ*9cLL7koxquw=Qn^ll+6P8kwz(w zHzS*>=7>WsjrGLcC2{6$y@6quI7h$JY@o zOpDx$Ut9*3a!M-Y_-IwvbuMXhcGkgPzIxT?EaG&%F2v(+zxmTY{$nMsPtSh%`Pq82 zzJLE@uq9{r&1Q3Yb}H$lJ!+cTNXey=2gaDz;ADRN<`f_*-s6L#=25+B)?FuQ_@lY+ z@_K#0uDxvX*(pM3r4^VVjG0&3Y|w3*Rl73VkcF0kB&SFMl1+P4pfF;Pix3U)eLLTI z#^2oaKnN3(i#*?s#J&6Xo;-O1Xg%zQQm(%UsJoqKyL>3hC1^`8mEvV=@3}hgFttCz>GQ=Q$&Zq? z2JZ4{11*^CO4T|fKoTT!QtY=63Rgm2YvuBpEr}_)v5e3@dT*G;-T{Xs=zPn8X?louV8s;#Bk1!z zCjcd<*RNi&ebmy6}IO5bhh zD)Jilk5_FB0f?ROy6aN6@ow##Gp^dx?yP^U@8UnL?j0x3X1sTNw0ZIB_0y+k&z|M8 zO<8Y_@8AFS=%|)5B(g6Kv!;0e`n7d^UA&Mm7Q!JDC~$heWynH?Kol=Vgc#r|#my+` zu*TMG%|mAfu^YMk&FfdCT!ZFaNf&jUn44n z2q5P^4UG0`Nu-g~=grPoSRlEZkfUj!NB{_ZSPuTe0y*zLc~B~Lj*);H%pjL2r+k(< z%ULjwAaBbtkl;wLT~(m~U~IMMc`79CoNFxzw+;c6E_L2k#=7lt_w&;bw{GLdp{iquE_S^n0{3DaAnQ`@pPdCWqvXYlywoY( z6OK+!>Z6s+==xLJenZ~dY#2hl-aP%+e|z!&{M+lFevDuRK8WGu!XTMwVvbs-6Kz${SAn|C@7;Z*ZUBzWmKO=lFf!IlH!qneMtGh5$x_tmECaeRge@ z#?k;p7*m9Rn(nURJ06HjpTr(g-xo-}6xt!g`G`5G0o%3I(+JG#W{xXtcHqM8YI3EVcb;yVCmn@GDCe zNuyc1hSmETTFrnKvf8$6LwRd}w{0vVvN4&ZjkPXwfRdmPq~P_FVj$2ufB_lpt{r46`_KN>-QBD6{rQc@pS}C?@BP)kF^F}VBI-DQ1Qj6(kP(+A z5z6ax3MEKj03w*u6cONGeD&r1zy0_B?ce>!T8bY}|NB4xmp}9B?vMSgzkW9!1{gwe zM<|=+*-oD+2Y6qmhZX?LCV7!aBfF~Vl-``iKN{>G|F8e?5C6-5`GW{ab+K`l)Kw^H z6A^t9fFKE~Hw+>K5sU{Yx?s}h!)$NZb5ULwt4GXf3n{sK8HO~BHXcr9!^Ng*Dqe#m z`2C|pH==tXx94+8X&lEx$lu(DB;4?wBcsDG9FND8QYq!ZmaSHouu16eX)M;47igom zX3rE*>qS}8J+18}G%2*>g-1CTMI4XJ`C-g98$s8I=v@F(vf+Z<40wTuo#yMJnd zZP4u6Gsy%CQ6jFew!XT#ha!kD7RW4p^DqAOYg#QX_dizSkN?R(JAwuv917}!C8#b! zL^SSAqn7Vlg3~g%N3EXk%WwYCKT!xk9miDtFaPXo`^7JEN)%Kos+*8VvjiGQfq|m~ zYv!@IJan;*z#RPFndMhMKm7Wa|Ff2Fe*4Wd6>SAte46^hex9g{)y@3GV;u2&bNQkJVyg3|FO2-FGQ@uDbPZc0YZgm9jj2pCFxgMHI8RiPi^K_0y zVNb$4;0BR6j2B6>WOhE^KcJLKCW*LWpCu_kd_+)rftk(!Qci@>TECC9E<||5b%kEO zth(d0SiPnIa6F!J&XNF?5)mRmyI>eD!Do^HT0wiAag75r5g_*oHe8x;!wnw>5@<|w zGEXR`T0C6s1qkH37_i)M1GkPyGg{8ctk1{XDy9e_;>vj?w*8EjQclU$63!f?Bt@eQ z2w{cLeva)$-QWOU}dV1 zAR#0L1a5qG>$S}k7a*eXTQg(NwnUJ4-V%f~F5=7zwZSSR0gLkE7LGPv$rUaI5Ldd7T$z5GaEG7$^}q1v{@EY>!GHJ< z|LM*B1Y4z}^hCd$T9?}y$fs5xBt+Qi5=I&j6o9xf+t0`Qj=Rpz^-Up#;O?l^Yw_2w z%lTX)L>lo!rWtR)0)iA+tDmR)VZ6ZO94-M|QnK(8ZW)mnhI}|4Xqcw+>(`egWYV5d zFI+4HNJ?Ai$sWL#gp?a*b}K0)0V4Ut6@t8qfA|FvArFPL`8UpuJ*_3AE&dQvDFwjc zaLhTc43^}qpPX~bB;8xuS}nCo5=bsLQ}*m!&|}3&_OWT|{b^s{a03Y=XA%80ovXWR zGd2-s7$6ebG*ThrhR@^nMm_KO+8N}DBn1-23vwPTwJzsr5ZpI`&t<^L9a!t-oR^ptQtni3TId5JGJK zuVrA6M5zr!;!81VQIn^Jp~0sjK($>2Y$NB;1(o=R<3Wh8J9nDJX_~i6G@Xo~L?$f38*N4ds4(6}FPlf#i>o)(V2! zO!Ijbbo6Beq zrO=Ey52U$AJx`50hZKEgbwBCdj{TC#^Tur7%PiClH_$B4zEcp~y&^^(Ojg9O1X zLiXfIvTYNDBnXMzInMXX+JcD^0d^D4Yfdnw1PF?V&h0psc(NE;NT`5Ngb)`#T;RhF5zj-}_%dywA zL#;KldMuDBGLFMY%WtMK8~&dxCq%Sep0#tLNJ5hJb&yZzEC4PH8j59TH)eaIzK_#5 zX*pl&Kmp82ks;a2%{~s+=q1UQx50~$MDj=Fav;G&s266GXVhj>^Ac`oO33+mIHr`R zX?mcwUY)cT=F&O;@LEZxWV3JcDLue!CX>3*Q?C&Ky+G`CpfD1v`gMfsW!S|@&bHfcSTb^WX z>fC^y`y0Ea*ez>r_8@Rs-bt$4$N@++r6G<)@ z#769Q#cPCuqQI^ylZtQ%3ggIRXb6Nv6MzU0FU@X15OqEOLZtXKLtG@-1@eLt8nkUE zh;!>$>h2BT3wQQil_d3w5G(bU_UQV3|s~;u6A2#^Z5tPnSbg_bV^` zozBK8rI1L8IVbm+rt>tFh05AHOxN%uoBiByW41YEf>W8?J>HfzbHfdSAa+%U$x_Q> z2OtDO?j8+Vf;C(oMM=B}Z5zlXNJdib3Ug29Ipu^*!i=ODXu;~?AxQ7Bwl2sO+R$42(4FuIT5(HB~ZTSk2@z}Lj z5k%n9TynwzhvIn)Phrz?rahS5~>Rps!whVpW8AQDW`V!*u5kB zu{J(WfJ8>%FyvQvry*xmeSGP|Vpw_$Z#D`GhSB>=iH+nX)AS~GIUc^JkJ z&*$3mkO|^rOAhXx|F@hZTLjp1PQ1Xna6#)}%?XNd0Ij>TQvW#MUUXN2&oc+ZaL7Ym z9Dtt14N55;4&&i)i17P2m(OK4kIJXdJZN2kG_#9=Rk}o3NtQCzdI`3=hX;r!I)?-y zt=(QnN|0)V)jCf_)Ra@smL8DDl%tk9mFm@BW}c(~lrr7l<8V0GjCS_^F;5^~0&&9) z9|ff4lxXJRUOgo6*?N{6Ztk>H#~6&pIZrs5T|JPahC(4p`&%)=wl~@PsrKHsD=&ax zj%{*NQYgYNExDUnN}w^dj2ouQE44~``QH38#j`fN8Of5FiKt!`5xTVQsQu5q-s?}m zFj2qD=a!wwqBWAwObD=D<38Xm5{A|>OC*zIAZP(d23Y$Z>cVk-@;lk(suSWiy0)eE zZ$Ec+OGJ1#A<^%!*#}9n58Ho-^g~^N5XnT+j7HFmOYUx50I0R6r_o!rae8*TTh?|p z5zscGcHDQcSJGOa^p<)oX!}hY4cEyl zhN4zis3+3;9I}yb@z{5vwIs|q4nwjuP38W+xa+a^zCKne@;a(c=#DhAJR_wH$&!)Q z(&R{DK}snuSc7tKuMda|Bz6C(-wz1TXSdRnhI}DOQQN?7c<%3G%OxI$^zFCzrIcQO zVIQ<7M5(1vwn(psmuV`sc!MMXOxn{$?M&%OU`?=;gA%+(sTGWz(&==vnK^@;B#)5D zXX6hWebxDV23pPSP^MBN)&YP&ibC-lZeSCR%2|qO&U9G4^hrIbWxIW7Kj*n~qX`NH z!4~9ivD%Q_+pLI(Tu?8=L%}wAC{RLYN7{RzBql*F0SaHdFWH$*UH741Z~d^uY^f~rR<9KvS;?h^Y00GcX=r*jE-kp@pm;eKo z?=rA05;AlU2vPv-+INBkEE$tPffBE*SQg^KS3AD4eTv$5lI>OQsMlI)3v1gwhQQEM ze%PB(sUw0fM@P7S;?@-r(H1$w9*`1~320L?B1j1WqSHlYuW_@C=Q)1My&AZ>-yNTD zW{Lao5#K3Mo~7`yJxVF#cpP%B)gfrHtG|JyLPpF1+e6Cf z{{GGD*RLhL%cp4^hVgL7DNFL|_j3k97y9SB|GL%F-tJirVy*Gbx8IKA;dnTnj;Cb# z{^F5U0!Vz>9~83^Rh>WAEZA zXlXKsoRR>)-ke=hCwh2?3kJZZ}-D@d%Y+mXgRu zwk=JiCAA9s^ylZ)uq0XIv*4I>qP6qZD;)`KV!NQ>6NU7`Ly}-*3qh_yIkku@JN~nQ zOU35}t7pX3x|I|RNJNrMY6mokyBui}RL||Ymv$ObSITIcUUx^2`Sjpswyu?2J!1yd zXLu0;`79+4${ zY^MOIt}s{eG_DPH1bHWnx>8#$k@M$2FS;NgLAra;0)6!b`M)R`K}d^glo1pRLIf$| z`bkQ2AZSP}VZzA7l)yCAdVh~;t?1omDR^VHxDi`RHq0_PrPR6hPozou=rY10A*@yt zb~v2cUjdTSR9XOb)n|_xB+-GS%}6OHvtb-^W9>7aD7DVVacubLQQVf0%sL(}G^~~? z_t>27K3{OtdY5tbdL0iQ{_VHlMuc8UhE2>bTjI4Q7VO;b)c#S#vP+BA(AXsm)QGBR<=-Lb zF)0m6LWC@(rg1;y-01d7V)?B8o*W)F&%^E29qc~CaY+RU?YHiMe8oStgT{K?L>M^Zq*Nu+LgmC zeJy;_7WtMg4zHK88g<=Xf)tQRB%3m2BXmZS93ciairWbgw_d|e?uuF{s@7Mx5Q(}~ z;M~-$eAdVc%P4w(>9)4@J94dlJj+fm5K9Tt?11>y*IyqG#|lxQa7PGCv?Q98C^C{M z8Ha3ZrDRT-$!G?cXo4AaFq?oNLL{Vxir<*+XOxh<>$irS(lBN~({y$(5=gzLN-Dkm zo+*NmZG&VG0r)r$$HT!a4a1?jpU;!G&_P6Fwj@Z?jBcoqK(^xmoQzE8^nl~>kSyin zA$v|KwG7pW*sjtSqPPUXchzLZbA=DwUK9<0q{g$y@h}egp(hsry}vLr0Z-~l~=fm+dODLB&=XDk*;9xH)8DI@oH&VUGR zoRV!XZYsmjRtdT=*tTzPf)fHjg|G}G8IUMr5F!N1EXW|mEhgF;1qs|oJhVwv&s~8+ z|9Mu=*hR6UyaYRry1j%h$3<$OHd(ej(~biTi@ zHS{FYi0?d}9eQ_@9ukBoDrP(!hVhWhpf{@3m!u@^-xY=Aah?iitBT~Q)>ww3A53d0 zya3^b&n^ZlT%paXXLoC8TawB%vn5o^6z&9{5ZHaD*+xa)4L9(`D|4U7$>;u9an&*v zME(GB%Y}Xlt#MhaXJ_?9fJ=;FQ9d`~}Q)+1~Z2QbDmV~%)TD#I=+GD|>BqSo6QXoqv z&_h@%oI%u5q2&Tm``-5Lz3h9gDY2VT?m`ufa6VI{N`{n-3T+1_ETw?rdgk8RR#49r zB|)HskU$e+xr`K0MUWChf+NBZMsK?lh?THzXJ;Ik;;7RKsp`x{g;@wkz8X z_T)p*)5+X8UPFT}KcsaU2H#)A<|` zz;XVxc za0$y>spWh=ms&T*{NX8GNas?7I3TA+$nVe99pbZ^6Qlu}<6$+Y3*04vfWF7H#f!M% zhAkNI4(TQ~6BRS_>PzXzd#?b$;d^LI>*Yh~y%g0@?|h`jI8w7z3&~sQhf9ziiSaNG zDH$7@m{ZdvyL2~V!R-h6Od6Eb^cOv%v1Rwv%I@ZthNj~!x&l)4PXJOag}^XgfHIX@ z_it&+X|*wr#bQBHPl5^P(@%uyOI7ulC85tpB1jU;R)VR$+GT4~x@I5?9^Sg7HD4;Dp5vSfSzp1Os`H~|LT`to$kK*gFpJs|N1{x90L1*JGK$@kVMkD&H7K2xU!+1 zJT^M1G^el6`n_xt-%Lgb5fTUrx{h5Cv_(55f-5^!#5zQgLC)W=lr#VsQ_k{m4?)~g zyQz?gda7dFhjc+kAFW+Fy@ABWXIJ~^bR~e~PEv%zTLL^i^jx3Y<%|8+>gVQ8r_&dA zcW-j~`RVIleEG%Czc~Hu)#-~@hr4681Xx_J>qe?YcP20eJcFS?&@5RhuEgtSi=z0Q zA_3ZO_7i2cH)vCp141B>=Yqd*KTl_At2+O{U}t^S0tGxC^LWS+7u(W@23Uv?-^Zhs z+`}VgetRA-A>l*LsgaOIZa+R-h*?MKFpk5J&u7BpMptGi4{01mcfZza5ifwo!{IRG zJe6rWmr@+iu8jM`A|j^A74>*L4Z|1@xDSsYZI`xbprG*ZT1rckvm_}beGmEQ?|~a` zASmuXQChV6)ao^Ee~vyU=y4<70!fY5j^k*QqBhB>aP?~0EwF0FoU@U0JyDgUP_zIJ zxUZkkQrbuLBj~I&Ly}CU2+@p4bNn{VjC8a-d7V@c(FdRe)LLzk-Vnm81fjz)jEx6J zsBv$F#68pgtfpK@wbjy2n8fEF)#(`~{j8=_MgSEvTC$ z6PdqwHGK8O7k~CoeEnztG-Uao{B!?C z6_DTsFm9AyAW1V22oVb;f+PZ<1YNL z#ke-iBgI{Tc7}E_x5OCQt~c@P+R|| zWr_%VJV+uT$>-ElI~*lKJFLa0}Ug|hY zaSwf{x*7Iax1AxJO1+@nm+65UFiRqj2SE4t_a1?Uy`rVAiS&ngqN0>KU4D-)J+VUj z!4q`g+xAdbN9K}B1+OfyJIs!K?>*PwY;sa_X=grq0+ zAhC(}Fn|ad!w5+scVjZkDUB%usP4__R?C<|nrZ=Qi9^l=+@sw@cqIa8@#N4Iy&>H5 zY8=`{tUe+3yVjJlPd{asVN4nb?n)N0g=`_rk@gyrdoYC%qCkSi#pB1219p}wYCR43 z;)|dC=|A!HpZXK|kNp}y`x*Z5n>XQAvIs+hE#uJ(CYQU^)ILGCXS{sG^jeh=jRAph zMY2Q?gcc@GTNu58B7l$x@XFw>fBDnYv>p5H4b_@_jj`0Yi*Sc0fi5uw-iIPcB zG>&W(5NOLrHdi51&qP#d9=N}+-@ZvDC|rj`wEhly8-4xm$$)~O<;Rfo1GD|bXe0f= z0HRh;W<-rBq1x{hAqWvr1c{cT5hA&Z1@mnK#-|^%y+IO5kXXz=BYZI}1i+7GlcoqG z#$h;}4msyq>+3i7Q*|Ic0O0Tbr%so%`d_Gbw14}3=W zo`M8wR>ty6f^=e1h)9t+Z;P zfdQ2VTyov-4&&hN#r^3K1o7I52(OjnC2C)$>h2&uEN-|#T1w57l8xh#63HMyD5Y#> ziZHWzC50PqSV)clBBy*9M>DIXlsb77tq~tvO*=0zG9qS2@hUwn#^&3tq@-sINTVz8 zlR$$2kq$-3gqSgo<8d4#qLg}`&H*%9OEN;GmMk-~@&B>+9_(!#Sr_JeZdC&yMcHzm zUf`?+kgWiy=x3i@8F5&+FMWyG8Z{qKK%9XKmEUc;ni^{WG5u(%93M$d$*3QqP3aW+uL_v z7Ow_n@hT!Ehp1AwY-c(%dv36;*2$jsY0^U0POxA)RdWX2dLzFvMJAfUMtD73r?%%Y zSA&qSc;y%;&EPKYi$^YT5GKH#=Plx7zUn~uJtn(Q_^G53 z_pXTDZkh{DIp~!%vg))hz?tJ}`78}vhIK*p(zXQ{Ut)OLdLLVx&lYv4tu?>w3hG;K zF}i5GTSP4}8-|Ut1)mwrSYol3E@R7{PTRL-$!{cEIcJNF4RRTVL5kknCJmc5AOdY= zZAi{s^~~()%(T{+4Sun>y*|$(wv`4`Mmg=?GVyFSM{H)`kNNy?V_Q=1UA8Kl(Sjgj zGP|?qvxRP2sNhT9DGTxDAS9+EqXqI=Q{o5|5w67$NQ96RW2QMyO5(^Z_V*6H|KXh< ze6;w@+AWG!QtD>)n~u_t)CoY3xTgcN`LF34`UQo*hqmio6na=5gD@~2InlY zD4ose%yd3iwv)J}cem7-Bte)AODJs8zCw`e&6@giI|Rsh6O|v<#b=|(OnZBc60aOf zIbRI>i&-6>CEQLbs6Bk9&KI*)S_>yCmLdEIfafux*>JZ!}dMnRA3Rn+@~%%$duR<=GR7b$+kt z67OJLRgYp_6vj6#@W2PnhvC%j%y*6b(Q?dYCKtv**k9p8Ot*?WL%~;O>NmDUoiO9^> z`twxnWQ2P*Xl(Iq@r9G@rlQf~YN$hrn9t|Y#*@{m<4I1qXIFEHDDjN`T%-3avh)qV z^aXk|t=9?dxFGM2gjI;5D1pM*0wq03P@=GeZ1sRUdvE{bM{objKk)uN4)z!dX`B$O zS>bVB-}>?eu-^M z;=w4rin{7r&jy52pH)&s51}stQP-;CnK5nfEU67YvFQm=D+gc)ykB?jLKty1$ zLDrSqM$+!2x^r0qeN6{7c}E8$R1B*zOKtU;wk<73r;#rAYA}7xX2xU7zPp9LJ=%)s zTi!%29x_6%h1skOL#Z`{!PvWpb0b0}3L>**X19#ESAeEPX4yE=2C}&{ER%YnUGJdC88+0 z$xM}k^z{Y_h&K^$+loVLV6%*Ndb$D?qDU1>V5~%zb*hGyxZ(QV+XsLDF(3bcA_}?n zcuFBMOD33k<-ND~@E-3REWUhj@6*p8e{uihi-&nUvg&~nLPmtJ0STYo$}+xmMB(my z$4q;tSS&L)w=s2YY?%I4nykkRSmd;2r^wFfsl?7s#BOLt*v_;iP?9S)&v7UzE+k|p zQ>$KD_SVA$6nr{syk@#cvt*T!lO9i$S%qX=aKjx83XuluMOjXPZkC(Xi$|IzND^Wa zw{;klt8@o?CM4L#NuP|#d(!**Hq|=a^mMbj**pdBSbqsK@n$O9+u(g{C;`uH0QIV5 ztuKn(h?vc0wGO>=eDusQ$_SKnw9~Gl>x&|MO6Y@xGK-{1(9QbLhE*A3FS zV^Scn&>34oe5P&6Ie9?D#zdeSx^$RhQa*t`WXCz8(p* zdxj?M-klg~t-~rPdOyCEpxLZ_G$%fkYrniV~89h49&6w1ff` zt12oaP8pF9l0dcyrI>$kuYGW@R3RbT?!u_x>ZIq;Q+5xcikWzSpLZAgKmB0;!IP6; z{&DrmgQuV0@5f_bHDj3&Mmxh=ZJ)FJ;$q{s)8=_8?6$<*PTQFPwXoBhnnsvSC}%Cg z0)hJ0DA^KQn=NPWgv176lg{2-%Lb*|xI>aUdUAYp|FE8@T9IVyLbEdL4Re6yvK<{B zK6gtb> zuumtePoWJ0r&H2n_5XDEubg^9+E^BKA{&w1tyH7#f2ETs%J zhFL8oPL-{YB-`tGhrRb~>7JwwzJZ~hKtkZL^|AFj)KY7-?yFuTjC+5sY-2U1jcr~Y z;+ES!TRp$45-N3L?qe{z7;m5SgB`Fr_mBMzy)xaR^(u-I1y~pEHrrZb3ml4}R!P0J zRcqSY`e848wR0a7?yOd;uZ@r zA$F2)^6sNDNEl%ef>GMaQe)ZrFpN>zLGs(S61tcTLXLQ>Vr4}q*_Wcdm&fX=RdMMkc zQawPnC5(3se4i`ZOjN?VH42)#i?`tq>5LahW^N7~#$D4UUXDk<`opjO$N#5)ak$V> zq;&%3;GK^?`rz+>#=-t^_D5gd|HJ?OfAvRC_BvrOZO}L`3F}jATO2ID_uv2Py=O*T zGXm0PH(rHbO(@lDkuh{I!q{Y*ps1t*u$`GWm6^%2WwO)za}nE;alKm7(;X#rd+gaV zx6`xP<~H68WqZqoc#55+C_<~%+6KZ%0m6FK7B&&F4x`qi!=vSLnLUj_*Gl47m&CTc zTCHZ!7-sWX&(?d|yaK#)#{wkVnU08!L-N*$@(m+Gr9|s(J3->2tlhSI4aTc+byY0J zJO80+VS`JF`K-=n!(uUuP{g+6EfknpExl!?mt#L(t6bYRaEsqgf6w$d_Gi1i46Tg4 z+n0wiZIjutBZ?xnRZzFw@)8ghU3VQ28wv27}%$}|# z>QI+s=aw5CnPRvw~ znx0(>p^#3u*7iv4%*^S&!4QG+`O8s+BmzkxJ^xRiEo+n_LQ%S>i}uQU_mA4of6pgR zq?r-W_7?X)e^}D@{`qHlu(*5g-Vfh;_raeZ#+Q!K7Pva$Q{s` zc)GzaSp;7045w{rkU&msY-YnOTV-6?&P->@nVB|D^=Hdzx;3Uot63d7WRlSB*0XJ1 zfFxnR>3z1hNKtFmsR^z9>hk!~c870@nih*iYi+e!_3TUtSzLz9tDZ-s%v>%{hM~;o z^K~A3X(tNMt5dfKrUOB(LntuDh&K|IQl`<$MuEAMvSztjj?c5}h8JdQr=)AhA{%IN|e(3_=1@cHnvZLj(!z; z?v`7m5D}$BNCMw*3`T68f6E)jls2P8skIu$-nvsej;#@~wmT>-b24QNFo3U^dMUIW z-J?q{?-JgCCt)iBS=)s0waON3likh0x>TWH_e>{KYLT?_`7V!ig3K*9EL86ZNjCP5 z!ZngWYXSmd%5!@2un|O;PhjuPe(dd8K?DRfaFRxWgE2W?dO4P!T8=a*QDhg8$b?P} z>Ogt-?)Tn%{HM?Q$>A(Z1SSG7a=LeA1_gWfY$|UZJ+U!@qNt?^1i{uK8L*~p?9*MW z$r!wBk8hz;im{|g=8FgfXWLsPm9?SktPbITihvNMQWs@!kAjWO$+-OD!I!`MEw$YJ zyN`!^@7($C{&T+!_y50NuO2cXe*waXEf-U@h7^E9%!hKYKVQs;kfH7R7fO=F)+9Wr+NXx{WR>HXZK`~^ z-m(exo_#&iDI#nxN5bh$q$owNVF%oDt8OK(0i{N%(MMS_7D{N@ zGbvF^t=Yyjx4al*VMxdbNd*ic!UStPmEocg0max(%f^Wy5;`2={ksP&7K%cYz8Y7@ zt0*y>&1HCc=+Evy{p5?2M~{8Hy8q`-KiuDockc4`9qrHCYW48pgJD+Re&?POl`_10 z*Nb9zwN%|{Ak8GRNJA;IZ4}#lDQmKKKz5*YkwVEdM(C_w63BUW+jJ|=)$sD_2nmDd~WyFo|Pqk`0l;` z?%!1FC;#6sj~_msC$$uufr=uKq+TOLhKleSUORdQ_M~w|3AvQCvuE^4*|z63-A>`i zB-766gNzK`PD~urHf(N< zrEBdyjd`U=0FyaEM3H2FyH-l0n^#23n42SzLBKa|XfQf=*AW7Sq1ri>{$7n(|8{!o zUC2f+oZ_SHhG#aEJNtX{*$|Rs`{Enp>CTj|s4>?Ju*W`jn;T+h>wWE%RwKsDD}Qaa zv-c7uLX!M){gH3EWk)?Yowluu*0LAw-u7z0p=if)tgL~JUyFCG;L932Tc0{eY~Oe@ zmQrdd$QxmN;iW=bU%(P&n9X`?JyYFZw?e9CSEIyGYw4NBwt!5_&ejc4YeaEx)4%>w zZZTj|q_pXxsv+c_Jv?a-5z=}gcG%|1MNels=Cfgck5W<*>7&PofBNOG-`U&$=%ep* zy!_&if4=|8=YDd;$tuS7@YBy9K6x^`_tr-r|KQGt??3wD^I!b#Hy`}ydvD*_*HDq; zU@vCF*xR_uYPzv0rMP!8L3AU)Sm#OsBSLFRDT&j*#92w(JI1yFi74gNQ7s%&7!(p= zK#C$lk}S?6jC(pWi-ACZx|G5qN|enr$O3iM+voQmjlGWf&OiT*gM+vK4afx?7f{*ES|cLZ^O(D3Ig?m6D1~D;=se2t?mK&#>larHiy(xvh8z(1YjQe`FU0uwFVOU#b zm{(8QeVSVnio|?tc0dWCYTAvuA((&aKP58(iyPli9fj)uCo)Z!LBM0X}~s z*VcF_tK$Xu~&^Vs^UsD5aL#v&#Y_5@WkDt2H7rg>7!RrqCN@ zgles&43Y|&hOX&J3Wc5#wwMsY7BXbIr|W#S|JGd=^B$rzC>F~+`Rvm#fB)6-qbEnJ z<()fs-g)nXr(ZodCfuhbYE%@E?26p` z$;k=J1+5GxtG?>)DWunSz?imz!PAk`CNrfVO0$$GO?OXkd{)x*?rwLJ$Xj0;VfE~* z)wtTI2n(4TaI0*6Bh|3Q{1ssBG~<_0NUb&7FDLb=7b`9cOU!3OW`8~v0PVign>+H? ze}~O^tQ&hZpQ?J}-g7mMo1ASawda@^3w;xXX>?xDNiyvGOk+?&Iu#I*z}d-1*9l2o z=@qFLyiwBgTfK01?@ksJfOf5jjSbc{zY(E}MywLrKHnjPNy`MfuoWyUsI@%z^v>L! zmgpBuS92D{Vi*>)A!Ip`1-{Mz!59c>sHM-=q2TR^EL}Z6**nSDl%xpMFYOlB=4shY zPCxY<1|W6GCh^Phf-r_leuw(TS3>j3@hgfB-EcCOzHP?SyN9W7`nDNuhcqWjD>HB? zjM?C`HU~y#YC$xu+dgA%yFtEYaQQ7a+?u{{dt)SVb7anP%@oV2CUfoGn%3kzAyJAV zBHjDsi?H=YJd5ylE)K|+;IwAO=cC(g>^&0O-oCo^L_K@jcyjxT2tzsz=!F+BlQ3q_ zaW%|mr59{0lc(2U>`f)-IF4Xx=X)D{OBXJ5^(RFbpDu*J_N62Gebtr6ugx3Zg3St1 z$|PV3B_u{diXb8t+G-grWW?r)2)c8&zdwKLfKmnnTI}EX=|BAAC;#*R@yB2Nc04@( z@Mk~%(ZBo)2Y2&-|38-xAKv}xk9q$+|MFLV{^hTZA3U1BfA2s3{om`y-)E>`2oCmV z@4t2IaW5U?GRK}t>$!<8M~F;kw;-el(3$e}4WUddF!s(#NwNqDQXxrrIzvb}O zk6xMtz7WRjU>So@i4YQfHS*-J)vT*FEN1sUe2=GJtd3TLL(^6%VB@hZKmY1@wS4mH z->KGN?B(#J9_;nKVR?AG*WB!eIa&4lk6S{-@YU0NdQ`dtZN015V*G8kkGX^t2*Qaw z^r@A-n<0v#jIHN#r5x4UTG`shD3de^X(#ed)nIG=c)2`TjnnnuT5gr?<%ul)wSR*P zgn7QQNCIcten~&xFO-nZBLtQUWj9|qIS)pHs!Tcq9Ciq6<4Y;L?+c21EwwzJlzC21H2M0dB1rJXHgV_nP;n9M@V zYMs>*GFX;%qwBJ!zh$jaYh}5#EtPAp0}{3~vkpT@05Z-q`B({KXU}fu&gK^u68LLI zmwZkXeRG6$;S29<>w?Zla{S%8jEfti7b;s{+!NwOP1o|beE)q(mcU5Dd`*%Gquv=C zvx=oF0*D=G4WE3B53^?#@K-(p8ME? zTi>*g$O7amTS6Qlg+jKm1=;&Jj*@!s-qEJLC*2Vt1ry^qVs@UXb@H_Wt8r=UE3GRL z7$;=*F?*d2wN&iPgxEdR*9amyPHa!#`v&P{O+FOou(4&<%U4}X#S5{y0!w-|dkJG} zjfDdhsiPFC3aJ#M1myEUK`BKMQrx-sR@~XgrW;^3=Wdym*^n`xEmVh-akW@3pJsp3 zTecQQM@Ma34D)_?RET=#4)a-tB27!?LwWaZ&Z3{J`snDmyN{inAM0T6>F@tszj$=A8Z*j} zxj1RFr%NJudg7r(^7%SO zpLvnlB{rv+&}Q_uEv~_9ZY){i86{Lo$ZdKHd;{U=tpHc7Gx?=wGl=Uj_SP{0d}W%~ zwiBl?_;!ZvUj^o3_4_NrpqC}3`r3nM9FbK+3WR(~|NT_I+4@x4BX0RC5zKlSG3gSU zHd=aXnQ2>VfQS+iwzH?52GF&IQe{0GIn(PMmA$7u@?s<=TJP8fOoU=;0A}RH_-u%{ zjcpieiIP*oN1P5Ri05sVy#c~TuRYRXzWWHw+W0?^O?J|V0E$GEu!PU=G=akF={rR3XJ~=F{efrDa90%0;snp0)PaAi$8r#@X5$1;IZ%oNBr82*?luAlmgfSx>EFBfZpGSR*6;TDZl! zym95bj3MpT`)Wjqh8hY2nMB^r*$St7X6xILf&7hdD{HS0$stNak-GEDIYC53DIrDA zZV=*yF*!-3k!mf&Fl25_5)8IA31+UvHfuET&05P^8;7APo{>G7=-H#kx(}(2y>ZJG zNRlm7YpJymWV$<%6bemJ@Y1h16A%W|L$SZNchQ26#VC*k##FFr!gyI3K740yHgtIW|BW0kf!N6` z1eTHhD}?P??*=1FWwuv_g{fn2r^*(|yHkQB6KVCH1liqM?`fmgCu_g^%JvpX5s~TF zp=-@l%vnPk*Vsn9etd(T+vjPG>`02R5{V7=D^=85dvZ)oOWFzNP3U}7Uc?E zT^V9PrxU})xoYI?gWp=Z5Ma_C+lWbJQtD8p@Z}b0c5Vl+ z@VCV+zWfmj2}!cqq*iyY>YE^;y_Tmo(zRzrLkQcs#%G592yE+IeAZa#1tqt=cV}{q z+kr0*zwJF+Yvmb{j&wR5lI`r>C1N@t*+xQolQ#f~Kz6@{1-|4WE=VXvrjf`>xd^I99xBGvHRk$p%J|S z3<*B#w=CnKhOWJuJ_F-5=&vJ%B5^$ei@#lT;TzHXS8$7+Ez@>$TlE-nC=n0PrxC8^y{508HL zyWija>dUrTm6KIIc>L(+zm5;@mA4O=&*pVrX0vkl&Z?HtZJyweQAiRJQKV25Bq8I| zysQn=?M~00b(#+zy9rq|NIYp@Xq`3kJ_L2d6c0dfv|1N zw?|l9L@zoLdgmi-ky5i0R4P%Mm)H=y@dqI@9feI=t@WehlcSSm>uDQZJi@<0%GNi^ z7)SuVV1bFYhM<1+$Kcn94Ss!EQVQFduW$qUOajc(CFYa5ek81&8Ko4(EBu`sWSKRV zEyFpdU!T>d_PJ|^F&RE}hQG907$eI80cq04HYDU_Nk-eJ`$Mv{jeP6Mob5Fj8;ool z&ZmK9L!HeADUh(P`E?mgZhfhyBhB{u_N}}<{K5;mt*5|rBOS|u!f&fj+OkDq$ zx{#I7Hu)}eE|Mc&ANmef>BhFslZhp~5v$!087kYz%G{)+*>|;+U zge09RTA|r&7A0yez1OjKvq4vHOS45+SpCU-r@2KSsYH}goVL>lu~{XGBGT@e`86y2 zdhe3#OSX-q7nC1g{zuYu*OUEh^vvrWUgflnYn0Z`sxN0|Mihb2rP<(HJV}@!f`E_% z!U`0k*h0G0RIpe99U0AsVX*S6$L;Cy(c{C_(aGVHlksTPAO7Xu;mUnqy!HOOQA_*e z%j0F2-D-LJdmk*`f43bSA0Ixf=;ZN}$N$&Q$6v(!V1Kq)+}*$X{@z_ay}vj1=$Tf^ zDV2~yTAO{dX-tv~oGTkJLee(MZ>)%o3p7~IiBmH_Zyqc`HzZX|N9>ue)0t@xmYfTX36T|lZXG$FJo-)|Jy(E_TBgX@n_8= zwVtY+8g~2GLj)_v{i@&V#_$d375+}WLO^7Ifp_^sz%S|ovBzHY+$u})$ zYsj-T6ksEKnR&h?%AqT&waG48*Mjfx06NNYGjOv2%k=nwT!t6OI!O^A<5eQWBIyh_xy6*XI#GVQAb8X z7^lGY&UMDZAx<9)MJQ~9otVEZpQ|nB)zyP|= zLSNTZYJJDN^u240hSfnSD~S*>@kQut{h} zcQ<%Dvqz;AN$KoP+mdc#1}+VusoaUM&vLYf>h9OF@8hc_AguSgo*hOJXTIGDM z!zQ00fdrB-3pc_o((_g1G6}MR0v9PR9nc^&ppvYlhUod^%LmJ&Rhn2fwXAi%x7c6Y zyZ7nIDr@}c;~&m{^5dt!`@`=aJY6gzBF5#(yFdK?-GBXOzW6c^k5uYM@4efLA3S<^ z^yddpA3ZsV`Y_?i@%(tK-9}Ah2(+ofQwfzovV_h~XcW6u_hu)37?MCvPtqXdG+=x4 zxk!>U($~G#Te1zdAVdfN&6CA(8;`#HjsCy4XZ-NJ4`Xl9yzpSp(+=14*N&v0r^xkHa zilViCvK*IV>$dF+DTy~s*;?BHDgtRNu{@a1=c>r5JY4Cd&pK(c*_Mi_-d@p+p>(UV zyfb4lI2i4)>av%oD`h$)_~uT-i4oPgsC6+gL`StHhs(a~;-w;V#b)&@o$f8g=~3R+ zE3JSwDO40DEq5moL9opjTbG(dZiQ?}dbOpQ7yH_JA6KiXgTO#w*?>hSMl+D}l6%#6 z7qPcD%;&QZ*myw0(emWU$?>@>ytc^a=YDI_8H#lU>VP}O>#7ElEWJPXZ-wm)CaW({{KdiU7eEs!EikEXfVz}001zf{iiS2m&Ef2JN&r$oz)mIjO@(5}QPmfN_Gc7U9W7fw9s_0; zxy@C+YIIga>pSRWEC`c2eW=ev2eXPv#nI>|4MJAOh`4M957;Q>L@qVM+sI%Zw2 zx^2G0cz2^3qv=uG9%3MKroKN!EjdJMwVD{4-Oq$(wr|Gs+ta=RSGY6Ntds_0a=dc4 zAaB*dcL!hLu4M3J6J=!U)LP8WAgUy2r>;^m>;&yBPMO>jBEt4Kj$U)qIlMRR0J5Zr zsHH@SuF=@Yw$GPe;L|Ui$v(4|(zDyU8`{_rq0Hbk_xQ%M8zMd5Yg3?tMS%SL79+D8 zS&&6ah9l>y)?CaOibBI|zW34Hy0^D?_wCvKU7|ewzyIG)e)EUJhmYs)9vwe=%)#FG z{{A1L)<-}8`Qh=&-J|1^M~|P<-~IT*;otswSPb8L^7!cSq& zKe4{yl0?RAZ}$mxs&>&kOs&1VH%tekkXeW{jD2?>KTP^bvu77c3#2i74rZVPIb?)f zBp{c|jt*+k2sbf*?b_yeY3umS6i71~U=Tn<1Iay(n)=TA9^p3=Mm2>*(1q+~G!Q@@ z$Mh~wS);#S0c=GzI=?AYJd1uUh0V58nUc(T8^iIXBswI=RzIj5AwJir8kMwcKmHdUbK-Oe^9bqc|AmPw(!1 za_^vS((j_PtOCB&Z@+r;!nwt^qu6+~|MQP;uWb*XaXdO*xYV;IgzFiLkq9Co2>=B% zQ;-8mkQ6|IGn1~fo!#;Ed+KLR@qndqsv|-{v@z`@v&^))wyiGpuAOUNIonxU>~=dP z;qolC1Uwov_a1HD**d)aaChV3?qKN5nIK`H3@~G$<&=fg>sN2QaCLdE-4v~@gVDzy z-F@_EAbI0HwVx3%fwZ73$g7fzb8{EY&aa;BoL_D)&b7({;~Xkvl=~^(+dR0nzW4e4 ztNY#?;AG;&zEMGN z(nGL79#ZA_1K&OhY*2Kn~Q>V(;dai}Pg|rQ-fp zedpfV;b8EcA0wu~89Q(}09asHSz5SuVX3M@R(g0ay1ThEY?vg85RMs03lo7#hX6#V zD&vcnSI#f?hr-70;U~8?4n`96yz%v&f`}8NS547_U2?$%Qh*y60RirikNr917*~yE z4q}EFix~5lm}o4YBDp<9>dT3Hotw-(L6m36Ap#)nsR^3k?m6e8C{j)hAZdD=FtfxX z^ZDnPvgAJ|s7V-MQ~_~nU{wO0oN4hx_$K9YFCUUXmE9h ztHs03J70XUzqP%+f6!eyYcF0yQE+R0ZEfA^W@B$V&UO1QUBk+PO3UZ_U0v#m*c)Q| z0J{fxv^To-;PapSDnB}`KsOVf0%{WWjsK+>LnwkbZowqLAh;YQEGF8)+N?N;*bgB=tR1kRo*SED0dWBEEQi=|BACTv=gruW$c{ zckX|YlY3?s#PmyjGEeAKzOes692|0UDMXO4;HJbh4Dy|lR4U6|iFOm{Z+az-F=N=SId z;52PjI5KPpG=5Ja0TNDlJMLdv>b!n!<+W=oFRr#PETC$^0-yjE6asyEWBRE8;l*|cISrPTQUgyu=y!`Uj=ml=CV|Q)+(W8Mo03)Z=IT%Gp z9(9Yf+|%_-3oqTgeEGuSg|kJ!gHDCOV^#k~*g2Tn+?o62_R{;ewm!Z4=>Ep;$YOSZ zkWV5f0h&Z7*=1(5z??NxXVF+t#Itj(&zW2x-96IJFSP#tz4P-Ocn0lg5ZQnr1cL>b z0f1*59%6S7>koJDZSH=)_GoKwRKxUJM^Pm;&C%9;3R*-AR>sg$hMHlC!=l98e5>1C z-rcEpc6P^}0wN~-C&48N{Q5atg5x%C(<}VtOPxP?Yhk&I+j|&^4>mqOY$TIY0UE~_ z;ki68$D5L=ZrkAc*~Pzn=S;7U-3GP!M-Lt(kjRitLXeqy2PlWsM!GT=|Khc?7Z?@b!fO!8_XYJnlaO4h%goy7fo@1h*W3elnuZ1l*Oy3{MG-8RnNN6Y&q?#Av}%W@ zLWE;_VC1-8q9@Cp8At@$t>vqii>p`B??Hy5f`JeuxWl!0?o8|TmmmD%Z9=fCxg6O#h{64Az|ZQ zzBuDaPB+4 z3$I>6D3FJ~wYGozG3F&G;?ma7lw16gBJ?`{x!8SLQnz*ywJTc8EW02)}0ImeH*TE0aN#GvynlCN52tU#3i`cuPY9meCMLiP%QQm@%2&^pfA;RhAHH_= z#u+SD0Dz2~-E{zrs0ym)%JSR-t}WoD^Zgeu_W$`;%m4I?+aKQ9t(#1EDVP95U@VpU zMYB4Wug}L!pgZctMxwjO@p2{rgB9@9uIz>LDtx?cY08lba%NJ%xI(_fBN?51cC(% z!E-~1av*h!X0?a2J&*`9*6Q5v+=AHMtII7UtvXr875Pyi^u*KmO6Z{oP$3A1>pLE##1Xi*BiA zc*p_BWK-&Jq1e6HUtZ~AM6}DoFd0CejA|Rc?-+Y3Q-q@Gw(l*M2bbs7YB30{vNo&f z#-mC)dBOo6piEbl>jgi!(#Dk%4-4Gxq!Oc(CV%u@?Or6y4pk1H?%b7n28~l-j0qWx7+lLkd z%)mUQHHmMrE$eidEhoa+)90{&48lUIH_xIqXuR>q(I8XCsL45vhNC7mEFm8k$BCZDxihhlw8fA24(@o<@$vB>3$1U#f}L zym)ot@BaMi3pZC-;NG1oxCA43j2itUGmF!S?YB9Gd6HV_DfOs0Y4-r43pwiG!ea54 zZ(jTRcdlGKgBEc(#O79Be{`^YFig#%rx6*;vc0g-UtH)fETZ4SYgf>3FILf-(T8`| zcZZpjNtOi!b19=1YR?rTOfXVk!i>OJI}kZ)w_WBpZY=-VyVqa8F@IqhEn)=R+YVb> zzPY)#e^9#{K_C}}7%@xd$Vlw>y&G?+wpO)n2~l|($Kw?DnzpIhvf zEye;t5*#pzlN?j|?%vL5p>zH6HC5%HX#maj`*|?>{Pq_y4;Ir1iHHV}M@K1`9Xp3x zKmO%|pS@oV8kS`iWdJ8bw8mgZj>kF@meoWzlW>W>daJgD?pk3{LaW@|h3z~>b zm|#Ldy66DPfL4jJKoJ$p3FjoO7Xh839YuVt+rFMw58?F7Y1$xq?8#fTs;b=<>EoB~ zD5WMPxo3BOLhMlh$Gvh8-_y)i1_nHa3l3y52YTT!AUNovj;lQaP8jMU3|5w_AG~w< zFW9lJ%gyz}c{c!!07(4Sv|TbA>Y!1?D5%wf6*R&fApYPX zIlly+F^JrX`TA=APu@P?E3tKe%7;;K5#&^|eWEYCrmTM)-xxYMxz*{Vcp!j)yP9rQ zzwy$_U%qki<}xAdJjy@&c;n}vKD@KOyM5@C*Ta$Eoo@Tgd~4-Q@0FX&ue@+}v1czV zBfUDmy?t?bc(A^k3{eP~6cm`9jo1iHfx&2xG{{Pn?tgh{B@83Oi z2VtNAI^t3YDJ4N67)el^(JL4el95KkR@pB@9HIt9IMEQ9LLf4*{%G)@|HlUpw+Gp1 zgb8NMIh&O@y@ zO7k26G{Q_0viI6}=~DNNYv>1TIUFuvqO16ov!W#T)9Vi%vkeSD304{tj+uZA226^2 zlhp3}3mHVf0f1BtDk935i^@?66;QE>o)V(N#>C8%U6V6H%`&qFV8}p-ny}mcus}>9 z!y1|sKz5i52^eIrlVVxKBBX?zrzdZY*-Gh4x99%b_ySXsLU@kv z2#)txCW);`1L4Po^hnM|wv(XWm1*|O&wYtsB7^|I$7u(?79;^3XZA_Lqy#TLUz)%X z(~219fK&A?&AyyPf^sPPWF&(^6b$MG>_6Pz_~^6P?o<~qzzlHs)UglN&FJ3#_T0tg z;>{P)@0BS5LaP<`_jB64Kiu!t!=hV($4_Juqh@sL-saCgZ0#KOWP@lm07zysQ!oXC zkt8KRkV81~P37YfjtG|szzjz+C@zSkTmmGQ0|H=tg3NN#!r`pDD1q^emLIwP0E9!? z9g>Whgd}-09NoEfyYzZdqoqri@Y2<#Zl_IK|LnGn8cCTb0CT52x&$adkqub|Iwo}M zv4@C`F?>%iJo}V+Wj(nu@MMt>PTnHK7<>I*SyiK^0qBBfha8Yxf&@+1VkKa*SiY}4 zTTs(Vs}ng$bHD%}E4?Ht(QTabIKG@qUv9nk_O-u!@8-=*=ot16@!_Y3|Med}|F;i! zwhn^>uHS(YsK$K3hL%tWfuI1f2J$Jt_N*Avw~2C|g;ui0$B#?otoWgRSMp|riil1t zbW2u16oOzd!XSaOt>o!tLek8F1b5*X;MjU3__W#=MR{bVnu|-Vw{D(0cdmqCeGmWp zmuvs?UqAWm!EZ#6zjEHKQ_Ajf=dioBKljDP-3Rwme&y{~SNmxw-+}wNbjjLDBwO0!K@Rk0@_paaH{?*>@(2QUK0MJD4IOp6nO)LtI zn%T9-ah@TlX&QGg%Q6l}IXfZ4_{#tduzAq@_|x^Zou)AuShx&K$<4Bbrqg+J?)?15 z`oLeh@XC$zE#h1cH_y-AzrS&JC&5bhNku=8Mx&-lIcF&l5O})j8JVewSw8aIMCMlT zPCx|&9m$xu++ZQbSj7Al+j-3UgHCA_D?mp=M zAa|SQ-#)qb=@%(i-A5zs)e=aE&^O%RdGw%XS)h3ig2u7cr%ozC=v1#!mm4jFAjmbz zhlFR07$ePc@|2QGc!qcmabiIrO2-Ljfc&xiKOVoE0@7G#vMcW2S=(r}FSOcW#w}f@Afg*Mj6lwkU*l>zWn-22ZO`Vt`H17#-_DV9^HO$`$xZMHn+PW zpfV#PnNzZ87HPpqkYl$*iJ&CqOe65vyFfJnm{|y6TChg|feB|ppyV9;D@)?>X343# z89XtnP|8_S2n?W%g6ND;uAAL^4;oVo*13KKm(I6|`GEZ|?&X6)kN|q1NDGwF2|_$- za*{BRkP9RQ{mvtP0RVVP3Iu7r-dwxg**`d#mQRfBKzV(B*ul(f?Z@hdN9ikrK<4+F%^FMs?ufKZq;GpMnlgl8I1o^Et z?o2d5g^*Z?!9WLq8qBA^lLSuTw<&;6gr$=P-N|3@w9xF0=S*g6#@skV5WtzVmgMPg zBLqU4Dfkl(Zwm7QAP|KBaJIrY88di7$ecz72`Bb!$?*!(Zx5^WRPlvlZke-Pz+=M* z0jZ3B`Aq-jN~cv}0DQQC|N6_dAKxDC<@pMXP#?#ovf z<}18(skpIn=8L=aezMFMy(qw_5Hv|^jyGSX4HcB%d};NaR~FBA1=MTX{69bX{6GE6 z{kyx>emz$^=?$5H3xfVE=wcgLBIZOY_xY3oo4Sd^p#>w{z$YIRH64r#u`EA1jo=PT5%^B`KH}G5N?n zHvy?+DusdwnwB61$zT>j$jaGW(7!XG1fa~^u&J$aC|etFQaG8eqZ8-iL_puN>8E4M zde+L67(=r)9FmXkJ2}f`!33xGl|7!5ijWW>g9KzSlZmM5wV?n6Km}9)1?aO@7ssSi zks^ixBf=p}GTUN`sUhklCu)8JiHL9TZvz-mUvK*IQ_poyCP?iX#BKt@AgpqbMeVjBtm97&{lvRilx`_@rO4;AX{6Bt0fC|spwS4yU4#peXrvi9ZgGSl>4~F=f=FXqiH6RfOM2!>sz~t9nY1gJuob`yXC^;WApp+5vv{(e8i$hYx@D z{`&Tyas@$77bQppgGD7YgGQDt8loX!1`Y@cs)HI>fW;BY24#Q>H31BwiOLe87@QKD z(o`C%GDJhBIjkAwR4WTW03}KAabRg;7BZZ)=$-+OO09ra8CpdMQJr>^fspIe02QFcPE`eySZC~waw4H@bRtX22!NRcuVcvVDzu8&X@z#{ zFjr>Dcbi&`XhG)5-9kC0yEq_$LJTyzn^GB71e9hpDg-Bn^{7c~QiM!$KnR*Bt_8D@ zv#Acs0SaIcBAUt?5QIy}LXr?H$wdbkLa;onLTXoKw-s9Ll#$Kr5t}Aknuvvw}h1oXZxLpgWW+C zZOn59il_J*Q*de2v---}^4GoJ^Gn?T?2pQS`Nf@o{^k1pjl&utAQq711c9l+6?!03 zit8H(KmO^h7gx?NzIXjhofW}K`_t_Du}|sG$>V6p(=un8W(BMNjsVx ztn0dpMXN09Q3Dwi9Z0M*DgerW6kRRC!~jw-N}#iF*xoST9W_7yWb5YD?WOti(a`Js z`kJrn{aOifS9ZBFQf|A~IW-J9h9(v}@DRLRR*}}IhK-`- z!ZR}vQ8@Y>^g=U)lAB0a5STqXTmnc_WAw5>Su(;bs}UL@xz!n-5G=~zo(xXF6iqEt zX>b#wmB9*thGD<5Hmw3_A!V(BgE(^GD1cPWfN^unlv^0VkSGU@Rv^~}JQ`aultmELt#12u;&2LkNhaWqAgMO{(i`EJ!1fQWHX{V-*Go&khnm zlHXy$MPUqnOD9_a(5w$N1?YHjhX5%Ogz)$u6HMD?8!ZHj1pub>%#+BmPQh2^LOHwS z08i@mh(|S2KVH!#A%HX2Gsc9BbR_PfCx1B+{N`W&cTf1|<15gNoPN{f{WjHlFg@wr znt=+`{vdsFM~Ha+C7bWU2u}4z%P@YSPsS<*5Qfkyk2$PLuz-gQ zHMi8F8I1;(E`9v{y^nEuW{1R5s~L(Z86v{n?DZ(2v(?=i&Ft#lZ|{@;Lh7vg6-0S> zy4x{3c8)KG12>L+hB(I7axr9CuZ_DsLljyQ7}7$RtknZ!APN$A?^OWsbZ;Xj;)y82 zwAiOpfft>xJo?r>N?*UtxT$iH@~rYF3Yopu`A;S#fWj~;mn&?n<~i|9m2A;=S6{8H zPP6QZ(!W*+mgwJ8&30CeSbWPV(ah=5&fawp3E)GKkY%%B{^Ca)&q&BOwkaUAy7}*J zpZ}kK^W>|$r-MNNiU9)>B|lw;1Z!=hSgb__7sNepx~;*upTD|bYD}XPHOm$z%#@G% z`s|7&i`845B3lLl$VEL|6o)I-;?{CwxlxWn!f#f)H!SQ2IE{rKq3%@LJbHC z59CS~DylvVc(qpESgNcq)vLvbS%Q$q;@X4py(hhcv+*d+edbANE{6^XX&$j%S*pcJ z%+{XwJICjzNs?@?EWNR{T#Z9j)9Pnm?Vs(RwbC?gG%8D#L{$12j!(`x&kG4}tk<^I zml}m)F%H;z;qcKxZ||Ucyx&f9mmwqOSq)Q&1mqR08mLeJu9r(o)o`^@Ua3`K$Xn|e z{qgAJw10Z)TJ6DjJP-&QQg~GtFc*`kSy@>s*E0_=(tTd}Pu5JxdGwyu ze>N0K-oX=dl*Elfu-OPUm!C~@&Biv9tlv*N=Y!LW(Rn*-^;0JVb<7H=&JRLmCPQU}L&YnmR1;NFVJHY0D5vt_P#&LVCml2*40EVvfl-T0=f?vUlwcNW;L}|m zs*;uB!TWElU)`$21OoOC`@i~l@4<=YRD!S}PqQYKMPy452o#ai@}s@({U;Y!H+6plu@W8qN1wshI~W-aaJru zwOZxbRj*bGacEf?!m8?M*d3p@M;G0!Js6CA4${zSWQj!zNwBh1-l=7^VqyRY62|UQ zl8;x~nRJG()z1~=ykvPUI+3O3LSpKbaJgAnUM>_Vw3!avAFfs}KRWAoI(=~j6@s5h zFT+SVSgKg7u%k3@_0!{4|M;vu80S#03*Ax+N_8}!ZKG3J zFtS`PC*?vBWSrA(HyaKeP2K7tv*K`Q>{7W7 zSuPgCFwf+4khcdTzyoj!0a?pQKuhIfy%aZUwPKQ}N)&~8nss`^)7GHXPg~uQcnK$r zFoK^+L>ZP^rMO&3oJuRrTD`tC!B(?+ZLOR{cIf1ymmXaVS|>e%Q^@o89lXK=UZIn1 zSe03Wav@kM6=Ex-PEFG4YrEIaohQxbUJ?Rigq)|1EkuqW5K%Ff!@$^DVw;uHQl(NU z7nn6?WQ6fJy|`$1`+0xlN263g1O??nR)`gzCmrCJMIVdJ!7;sJ| zC*w$a

bc-9$q1}^m$A_O5!b|@cc1`SEs8&8xM;%L3uSg96DAy4XD-BeCGgQK%^ zFHRNko?t=|(!ir6I#eN-OJF4mLWpFUcV2|Z6DGR_DNnW{YlB9kQHtY`x!3C*_l8cN z6PC-R;%Xr@-Vbuq9_BeA4x^=NWxZA{mIE7;XGy(3>-1Zl(Qx3#!;FCEB2)T;q>X-m zxEA_f9oW8JS@bnSXVP%`=Q`QG+!#m;9T!TvFvm2%=A~VL7U9bCXmuJM$?L2p{|)C> zRp(^qJkKY6DRVUUQiH^l9|E1@qPZ#Nl%@1v72M8io0w_{(o4w))qjnkmA=@F{KuF5 zcLuh!;8*wJM9mh7nII^64o_I8`oo@J1!Yll((&0tUoM)Pml1`|i=9hr6KMX&tAM^a zGn3DVcWeU;)6VVty{{gY+Cx!*#LZ$fA~Kd$Cs`Ra3_sy{F#xXsh(!o9QGA;GRt5lB zZ1Q|TY?KIOm_Sf_n8T3QF2^q!5VmxQA=}VJF^B4gTkfFouxYbLOa$t z#VAGYrrD6t_+o<4fEDEsrSNT3UcqZrAvj=~Wd{iqq!FhZ)!^svZvFMoc6WCJ0Eaz%`eo-IfAiqZ-IH+| zFh?RZ_2trx;h|#S%htbh{nAg~+BSvYu$}+&FYkVJxAm$?$%>(QCY1WqcXpyu*zU=% zK06$Jb>c?ZLQF|!DjqAFRV>%4*Dr5g+g)xpU@bhs6I$n$r%&p4@0~t;(mFWp^;2Rt z$^eWi1XhR)t3lSN^3|>CJ6G3lY%i}Zacsd1GDtmoV;r2y?fb{KAD%uw?DmGL8Zf{h zffe;}n7;qc+Vv|d*5Ka5*6)6Qe(g&AC+}Umz7-Zic;I4)tvj{<_dovn{G|KAjcf0} zyJHv+_XmIXcZcO9yt=#g-rGCdYlU)zBt)JA2FDlm{gd&>UmV?ebo%5Zb!uh}SPGES z5Gu&~mBj2c3-7+MzO!E2SV>lvAcTzI2$?UQpEr;9v(LWT`|{EFtS=cNRm+qc8t<&c z|KWpcTUB>uEsBWXP_p>ZwOV7T)*0jIBLC;#e%=|4g^X9SL?wgOdT@2SbbYsZX|uM{ z2rC7!g+rW<(d#ZBo{a82I{o~?>C^MHJp%a_*HS8LTO z!Vt_*fme)EoL~6IPy2VCp5NWOIO`2Zv+jnu%KTi8Pl1KL0#XQr(`GZ_Nq2xg^H zUa7>Vt+AVcbLAH$g%l(N&BT5A^wjGx~Y7|~|hQ`0CGB8CO4tg;feEO(UuAd9sqYFHI+!irJ zhG{1IjPYij|ChhK)GS4$JQDi*-}KJAnhRbxCWK^{=V2VgQJhLfq6QSI4FOeHC7vlu zC`3WJp#RFMAZ5)h@+hDQxETZebeMbbBn$z_DUo=iu9CQoX5s2~^V;r8tx~L3Z5$$) zxq&)|;h=ia#r|>s-qVvuhpqnD!xY4O=Ir{`@=xEovRcZv>S8PeTCdSh-deeId8zB^ z&Xco`Za*2PnfK(p1EHSKdM&=XUA??fY&1)?rKG^H9;w4P$5|hbk4E<&KU0f1@ANbA z7VuCHWS}9S04ylRXsePht(2}^SzBKzEtOHQP!E@*H^R}W`~3dV-Tm{U^Zw9JfbDBT zB2NY&f(zlVuQq>tYr9ZP25J7gPal2y)&5urWG9tX5JROFoA<7)-@Lq3EO17A`*H6# zpFKJ6ITb@>00BI!1W4c5Z2aivr6h_orN961QFqi+PYNQYfMjQ>xclt>H)>m}wNimg z1z3h9Wdo*W;Y^+u_8V1a_b#Bg~rkT-Yg@4vB2O!uCie{uWH##-au z8<%cu+9*Woa5lu3Pvyf8$Gy`rshM;u@U*_b9%gD$14=M%hVI>~n{QrjmP%%n%h|wx zc=xb7I0pnT4OX$p1?*uMAhW0q7E|Fwqqer6B~X7k62^8X$!CjPZg^!pzR^=~GUZ)tT=h2}0Bx7)ot z_ctmfT3@ktYPTe_R{t4Lei2#H0%G*QFw5^fKD~WkdP7U|T#o?ElU->l_9Uc|yPU1T zl$fLolm*sB*X_BMF)8#VX03t%W`eTla}jva1z9K%^pCcTU>XDR)F+(@V89Tn1r%eD zdG2d4N--KE_wypQA%*IcwZt40$|x0J7~-ctpA`%+l;#48OavxokY%1V3F{vQ33NeD zRsU`JY$s!CsKiV&MXp8s$;3%gh~uQ+@2SoVcV8Uzy^|;Yy(vf$y==as{yDORo8`6} zCHsrFcmC#s-D^7$1E+)ZquZUo|IPlVch3hSLlgr6hyYLO2sw!(NI+T9>XpspXYT|i z!NcR=tIz2*laeK@n@f2AR#>g!q|Jw?!I$^OJxM{)0z4ohl*+{qo)JNpHZSc|Dl17qYxr#`Vqk%B4%c`gHyO z{{Q{>xHWdOo()mJr>`cw^_v-+t2k zm(LHLoaWC)WZ2Mft!{pDvvj?Qfc)ex1H$EHY^+I4GcDu8{6AEJ4?U({OqGU`zJ#Ri7mXSI--JcTPwxCdUxj+ zZ>?>uAql6+H808}fn~s_EB3wJ>YbzNKYSGZ>eKssot%^iJSU~hbKeS`15_baHDK#B zZ}(cA{`pFy6f-^hb#GtW?T(B;&-RZ!HBMT(*VK8X6ItKug8o zYK#pwJwZ;D`v>G&F3X(vc~VH;yu9@DH`Z=$#T(U}0(cA=#;Tp1^4rV7rHy1~bGPzE z<Z<3U#(DZ?4f0jrXg0)FRm>1S`**dQgu zwtVklaM2ePVG|GpaD*(>{Ee;h&u%6r#7Pgo{A8?}8M00hgh8{G{NkNU@9Y#et7sGf zMaW%~v@%D=vU za(vczDH*7N2(c(BTolNq)$#||*KVv8H_AvtHY6bS0zav!6F5Oy!S!1G-b(#9_f9{$ zyFVI%2?{tEA!~G02yt`s8RlQRw$W@9UY zqKIUfvo^EVSZkeksvZl@RmLz>qX0XN=-o{a!Q^3VWD>#}XbEzyUU_H77DH4ErL>)_ zt*q{@FK-l3i=e>Rt2MA7ZE(L9H7ilIT>s+E{o_{0s{#GcXZw0;@%K_eR?we`s8q!3 zf6a$Qvws)rVx-WQWH_dnMWBR#H{;(#qgnrJ^5=;NO7waQyE$}`JRj^GMx|mi46)V( zlUJEjfM1}jZ^~Ht)C=sTjk{-?zGk6-- zsQ0Q+0USdk=0ad10|Oucyz@>}L74nxGc!LV;z`r7B~oTp)#(IaW?}tjjn};2Ri{65 zBF+V7=vV*d|*Ch*M8N#PiSn0sqW@n5~Q^M$LPhy@eFTlf&Z93TNU8hPpc?KiF!jX`GsC!iS4;PhNOogphnBn5*Ku(OQ+_=_v6 zYfH@~bVfKn9H!$;Townqx?EXaPO28W>)7FKiz$D*j5+>rZX7b-|NcA zG)s6kOP-JkPP;hnjE55Monf3YBeKwh`n&buKmBCq7w@lKT89GN6s?PKzc(I_Qtzdh z@XB(jUN2r=r&6sL6^m)sAANP2d7~7urU~3yMSt>U{ck>4yVOJ`IBetWtUns2Sw1!f z#Zu5{l-3${wS@R8lr}rvUhnqFC>5sxlaeaYRQX~ix1h6B80q~Qz4@mNyZJ;ve5lvK-y~2lq zS*sP^xv{yrT+D%XhR+|JJvnJB(d6$U*1%AN(dE_3fBfL``&W}%3Ax9UGrhRD$g<3P zPs~9W6$?>gsZ=jur-H&Yh~Ma^>D^Q3gwr%QKTjV#=+w*FsGC|X0g$FRZRhP2T}S7q z1NguaoXAq>-q7>MZg3rY00`OAi!4DMYqW7eY88H{wQ_gl(iPw^-8Hy4X(#Xij^p~zyJL4ljFV@BCse0paP{k z_2d_Cto`(AqZC2#hpntP7!6Y$jD<{SY*b6IS`EtuDi;vN5Ls@u+ui-1S1&|XnU#vx zUEe7G{MPoHJJlpa3Y_%N?)1l_5r8C#>-Ac>gg3&a*tsy6c16?FRZC_bzmkqqN}?dJ zs)6Q8B8zWMh7XV>G_FulfDn$=E8xLOEaYq;iA5RMESc!qR=rk>BaCvKo}xb|F4s^g8cA0MgI+700H;7$zq<#i{yXrn`T=$8LZqwTxAD(}8f&cukb5Vh zpx1WN>A%OCEui}6IsE@?`*Ig$))&9#8_$#gQHb|Yu(1TA-;Qb*y(z}o#p8V{Buj?T zs2S$@A+%5lHK6&!W^*EV+=<%%@xlN>g1D zorxGvviTrpQPCM2R#pF)^D~JG5zQ-;uSa6~!fet+zE+s0(-EqeTrfyO17MM%;-G-@ z7^gPNk$WiMg}hqtRVO`;j7Uw+h&{|C))Om!_NW+Ov_~e*?F`8ilTOOF(^jo%Y^Z;P z{!ZVjFssvxcse@^tXXDe0}#y}92a$1<2Wu9i_Gkt|8H-#_3sqf&c#s#0Ogj)YfI*@ zKiK`Besb;Bsv!%>aj>8N)35hFy3;<->tsvt*-W`irtz~fAz+OSGE&EG6iS{TDCjWI z{B^(AC<4u5873%1k{dTBkyi*00>u)j5cGT5XSds*-99@!X%9y(iCm+~@4UJE!F#)F zwbJ$qe(|GXyInu;wniiMBwp00tHkE#Z*TpV57w@4AO!Z>`0VcKm$x4t9-ccVhWQ!D z+*+;N+$`U`w7OhR-n)u4z1kmk?>;!s$xfWeSRA=g;OhpmR?9Y*Ypi(iAiekS;UFC+ zwMtZ8{_@WGpr6M?M)L^57&Z!Fe7))k4<4Rvn$PJeP!0+{?&)~ z$G$ksle4p-p*Z4ky?_tiSo-VtYMW~y;J7vT{Hted+fJvGbz8%HO`8V5OAXumm?S;2TnWr`G_+O1U&K7o{%=b(Md9EY$XN(3wQcS*;1>(L|--!|*N1^wjVUn5pSr`UM6vbgqHh50D zR$-DTjH1vq>XlpD<)6N@SuLTPgOPvWXYdFNX7-K)t)2}6h5 zkMfT{yM1(gJkFddDVZ=#>Q%aax&Hpm?TuQg8R5-c^YFNK@BB2))H`$k-jSR<-CT`- z^5d(OYMBYG&hW!e?msxV=((WpqI7J@6N}qzCjar)%8z$Y4bUHB@2LOL{i8ekz1}Ey z2omkASNX>`ci+9b(JTk$CN6HSKIx4gKOAL#(F#Qq!bSx{hlBmzod?Hz$GvVp%`>S- z{>pOcS-oy+tyCzKAPsZ3)ovXh^Tuux>UNcPo0SJo z+nMOJQ(QH%w7Xv0Xjp~BdrHqn^Zy0dRgg>h*H(Eo)XO z>R;Ym`Hyc_mrGFKe1H$XdU|`mb#T#7GZzOY(DCK9`j2jGy>Vrw!AMpxcx$UQ=y&g( zq+avfCxF!~(s2YZ#_MHWDwRC>z5UMqUT4rBmI{Hbu0PltcDsE51o`?hG^Z%457ajj zdgs#e&)!;3J5wLbIy z=&b+cgOevGz4L)&-oxldqVH@}u3gz$Uans%QgF4P*;-f{QyyN~uy+TD>1ov}d}1mRkky|G%nb!DU3C@$OR#xfqPRUfrR-C^b@et~4x zo^A5iSDtP7jnH>t?w-iq2agU<&PKx#5lV?&U8>%?w6(ijE5w%Nx$F-^pgOCWz>ClG zoH?|C4ZL@v0!)Qg#50^##4?Of6d=$s!zqlLsAI7zG9w5X%}|`}5jDOR5;P=CoCM>oke*Yr~0#)^@ zs($7-@+z$?)$iU%Jw-nM?T7xEw)*-fK-|Sv+d1N+S4i5cE@Ufc;rqP^M2U<-RG`AI z1JCnvE`t+L+8rF-dzh)#*HSMj5j;%##p= znMhfuq+zgSOYkzk_sJ^(%^N=F!c{Edp#r>?j88-~y$hP6-&czpz4nbogejJ0uRQtK z=e^$W_FXANnIXrN&F+mRz2muaS5@~ItAnlee#H+s8+4U zNiys^{CVRCWNUyxz<^bWI9qK5x2~@KhYv38tQjIeA%Z+h2ZLUgjX*^x&otWR`t=qF z01^g4U+^?|JqbslofOvutG3G%&@ zi*auL=BID0Rk5*wcW$lR**|VQ9jBrxq>Rz!wZ<>r-nz1a1UT#C*B>4H-EZ&h9k%;p zg|ant*cm=ON*_Kr?vKrTKiW>BXE-xDI^R4wZ=bY1!9tlP;6z~!m+Iv~hJX63|H}{W z-@E^GoH-k#TwCf7!mhIv=c(~IaR4JG0O;=h-v8tO``MjG?am0p)Eh#GZ13o?Q!dq4 zmWm|6rEP4k7LQtOMGUY}U$vuOy#2-pw`xg%f#9pX{{Q<|k3RqEsMqu347tLv7!2+o z4f|sM!(UWamUwFkKYypT|D^oIqt@8*Vc&KJ-Q_Uf+*)=30nTIpWc2C%ek!?W&wCqa zR3LYKYw71dy1B6$fv|s}zx?gqho3#^rx<7GwMO|!tjV4+Y4^@$+^-tQP)9(1>ywmOuhvR(gkb9P?f}9X}nyCZ4u4+qD(Pk|sLA{)mA{b!o z&>Li<+<6!!)w#<5RC1U=&5g>?816xEQ)$escPG zbOCTLHIGjQHY^qkIv$PzeYMjKNI>3_XuqFn0wqy(AftIi`Oa$Ty&LPz5+FD@@&Di7 z-2Lp%)8S|&WR%0qJFqVY+1_c7eOUid92aakL9-qahFLyVN{)JIw>#)$!z)+U05H%S zyZgtzXG>))#h_5*(5f33!~E**>WwS4C`QlY{?YhXzkl?_gY)AqjT~}vfsCHUt~a<( z&=20Zl*HKFq^;x9@gYV76=7%yoo>0g;?9%aKYw)p_LHNtLGChw+5+j_qj5J){|CFX zx>XJh)|Rnc4-Zc@%@r>VM-YY;F*=D8gHFZ|Pe*SK5r7BiM4-d!VtPn;~rGK8<{wOP0ujCwwgORxvHu;qzzrEA^Pw!pXUS$KE z^!$fk9{lF^;e!k5rAh$ns|zR|oKZlaP~B2u-`ahaSgb~Y8{~B7EdAw2PriD5-Wg-$ z13-z~kr^L$2ge!y!q}xoWjjK)Q}0}~e&6q&r-no#!dfMIbE~;riXr}}<9~Cf^~s&~ zX)hfKXJjnrhTT4fr|p5a$5qxj%wtm`OmHN0N-vvh#DI0i%_ZnO-gtd+7-~W4#b0y>^2k3q zBxVoNnGLr0;zdPxhTY zXs*?+T`n&b@b=B6fBB?v)E$v0)+rZMX3k~D9sJ$L4}bmDMQ>2bbJcD?xabjzs$tBd zjD1e<2!yEB$3Oq}{%<}zJs-tBs40i)U4gypwtw^K;hVQw%{LnsSgvDhz5Lq{UD`c5&whVUtd{@Rzr0*D^u`u$ zU1>hrZ;dq`*^2kQBIjd`b%JaKnHC1JJoXGtt!kjI8uQJauL3fI=5!{O6X2M2sk~)-b7O_2Jk5 zS5i-uF*|TzLiL)Zxf8F`j%MpoyclNx&tE=>4463+9%Qhkr8werIC@Pd)#N`G)!}Fq zagc-@havgtO>NciKmK$*6NJG6KZd|SL`Xs~Ar9dMM=kl{_LDE~9Cb#akX1In>= z1hUaAu2pRej0fX;kIp}Ncz!$#orG#bk1zv(VC+WjcyRk+>&I6&*9z7sBAYWt0`Djs zP-#rIspHha3#@^ds85BSmK;?FE>_d?0}QO_``coe{<*bykkpk6X7wJ;9wvWUA9CU&m|LJCzlm`=tIP6p?Fzgn}A#d5h^WQx?M z2mlnUU#msSO9jAt&9Xs14!un(TWhs?B|Pqp*eHuvmljL9zFM^*Jg|R`r^l_#E07hE z?B?~QwI&QO0&YL)|MMqL_uDdb1r4Iqrw$^PN7-m}|0FP%=JwtiS+<098u8bjSSbPY*tOaMntpNbK06KXM!x3V@-)dlC-_O=Z0c7H?nP_?x%3 z-`QX`IPSPF?j8R2i~WaZS<5Fm$iVSyqFo`*IS5rFbs=R&~Bnht#btUVszJ*bBC^LK7m3*k-$*H+^P`+PL^q54w58@2LAHHsNJ&YzW< zKY2WS(og!Rq-;{M3}?irN?9jcJ|0}(_Df}iKoTI(jKrz1QQs_Sl(GaUm~b)1hmQ|`w|9CvGMOoQYoKo6O`46L73wc<_f|{w(4rEf zQV0!ZW7bAQKY-i+1|>oI?nFUjjI{xgrfKTcec#AdCumA_@?y~ld2xt{;c2fSW?rN- zUM$gze#?q~r{m8TNv1_SqOZVb)A}hfjWqX_izp(^Ln#jCh9yz&P>WAL^C@j60ZOgt0(v=eOy>Pi|Q*nYzcpk zXdk_dhjmuAV=yx-k)JZmUvRrB6p|>8iCMk;IqUleWIHz$ty=T;cH`1!+-!!%;Cx8K z0WFnL3bE3_&)-`==?u?5IqG!BI{5~gt~0>To)?2WpwC$be}Z81?2WqcYVsygP`8KA z?i}3PAC6oAGNfuC1nUtGef03C`}zHY%e(cpMp*kp<+f0{=Q|gf7j9I{l}q(v1;TjR z!mn?iJUHm4E+CN6TvedXX*3qfz5U|B@w4RprR}n{SYIk^EZ6Qn9Om9BIfY>6!#xC^ z)vF%wb%rA%SSC?Tyi-6e8H0c&Gzx}Q;OWu$)4ONgEKv?EGZ_L~fStq#r>FNGoPO|T z13+X^tCc2EG~kE^yO&m$*UK3&Bz*a}_wkoU-A-Ul49b~dVOZI4NIp5g=>Pu1`#*i7 zxxQQ}SMjVSR{iKn_dFxwkW4_1Rp;-piS@QX$yo1(meYDYxpt{oDI*g)YWv^cy?Aor zg^SMfV4QgvHijUMsXOwIpLU-QbkB?eemS@mhooK2E0-G;23TrF1 zRwqxh+)afcBOyNSW`Fn5`R9*PMmEfXR$o1_RiRl|Ju}ZYC!O{+I>0N0kp{>Okx1d- zV$lDz_2ewOzT3FEovk+FY9%TaBg14Z%FnJCjWRZyNVm8rL{|&^(36O=Y=8$;z>m(+yuiW z=%w<>y`u+v=WJslB~6w5AfwK}TNVP844JSP3A2gAIL|UseFY(T=KCiqX`W`3Px1HH z!^Eh?0ALn3=GkJfbJawk5OTc`tW=ZMSX*P4!AuHGUQO=l_`-dBzr9y34Ew{69<>G| z5XAzoo=7AB5!8rR|2SNauYzXa)K0D|aZy#^3*3a_zU_Z~YKs5toCe=#=;O_D0 z?m@pb4hcaKU}Glw6yoTtmp-`|-ORNJ3K*q`0|r4~GKfPcm_!>21(7JF8XdK>v(7MG ztr&}D)l@5`{q6v8%wjleEH%pI5@3$oI6EBeu0@p?D@(k&TD@~P^y*o&fb+FRX`@lG z1|#6bDiN}kY_7E0z~4Bh*4q+BO0z$!>vZC5>N&_ZRO9VeUzzT zvf@N!QeMdv#{>WQL3ek1P_IS-;0ZD>T)ap{bk3!D76sPEksFUyWWi;R7%+f5FNW!) zW=nI1A7KnUzHoPs+7}rM*pjy@B3@x~jxW;jLA(F9)&sy2%Eg#Te*m)IuXxIgOFz?Q zAP}jj5@?1Ap*l13p4`B5ZP}OO`l->U>i7EO7P+!nnk&?cs4mzM>nkRXnb??_8NtFJ zurZ5ew}q}y`rZ_%|A?QhDlH^+|BYLAiT+IW<7vruK?hUk;x)52RhmZ{ub)9u0;vKF zsCb!tD~P8xMIseaHPBg;(?r~tnAs4-KrIJMWEk>1&Edp78>(uIS&$NkF+6vyp4=f} z^1_?RPo0WYO|p8l_(TZwBGFsGYAiD|DKU9*vZ%zVcy8o02h2*s?Pwi*BmoeUcUg+hUi@!kR7sdB8d`Z?$aWV;}uZkcu) zQEtdF4ln$NUp(DA8UOUw#(S5GacHh?;O9TS`lPL2e)S}C0EFO_j4CNSk)MeCC}c6U z^IgI1Y$z5S!Df4#2aZqj$NTN`VU&rHshqATj3-03+!_z=@0}l>_nWmwu^^k9rCN=T zI~j4bwN~C-F9tDE;Gp9lp5=WXahNbPoUlL{CLjcXDb$Cf{On>l9-~@9ImXt?QaL$m zJMZBqe-|qVNP)*E!&XttkB86Im(5PWsQsR@-*7uaW7O%Be6ENYd^lnwiWv%e^*s6j?%(5S~-Yc>Q1N*4n>X=!R}VHip3h`kM2L8{)^ z$Z#ol)NYN{Av9=|&}>E~_PKLokxsui9QTqWu2rM=-npI^{QZZ=Cnud&do&tJ=8-G+ zUDCavNA28eX4zTG9&F)3U^3QPV+>U07T!>0$RrU+ni(i*BuG+?Iz2fZjqjbDtlmA| zS`Idv<&EWXvuc~wpi+tpk&R3klR?Oh0LumJEH{>G#Uy_8(VgS-0e!ciXr>(`qUxQ( z(&V$sFD9{Ro_$nbHtA2UH~!+i)lv+{01yg{!Vy%#f{9RDN6_jIA03{~_Q71BV+qw% z-K67@z)ETAlE@};tRi`yzmyAHjI}k*(xIejTA1CO8#;Y_HuTP0H)bIqnAvGMV|)~I zu^1&m65HZ(9e*JA)^-bb&Of}nf80wItSSOK&i(zTt>K^*Mu7vLUbu0_URfsxEqoXR zfi(pVnuY3eb18~qnM#HWO`alU3KrtI4f`x42&@TA7z)UFDH^SpgL1(TFwSsrW{=v` z(*iM>2`N}vAqpB%{GjbyA3Z7-4xLK|eeQR2Kvn0+7Jb29PtQhnB%y_i;gv+;M6H5` z!mW(Q!qbinI0H}Mm&N1*CCDsnB@!j!_+o_p)An$;QVYgBkms z@$oRr!xDTjQOxj0gvcp@0;mOu0Lxe|BtO5p^1-{iSJ%QIM0B;SznTgp9FbP6ZZwLGB68q3Gj~r%`yCx4CME_LZfY4yp&%Z5Jniy9D}7@F zfx%L28>PgW0jXwQ9v%1kBNv3m`TS&{U2j1|BmqeXWoAPh8XJd&q+FsPWWY)us1HeO}YW)MXvr zmF@>G&fiOI^0Wv6X=bFnNM9r<)8sZ~p0OZiFhJxOfZ-`le4cg@RWDNjDdOkVXr8;y z5G_a;>5MqZ6Yxkn{~Pr+p}Rso0AnZTJBzA%fz00#b67ujex9ZI?5rvV&`gR;bxM(9 zs8LZ}n1r3$>@8AhP*;L-g<&34{x;b0;m9Gu`^ zKHmGsfBE91llwt`wS04Tvs{UAYZrfcYb9-+KRL-$;Y^LppNlTC@QIXW8k>r_wB(CJ zXHS`*6BLLsE%Okbdxbl}*+r|}>ExdEIWvEL+w8rZpPzTSUDbvKFEwhVQkkgDnk_Fa zFIAI-pn?*)YimgUg7)objPtYe zuT^1L#wI1fjmB48L+@&tv>idJSm zS%%rL5s@@^S(XK*pj<4)AqY6a!NuXfWFio?gqrTw*Q=%HGz*shh zF&lY~bUe=U+>2#Wn)V|p^iK{gX;!OXf)J@9&9c3N(^hY_T#Didt1HFip7zE9r6jZ~ zjcPH52o8=eo*tb)^MKtlWC+bhrBaI9EvNaoR1Vgc8-*AKIB$3ka}T*B#xty0OTH>-e5EydmqUh*3P0IX7 zrYL+NT~CV@ePTcqdG44fuvWa!vs_;hVN6RR^MGh_0m$4R9*uXN$P&}E+AwpUa?WLW zhPgritS3e$HKKnzZ_XbEDQHnKe{x!TZ`c}UCK1d+YN!{_Heh3BZLIoj++dw{2hIbJ z?_&_Aykh^I8I1Gypr}k#8wn77E{QauF--o7f6ofZ-w}}OpG6Xw<*Ia{Y`;(hCx8}k z(d&V;zWVh7B4j2}kj@tCd>ftNzUfCVn%6NIVu7lH@+_Y^+tSpaM~UgB-@BNOd%dZa zzOcK^{re%)w%gYjWj)t}nuhzTGyFR@UgU|oz6xGs#w%pLea>7nKoeO$5Al;~tMAp#6iVeh8iGaZM9l0 z6bj?<(0d2apK6HxL(P^HlOeg_tc%}%bnp+qdhlRxG<3mdw+}9@M)m4jjS?>x@r#?4 ze*02y_y=!^%m=Ctr272n%DhZZIz_hht?!C?KkXdl&YAH9vc>@Lf;4lvbE#hzx^T{Y1J;7*8|;NF!)P>`*rpSy63w|bswy&30EQtD zV3=k+iYd?D02aO+nrG6rmp5083C5{}p`Wy4uwqgfCK^XPTCExY1VrW!$d)-*2#7E@ z-d>QXV+Df)kOfLnT(E#5By2Co|MM?)y&5m#omy*wdG!qu5cpxWq$YF%%K=KIs1%yM z+L4PM@ArQD;eI`-T-zv?qp)1VZWW`gxId2D{pLj%M<@CH)8YNI(a~wU(;W@d)G3HO z;Fu=V7oNO(MHXz^W>r#8JVUnR2m&ySAb^^DD6M{Y+zuX{jN`By2N%^+yj;=MhP}Mg zxU$<^tu~4#XvTQ^5{^%Ij?M;yVMoCrTJ$`B6~UN6c05jp!_<2Xn1~Q3fiZ+!zmcLN zD8c8FpLbdhpN1uyF>(X3LadyN7NT&yQb-Cs+gZnen{@-&d3Iw_ufsc^=Xn$cHn1io zFS@`Ao0V;K{$fIZnp*Qa#aUNwKOG;QrU8=Ni6<|llhPxzgn^5bv>9iIS5N-p?aOam zT5iO+v5xNBTTjp0kDs)Rhf~fCWn{r+OXFtQZm-ps%2A~d*DK{>WC{U_aa1m%6rviT z8Y5(k6*(ZVJX2qLp0Nba=8M@FHYWEz%hEj018Yl#f;CVeOVRK5U7iy~v!hyR5=1Hs z0)mBz7bqK|#kb!AvQ1OxoXDhIX?g)V`TF_(Lw^j`(n2uGL`v|Y$EWS%v)*bm3Jg|P zN~I#5c0@#L<>E8Q8W{}wIJxK^o^>v|!!(Bp(X5yMFlDV?WKb)`t4meO$bpmd?s=<2 zYzb-!rD7qNFvUHF!-4mXDOS(|_TmLWMIMdD!(p1&iezBSBoi;wNt*>1Y4(TtoDeA) zU^c3LG2ohK&MmOjEHV&Vz=DAw!SecseKN?pd8QP?7&WY5Rnph`W{Ikjgh5b<0stZy zXEGe6^Ke>K=Ky`4M)KZ|$KyPAVH_A^3Q<(BW&q7Nc22!$AqcAF>SiU_sM zBFVk`?uc*h-bwgIk$2j2IQK;T_sc?85PeU-RcAEAd9772(@6*P8^m2D@E`f{H2asP z|IAnBycjwEc%pth`BGgtsumQliHO-75do1GA!1`_&P-Mnf>{tu(3fDXVlrN%TlYMe zp2o?nq)Lz%_?x7!G@m8{7N=>3kpU(Z0+C<_)TeF|GK+p@@+NH921cC8vP?wh83vk_ zc9_iU0Ra=;EkIPCOVui9>Wk~edzp~DU+*U~L$(6=*OzTy2DhZ3MRw{W?)54PU7TYE z6}}$rwsFq=8Ef~4nyq?;6$s2p8^8SM{2zY%=H$sam zYxu=`t0%42?>;|B(*VQ{gek>R|D1#5+Y9Rx3@9Yt%d`SH<3`N0F>I8K2t**t0D=%8 z3j~a>eII$cj*10b-X*{+=5A7=ptKLj-xAJx}w~i?PVUa2~2M z@e^a7|4~!WqM{ z-PvBNRw;-PhNy)oSFyE$>z9K!2i3z)^~rwe!Q->r4~`En(oEpN^93Pa!+ljzkj}Ss zwWMs|S;dO8lEIK;QuPyRw?e_81Ev+4j5yukL+7nM2FGz|kAg`rQ9!Cnq(GP;TSBVDNl{}v%8){Ib_x?1BBNBgS=Q|g``w<)(Wn+zmI~zn*EaCR?)K3^ zr#E(Lpp;wA>*Z*7z4X?#)ytdB<%*3gk{F>yFgFNx$cE_jT{(uvR%Nn^j7$Y0X^w5@ zgPDoQx!ifDm}D_|o<>sD=N>ByXZ%#~PKgMMDi`1AVpGASa}v5(pVU`LaDDd*2Xt~X z%AH}hf7*NFYBLJZtnqp?IXJoSN;_*y%~H%jr;D?$oc6JQ(eG!j652)q%jNh9T}0$o zs)dzGAp%+sj}E&RgDllBWF_|25;JtR2ACN14Td}x@`*--@?4^aW4r!rl2>XriC`IVh>)kDWbWX- zrp~c}wE<>>)CWoyGMiUq9;t_$YlYA&C`FUDTm5d#RsZz}pAFH>WhCFxYIgEy{9KK5 zn(duam!K*?G}it%7}NQmw$$=mT1g3_@FJqZY?#@LpKA?&n}Dz?&8SJ<`+wUCj4If$ zF=nRP?G)c>SZmp^xJBs*ny*6R# zEGF=(%0!0Eyj(!4mcWEy8oP?WxX3OBPIH-WJad~cf(D@_K)iQ?k+(jmoF0wbct|SW zzRS^{q*zG07@&&CJQK=>lO!1rO`iKO2m@mo5Fk^3eAesqM_`ue&mdA`KxPtLLWR^r zJ*i`DkBb+ruIgf*?-i1E-{1j4F!s_O4v)KCnWfC2O!)dz&&Uae)GCR<3qnD3cARUL z6LF4ki0XcKoPBa~a@x7Fb-cM+SYEAEs&S!c!Wcmi)Z?HMn~f#Cx>~xqU96RZfBOC2 z;YHt5L`0L859gfsUS9xhOfY~68Rc@Z76&oQ5IBA|C4DboBI}8iCNMPyVp1Z-NaDdL zZ6CU$i*rp~HNw^HmC)L)CG0F0K8|c>Y!s@mc#iWWIazQf&#B!WjD~r&5HO)uFBXe2 zcgOIH6FEV)CX>R`BX=y0fQ;aY2x>?SjUqjN@11Km7_4H1S##FhkYCK<>r2iI-n;R5 zY>f%S(B*EN<^a8LmoG9@RUse%9&luw36K~uP~$|L_md^Zn2Il`h&kw|pWHuv@A~ml z{c>VZPq5vr7NYPxbqdEA)eCs%#`<6U=+gDgq?Vuvqz+!89@!`z4Tt?+uiwkh&IU)mM8(5+EYZ=@h@s9>)aLn1$2#g=Pmp!N5#xOg%6uMdSMwXrYCW zm$)}{`zM{=U|cB#m4aDYNGSvf{K0#XT$R+3 zxTPR_duQp#*H>?DR?20|1}YE-IJjJf!~S55y zf{2h}Vx^-x)gcm{{knP6@JtVNj_f8ok6}YZB0B5NgV(8^iz;T^mW6QSC5tLmF~$<} zw5t>}J=`%ZoG<`pV~EIzNDu^*gLRSJI{Eisox;!=>r91&4I477?8S-5OvqhnHY?0z zm2huf&dcn6B8Cd=CuF<>9i?^ttF!Y2Rt076Q|f*ysA?&7O(^Y=92`l zijV**;=OoJq6QPnn|cvEZ!edHN1<~f7oe{*#UgsG=+nH9eAZgLSgD(@t$CeVe?Yc> z;whlBejl2W_l{3H2gjX}6UPczQkTJ;56x%yPuD*#7Lw9#BS>Q0xK=r7UB2iaK00hU zFfF*AENGFvq>)YZ4Ox0%%7{>a=InY1>EarKF$jau7@n`GMb#cZt${KdOAG}@DKhUB z9uS511P^$TFCU&h^HJ>fQq|za4GMr%DkUOhnRB_OUeqM*0UutBGd0s?rVK20;JmJ7 zMFD*k#c6?fH{;jqf?+Qa2SHdU#2Dp33NHWz)a{$U{iml7o}9u4LL{WB%1W#-Gfs*q zoT`TkA(iClLIp8M0XNqJP`uFUAp}G43`cMbVhl(57Y|PV?zeZxnTmor0GOuuHV0*Z zfrb zI6gXScY3vIg#){^6tAqV?49nzzZl-nbA6ZqT5^n>XtIaHWo#k=TZZ#Y&sD%h;TI$# zXwGq;uZ%p;N26gB$Av=C%Rsz}C{#e-7T^&;qSyZT>#cJ0z;5F6x*{Vq<=M!nIvH%vQ&QLEECJ8zA8 zLpRQ^t(C9t?$i~m0OdkXAb^=qC(XHA1X0cM%sD$xVbAZ}FoG~N#=tKsbIx;89^_xJ z%ckiMUVM%(J8Qq*q>#QlzEuE<%=^=ei&nR{vAk4Du(DJyM23TKd8u5CFm^a=b-JTG zr||IP;`sD@tFciCvAwZcE<7k3v$M5Uj3L02z0UDzM`1+OtM`<9=fs07UQ_iWfOwuw zQL;7&!-(c<2&|A-xfIZgO#AtZ#dKldG}v#C|KXGUJ9qX=K@=BP-n+hBCA_t1M*Zz> zXL!^eXA}ZV_!gC3WvGGhgp~#`!CGTXpgQ{IG6fN)iXPV5IF6XjvDYKFD$Uv>wmni)r>$`=p}RBr&(s=Bo0GKnSs|F1oVnSLce=BNsG7h!=M@2l)i^a zQB~(%V4sb^DZI=|PomyCW6amJH7An@`&$)UXkmQ&ll^mMfZ)9s5dx+_iT()wXW}pj z=C&*AYrqp38NOIqmReo>!Crzyi1 z$v;fbSGVd^W(Hwkh*@SzwW_So`S^xS5QbqK$84a$D1-BU@|P+3X*+osT#U>!C)?xm z)M*0x=6uu}A(4)Ud7e4%ouKCsc|<~lnY;c%07w9RhiS(2U?5(sY3uVA3SIo3G|LOc zrU{gAnvO>RpvXNQ?c>8Qhhkze#*>J82FxV4H-l5TpVFfhxgt@-AQ5pd)FgF&Pozu^ z@K8tajF5pKIP1x057W*-#H$Si;MtWajTKfQte%krM%8+?POLF3nmXYWK}tc_!Q{hm zFp4e)rN^#OJgXJ8Q4O9!+&9;gx3^2x=DH2dl{LKk#^&QkhvS3cM73T7&Q1G$6!V@; zt8gF}YG4F78Bmoh%a~1)1WB9>ha=E*v(wi`&l6+|H_m)2 zCJ2a#34t+zvCg^YO3Q2~y(^f^M{DF7s*KMHGSAO7(I2+fnZkQ|?jfRctM5f#G7(v0!!UTUW0q(4`O4r~7yRFKQO{L?JjYj$ zTEF?~wAJP!$>(z}OHA zL^_=&Ls@4v4r{(HY8f;6M4D$~@x-oCif&$7zP?c|#!wmUpSch3xAu=uJN?nX(NIk4 zG@Y;%By7}U&mjW_-avTT5k*)iOtFCtLS{3c%h$C?y{H|fFNR_CJcXk0UdCw}+b{?N z4FYB%(97@5MZVo4tflW1axS#1y^Mo?Xp#LMykSKYMD}Y4>&4Fwg-J`EOvs%Ok(v7S zw>(Hy$@9!P5)pmLO_k_RR7E0M+^FV7;0eU}qs48*Y>XkObMA%ekWRUt@=b9pB@q5Q z9rPXf6^rcBnW~eucG^dyqAD|~UlNu1vmhcH*sm9W2PlpcCOhfsoO=#Wj;|<=8*6K@ zrhRxc8V+gl?K)R3o=Wphs>H^cAc_jb0*GWXycaiL3t(oQi>%P}9_c$fDl7DLJJGzi zs8}fnK>%o$<)hKa2DZ^`!Uh-n$AhfTm@=Ig(JMU{2%h26g-if57b|66Uc#W?K0S@1 zs2JGMXfzlM>FfC^l*WJA}lp-If1Bh}!0G_W3#boWU11bjhxT12rpR_ zNE`))gu}Qn9ObRHOH(`YI?lB_a>02X*m2?E@T+QmbYtn~Kd-N?7oz~H4ODBPJ8p$G zv_UY*$$5o>Xm0vQ4se3p`D{4oY3@qK))QRc3BFt{wZ673_R35`a0Z5Tx(UccHj7Q3 zk3HpPYK4l-VKDUK5wHc^3@#|mSU}LlFiDhAdq~0&a8hU^YRhGF7uTU{&OK z&L#_kFpi^fnt7EMOJ7Vc7M>?eb?5wWENRYU3|I_v97X|iU?%J-5tZC$#VD)5nt-Mwz6-x#wI8fkHWtB2z9H3y^^! zwQvLx5Q8R~5NxLJ8xeec1Vhv?8>mmm<3Yb8`3fwO2$hnCfv{s%VkLoBcu&TWjWjC+ z^7c;Yz3bayl$>3RfBWUrJ9`eQ44oref}P1t(|m&Ux%f5D^68VE@MRG30_dHk&S^rJ z5Q+wAwtq5sb4>LRE9G#b%EdC(3e3P^*Z;xy*Vv3q2l0cm@s9_2DK=|Gyt!7Yl(|-h z1J3&9>2aECQHcx*kci@}Jx(16h%C03YxN`;cL`)>svt6pGQ}Ei{BV7#Q7ae%QlQg= zQ&P;!*ZQ)u0g}24*MyxAkv^-2)CN`S_lzb^v$RD5C8eAxg&^!2#B6%R0-B_xeyh?nO-i5LcWcJM^GgZNwYMt z9EG7ZZeE_9t+6S4arz;iE1k^n?d%!Nh(hx>_3WbAj`72I@h$pFmV$n3TYxeufwrucD$8K~C3Q| znT)lBX|;BaG-eE>Z#AkR3TEH?jgpX^`O1402YC+rh-Npv2>ibHa+2!zQ$NFmWNyOG)#}uH9IJVThvD={`Nk`-5C3 zj+A7W)p=%eB$-I20w7Y3iD@pO1m($%zT{w~BG1oEUGxQ_RK=DGwY8PP`YH+u6j6Ub z$0zMxuW@Y~m0C0!4n~9A=Yh4)4bOnwI?fmv;oZeGRNS=)_P zw>E;MW^jJdzkSd5$Ki4_xOo|^uHJo^I+qbbVv*ERj`sb;IsckZcxvChf>0-c~BBgPZuEJlS$y+oaX{ruipb=lS%l{Xqi z1H8F|){l2uozb1g7oss6KZcJKBJ4#xc@Jx&pj0jgfp^a7w8}(Gp%ODtE5%DoRa=qTnm8C2!}^!o&LzFVc_`W zg2pZ|24`Wj1*6xywiWeq0C~NvqS(1u(EN zHg?uawMPf%P!6ZrKZ*e(B9J^_+E^+5$G^F`x||G$gX7cm*B|cp4!T0t2|Jp^k6<7$ z;Mp!^ttkdc(dOOO#p%gu{nBb|u(6hGZC&3xy^}feu(^WB8_fmQXlTgQ%E?0(_qy&fHG!K6NWOVEC z`AVr#PNM6}_zyq6)*JX|g`>inaNG4i1I{?5;mMO6p6;gtSco-JP z%p6)vGsOZWyR%-|US5s>f{Md19R=1d6-uRK)U2XeH35)$93N)4?_WGRm9Yz$t%AIe zQ=_n=4xCr3rE~m-tS#nih^Q!sCPZ(bMQFFcb`Boy(w@Iniu>Ow5HG6+9<= z>*RVZHITj;`I3Q6hPi)o*gkKMYn5QBfGaD7a;;DbkpX+>o&8=q1ZBqbe7N6p0|x+A zgLk&;HgR!)fyY5h&U(l-0dfX_M8HR9gLdDqmQ6(1U8`K#T08mTn81r>1_U9>)eVv` zzp=YsNdO>qIByR|;|vI}5LYdvfN2gt0W+PEg*{0DN5~P4T%@+(;SPJl-#@%qS#371 zmJ5uXW!$>Danf~X?G$8maw~w$j6LV3DF&ou+vCxGtKG|&qcARq*j#B=OXG`fmjH$6 zjH+YQv*Zc+)oQ$4FIxi#^oGMhe?%^dD5%6iJq`ngC^}tzc+`K|S3v?cfPoqX03<*u zFwIJ}nuGux0mhiAiL%LsPul&9PH(+b4hh?9rIqF7gJIuiL!K!F>9n{j#X_;!C?{5D zf0RzHSHjwb*OX^@${ZUC!+@A)d;aXbxd?P$$yA;rTP2!(_)Y(fnd<=Ihq3tHev#Yw zt@HC*C(4Csp2SF^;=Hp&hDnGpb2Cv9@y>8yn4qk>ppK)j^`KDIxfb5U(c4<b% zzMx6Ng!xy_tU;8S=f-8s6h$F`JbN}gh@U4}==I^Qf(X_a%RCW-1uZT=Si{qn9*85E z`BcsCV|p+D6&Ig=Z+^A}5Wj-8@p^K&0`zKWL7AnjW|^lMP+`iM<8m37uAtLp;pW;Z zmg=3oy&+SfQibYxJcdeKER0+}U^I&%)ysCR8LI}v9K)1Um^i4G;wWI}CC$Z)F{Xa? zlD)bstmpk+1Qw$majcS*%Y`H{%;Uju*zExnl?rhj`7i(=oq_B2jbPeqw;(vG--u^b zc&>D}P)8J~vsBw!E8MyPOZj+Yl(^Ig*pDest8r8<`pD)aNjYg$>tURb*I}pI>-FNK zSYK)OUEb|=q99zU*PQnkt#c4+jcQyf^kT|ZtEeX9QFLAiDjUsYdk2kLeVCr#d+bUH z)|D5Y z3WLMFed20;ZKWK===IN@?hh_53O9DEzjzNxbo}V?Y5SsBudWGtF`Je8rH#$v)=Cz! zzvx{&IygN!udJ-DU%NW3;1)}0b5hr1ejPwtw2te}{dxi`^8Hsa=3u&xZj8Vu&U)PNJsM!{=XITv>Wi|f3wznCjW1;q8_g$qZZ`6w$eJNiK<1xObzV~MWaPSj{N(WBqV1YX zw#;~Q+q{2sIsN!>kV~%4!Q}9OGIA@!<#PCw8_oaxep0QVJ1l;Azxd(Dd)|-L5;V0c zR)wKcxkL7B$-)$($XYXS{^4Q!tH-C?+p7hG)e?U8M(yOlJUZ-+k%-0yV5pMM;gi5! z+b;fxpKiQ;9jZ7y#@>D~hA8q%A$<4d@=xCll(1eaA0FD%ZrkT~piJbMm66wL_D46@ZtYY`A?3jSDW074N2#b67=2Nmb+J==rh=(DQMW!jzxdtf zqs=A0ae1{6;?2t#XS-o&zkJ%c==)slOvzjSFlbT=QGd3=SJgE6vaVcmEv)l=FW@iH+*y@6$5G*0`2 zL6*XaVde>Yhcq*kmgDr&T72V5Gl>--D@aV7*m6iIfdl4%VT~}f?Rj0@&#fy>W zrzeBGqqCjLYANQM8|8ajjpMVE-bg(q%B*CKN{qB>(kspCjrC+DL7&hY;}7M#mt<^J zU!U36mufuvC&tqz&Ru$V(m6TnU0y3!0=%`m6&9i-iuyT@PurtWE}qGf!gRXB<5qvS z8OMy>%^F8OF=#tHI6Ur-M;@V0RVxGa`AMt$Xzyrud97$ovyQjlSi66CcG}N|sTT^2 z2_{R_Ktgs`H<~xEY=;B~w1zl3y~uJ8iZHj-d7<1lkGljgFKImLj3t{B93UW*IrH?W z^XctJyYoC0@ZI8SahaL?I3-p`^TM=i^0}fTsL@QjUVrn zKYP&X(TSy1Z7aB%uL35Qjk&1QWFYPv*EYIymoKUMYm3*{$OGdg<{=`yx$+0w@Dy z6d`FPkn78ZOUvbA#2!$cmA3y#Wl0qfl{`yb;H-%!k@yQI?CiBnUVJxw8-3La#Si0q zXxE~bKB?-wHzs&CL{w&eZvxGn15p?R#8%-&yy8z76Y0bt2*`Wy+#)}j=58o+%{nu4 z#hd9Qi0aIui1d#}y^A>%B4Z3Oi}zk+c4W-Tcg6saiN56z|6<6QBg-ZjR zbq?-4==b`KrAF=YHdl&Z<4(^{o*Xc9d}$X!(E93L>*UmcEOT!(D%HxhE0=7!1Y@F$ zHb34UbbF=k^>BLwVF*>Ux+ixZj1SKuF9sk$<}R~D-v;DoHCsmr%B9Mc9W2$`pL}}q z^dNMywzAS4kJw8D05Vi(n$`8|*U)HyEsQCgouAyhA6RSNxKZs74!-zY;>f&p(`4!R zi&3#wO0Hi;sZh}zg%D9}WtyD_Gm3@e>Mjok!;niVPK2x}H)_%5Dk>!eLD1<12S?*V zjCvViG1%Fx4F)C?Y;W3L-@A-=b~yLwcPNhNy_*=19c9gp^+L0b(HJIZZm;z3-Ys0; zMXl8O=!?UD{#~cn9TkgxYmMsqdv6wh^cD<<5KyqWy4w2ucD%lUzj&XB(CeX8hSQow z=XW2YJpe)HhVqaF>b2@ZHFYxZefE}oPw!&}@P9x;vZ!KaG4S%<$I5{kqeRonRgn9_ z!J+*2H}~(}ekvXetXJ`~x2x~Hd8u5m5Ha481kRaGd)({g3?!B>Z8vV-SgV#}Qi4Dg zhVoiP-@dW@7w>LtHzP>S3W!lC_i=W*Cy-eKp$uyt|beWu8*fmJa&B{nc43aM0xD#auTZSK*{!u^Z z-SuK26hokizAX?%!RC7P{kPZNx{Rhp0*u~1#x@`D?H^TG9NYe^iS zGsagB&!3+5CnbHIX60X$FL)6rDip*-G|KWX?(hH0XAcfe+XO5Z@Y7rI|N1vq|Hr?) zb!~U0UM&aK5?R0iq!7RbRx3q&WxMiMA6)t0|Ht?L<1gM^ZJ4QQF(3$314XC^6o#OU zQuG%;e&fIV;=TX*uRi#zAHTJ^R56lUfG4CiKvnZBOVhE44}%~KLu)OFUaTpS%&Y<| zP>O;|7?h*1l7!VbEJY@<6dG9a#mF{ml}l^2pTD{Glbh@11OUbkkM_EEAD(nZc`qCG z#=YL4Jsu4VUR^Rr~`3H+xG@@j+&!<5c02I&X^UIL`~m~pRk;_)--;bZ1W`BaGn>bz4`YwV=i z@=qfEO-iQj>*2*uAnOF{hEcd^s7cdLMmDfSQ?VlPe*CrP@&pxsH&{zdWG21AfS0MU z%)AbaiP`nM$@Yiw@*L{y0?ifSR28}?2t?DY7QuggIl~A{ac#N2vw1w|J$U$N)aiL^ zu-e4Rk}JfvUd7gWP%Hbuf*DzU(eI2Riz_?P=885ewHudsXZ6VVtPt4U4O*&FxrjJK ze>gfk4bS`bqL0o{nd_H#?AGeB8@AmDJL~1GbxiQy#>&~a*O3gDHlG3AJYbo{r7~kH zd>7uQKy#J2R>4vYz3$(+x*(9wI|^h0PV*yo_RS*1yP&!TJMdzhpjUefe9>3WC9h%aG3Tws1)iqE}KSG zn8tZF%CdfzwtC(3Zfj@-YmKZ~IrUjP9U)IIx~;R}08xl)6{X}@vNLj6Z$|6OvRp+u zI?yy*Z^Sou&l2-R|6&vaZ(NGrx~gl9oON`1Q9A0ZUZmB%mODHzmP(CV*E{v(qo)r( z{OWd^r?|X@t>tbtMmajmy1%^p`P01zC|KQEM!5*$sT&b7;W_VNI)k%f?tI|;9r`C- z|8BLB=Yk~kO>d@ex;E(4RYJ*lEex~p-jnV>eSB1DY+l(61B2ZS{QSN3({}gwU+%Rt zD==f{56?P#$DPgfQpBU}wZeb?`D!g3K74R78m7iltybFJuK(m&d~>-Ba!5xSg&-(W z#Sn0+5ly6`go#3fP=W0={HI?ui%Is`ooBx-%bZUF+pI4C^qrNTy}MqmA_MO1yWf7f zcX;0OS_Y{iABp+=;n_cZ_*IhJTwk#(75vRlDwX=14dl%-iaV8TTYzkTL@^zbCjL+9OGYKUe;Rqq^tZ?mlU*Ou$ygQsWhq3h=o2B;N+XHvmGe{=Wd4h)cbJbC0k z`}A?Y?}$PtEW{orb(DGz4T3*J$&H`2e5Wto)6>JW)74y(+mD97`sAQixL&JZt&YF? zaWhQJ=l6~ePRH3W54hhMKQ0o>j3Xucwjzf-h2V+KYenS=R=K zfAQmZdo?aac>5Zf%kld1rNguS{^9vxn2$yoz)WJvuQndr=7FQOQtK32S9n5JFWYD9d3zHcI5>;?-Glw3vA`%S6qr|T^BjaQj>FJ`pd{iv z$vnkM6FFkT*jOq4?49*~W|=GjAb|=nnP)APO7&X3UZ$NDG%N7H$l?A0e*O8QCx7T5 z4m`o7e(zNGPt&VSfU&uXpT57_$@1fqPM(fJ!vSe22{zZ4p6P+zx>BqsP$!;92x>_} zJMmz)Q78r}Y(_-^7+fueZ(eRXHyk=M%%g*oVLsy4$bR|g{N`r!?MgKa`1(%4>$P&h zJ~-+Q#+g$Tf?%yt{PDHrcW$f}$|N3l4)u$tC+)%5fB^yx%H;UCljO`Ns^E=Bm!}p(FoV5m-b0%aP1WVQA7w=xW zv0Y+D27GmffBx*z)AN3&;k3@rMr3A`H?2qV*Ov;+RE`+aPkJR$R$wA#O*&QIRt~(< z(IEZp-J|8z^>;2W7a2F!Oy}0ldH?R-SVCN!{i!U}duf{FVEw0RK#{Bk8k zMFDGH=o9tMF_V64Z1ThSZ)W@Ny(ujDe(gauB9!mnppj17VSfq|D1 ze87M;JlPby&pO@xukIJ3`0Cql>QaTt^ZGK9Fz*Q)Mp6vQMadnkL6-U~C+7oey=oLC z)#ehi{NT}((Ae5rWh~c6r$v z8e>2N?^RV)m|1fl2-t}NBu-0lX>A#S-Fy7x_~FyWa`W>0@6{UhmSKc}h8JQdtb zqOe>-nw>r?P!9*Oq8=p>F-`)n7^Ua?hfy3at*<3aVmX6q945^=H*2YZF$QtWUb0S) z+C3C#aM2zPhQ3sTH6Y45{lU}2X0;ydu7}LS@pylKe|2@0<0KxB%TbIpR~k;dKLa*} zTD4zJQb-sCTrJwQ6_l&v-l+ZbBk`zO+=ag#b5p@E7lHx7veGpN>^{?B(dJ_vPLFE8Eq}P1Is}>oTuZE?>O0o2HJ) zCIx9W?MfBhi;Ls)!=0saDX9`vHHVG>QVl?`Hee2*HYapiW3sxl78jF^tGmmcb~+xT z7-6kh*xlr%G92*mVEm82d+^MbvzH10P_D>E!bd&%^{4mi75=Lq@2)hG?HZNeqgz|s z?Y6X9-7L*YakNxRHdbw=B$m<3&=84RNu4b*Fyb_Gy1?omL81ZpLfNwoyd}9JKS_etG_z zJ12*oJk3pDY-hv|o}7RB#p6=^#`+TNHeuuPmDQV__T==`C`7qvmm9LSq5#gu_~^^M zk8eL5<*=d6+_RDEi%N`2i;I>$*;LJO-~al{^AO{}EpKja6yxMdgO@9OYq!xEG)5zg z$M8<#h$=-bCcab(3IQz89cG_Ca)0-my*p3az1#?(Bm!g9Oma6w7Rm|`87!KZ&xP+` zM_PY;D)QUC{$Tv`cQ>x?G>TEMUB%KnVSm)P=ro33-vANVz!Dlotkh77kr+&P?X4m1 z?GJwW)$wn>*grgViV&(yPD!A7NW*4XHxe2F+Ca#C=Y|@@1O`|aB!#m)Q)m>&LBQZJ zPct`8iO72ekxad%u-d?X{)=6Cv5S~zJFGPbLj(o^1HdrDgCqBkA3yo>(OEBrSgSB2 z9~`$vUp+j%zOme>QZ>S@>k-1Yj*nXTD6=X>YigxOcPELG|KTPquD^%QphO#k_JkDl%i#~MC48~p0?ryR#OcI(x| zymJ{#^-BkBbO+femuld)o5j@zRSQT3d&k3n`QrHYUTctf%S=pQYXbn#iT%3wP`lVg z#mwRbfH9T`02zZnBWdaZB2G0pycnH!`rFI(SOJC*PJ4N)mpdN-LaGY(8tfmpPus(t zN&##blmk!pTc_>JDJcnkU5)o-lugFPv;N@2&-NQ}^2UvgLIGPPlx{7(xv_M7F&ewf zL~LU$Rm06jPz{j__PhAyXOBL6dfxLkgAIt7DF{L`Rsajw07^VZwzEKYCRm^(G9x1r z0H#?W<6t})Nc`xc|LMK`jaqS~UI=2mbE){`Y;`ysby5Qz6A@@`phFYaWP$Yv`p(}w z$!oXwVzaipURjNi?3$J4_EBrA-FKS%k`0&3y568hi5WO|`0P>d!@E!R+C#4#im;mQ z&^~vjHObE>Wyf3iA%ugc^+6EQ$OELPx#IBYRdgNKoaUg(9_10NI-^sUp zru#ivR};s?1<4sOvCcRD8Fj!|3l(4y7t!2gP6W!|FTVX@=(iPD#j`iY8e{xKfp)UZ zf}Hm*cR^snFqABHUg=vEUM8&Lx$?3TdHIV}Rljiq;^CqCO1T&M7R%X1@off$Ut?GA z#my_dvx4meJS*TuM3h)xKRJZCN%;?BQl4WbnM+9#0pG9)hG_5XJkPUwxl-QQJsG>c znPfhV3yfF;A}2B>*EmPxEU+Q0=^KnfBcbtzl(;__MBTwjyJd}8Q9+Uv3I#XHr*c+i z7P4l}8-WOtupdwc4ReRV7{m}Y)=uNqxy{UUVNUhEb%x9d^inH?8s^bph`|8$`uh4t z-%|yqS}Nyxc9M@Nu!sWx>BW)H;%2S5w#HdT)|8eTh28D3q~~X6tA!*=)DneWi%!9|_;tgb zV@%AJ{BSfF9%j4SrOR8P;Pn2ZqrLso*MK$%k|Cm+M744{j_`0 zsn#1=A!mf)T`RL+Y2dF7xsR7SVNOZc3#PQ&BpTW`p@)j7MH;D(#2G(0*n7G&~W_1{ZG}dQ`jfxAO4S0JSncxKTK6T{#{< zJ`awPpS=~YK!}J!Ym5O6R`liZV+YOIS)4AX*N|w{Wl~8708_%XPGe& zAP2xqLGTPJ0c!x^0YPlhmZG_qQ7`& zx7IXkC9IZMBox6!{FG^0={jiP^V_X|{OyA$CqoXa){5j_oLeAqzbNuJFM~kF5+1g) zfBN)fFdBXEPT|UCSTA$YqFlm834T&>Qf4!?nDM}1jPq`O`@zAlKRLVeWZWK#Q&K=k ziD)4qZYSX*6VbHFGCd&{sj8{Rgy>Hkc<+R3Q}Ph`aP$@v(1q>@n#Ai0@6hj|JH+FI?uTFPKWiIq4GcJB zVm2b4+9`i>_h`LR{?VV^*!g#^6c)DiyOO}Z|xv7*r=mf4{|~h(VaWdAv6{4rRMWT-JI@sv#o2p zo3#?QSFyf|G!HY6gisEU1FaDro#9_TKmOgF!;1lgcv9CEg~)6Ro3;}Z6OqnYV?t1x zQf38!2>hv(^b!4i3( z-NngC&*#&Y{duG-3gKwr{O#i&fAaJqrz_h_e^_HxRI1omPe>COcm@#?Mt_7S2mRmQ zJN)?ZQEQkB#}f7~m{<)nnG zWo(yeEP=Sd0zm*v7>l-YZuNKe@HLRt=&EjYc&tT%31>;2GM_@$jTqpp^3Q*4AdJ z6mJ!=S%ipXp3*kPIUXLI4M*LLt<_=`PT)O67I*QPss8Ib@x1es1?g#~V>Mft#u!m` z4lodTvE;qr+DtP!J!2pWH5wFi1Dsz_`L)Mzm z1}eGeAA`!!*JabZb17g9tLh83y1DbtPoxd-=MH1o8mlTU%b*I(^n8iT*F%9k&qbB^ zk5^^?JF493YrEQC8uSWnWQLN8iYR$e<>!)e3I>c&s01g&)6x~BTY;b()2SYr& zmY1%*{icZn#IZd)H@!i6ei3eN)~{Wv5BgYXf(bq&OO^qZBzLWgi`M-|MH`@yAWL1I zLM66AHH?ZjfEOzosA`%a2pXH~y}`hVhs!9-?0D4a^gBmK%U3qh9m(CtX76~cR#(@e z)uo-4WgVsc!{g#nYuFuhiskC|4*mSc)!c!C@bruyKWRO9T7RZ#@x~kY|FL%!Ty7-W z`QEB(5DbIq@lrU4@`LjO!f-H*50}^W*u$ch)J)Y~aHH34c|2!l!< z4__=!2hGWGn%MoctfIIw3XSRdhlpv>Twbrnv@4DG{N?hii?f5v)o=XjhwnYz^qR#` zkR_@zHP6)9H2w5>^!#-;kFmBpge{^Gd^0iO`oiIs`R74W6=##%;Dhfy`p(nC{r%x! zAeP%H92g;3LPWPFzQ|4WeA(1L z{KX#}e(?RhXO9jC-dJ$-1%PF?Vp)s9BE4Q!&!^c}=c__GR=B*2^5or4OldY-XMg>} zuXH27|DA*V-GRf*lNJbg+?1K6|`>xaW<0#*r~#q*z|s zb*9f}$(J|j=QrtPU`oB`q8{*8Ba=9cm3=;`TKOUm!w=qj^!U+mcX#Z%PB)XS)uxv9 zI#<(G@$$0z@+`Yv^wOAkpzPA2NTo+$PtMu%nr{VWbhffcj z{llH{*azlX+*bBPS)yjQDu#&TNn5UIRmJ%OY22&5m`|+HU;p@37N@`b{>ih`!yVuCz>rJQn8rdBw~KZ5>T31;^`b1& zbqhPk@yU?lW;}Fo;EO10N@FWy($<{agvXy;82R|*WGFl^*rDqhxNTj|8_T5e7lmpr zqEyziW%mBF~L;c%EMFN{mALAU||O{WsF)~yUJbNS`@a(A@*XxD2rW28{p9HQWN$chcX)@oHOPjNs+mXS z(4cKlG$b%9@zu1D@>ibipPU{IHvn05fYX4nkSA@JrdQMCauUaR zw0F2W8hP@_kMPhEp>Q;4P8I=MbfQMTm?mviycZVlJwAGTd^qx4!ALl)F)GaS+r?%r z3Y2YgboXkiwaq)OG6K6V-+ma}_Oe-JX|axWxTe8=nwQpE-sztiV5?M|X4f;8xVWfU z)wKN(%GLuM?>?S!3w*MuTwAt--K>;$`|bbV{v+Bqf!lu6ly6Q!2&RFx;vRMbI~Y?!#Yl2V#{qTMy)I9M_u&T=w-CeJ(b1Df0vS|QRo9zSSyjW~;mNbdplvY^^2wr=a^YQ_vhN>{cLev1W(A5Y z&2BHQj>(Sp_q~HXK<F0xbK!uE#0zC&aZ}+ z#zzNDoK#^nzPj4{aZj6KzVLyii=d8^+4;rOMib*o$|5fwy?kl2(vgiW%jwIP1vC4l zqO0kCR=4W*CD6fp-`yFFWLY(n`Cd{*X?FFakN1O>INtY&sjQImuw6td2(q#mAM7*N zQBfM-Oxv<5hu88PU#{&>KVIp^RC0i9%a0?Atg8%cgXQSmaNq94u#+-y>e~MT zWc#rH&^xRZ+n2*x*Lu6WjtYyGENA@wI(erI?1}Hz*lj=_!M%Wo{)QtZ{Oqgr>L%h0 zl{U-7)GfT(Fz>=??Di+Ix0$WA9DMmgu9g=&9x@V)+(bL8R51`fV*pNJ%-K>d%lXHX z{PbkBvpe!VhcVH%Dh!jwB3vYGTtVhK+21a_k5g@IkQl9jaR&$$3}+$z@!#G4%_q!) z$bXJ|jR;nYHEht`!R~12aYLq(&2(8zgCba0MZ+|Cf;U_+#vLv=6O47WZmD_IxOra9 z!syeh?D%kay0IME8TV_OF{CWcmgQ_!EW)g4XfQfr$b=)5(p^xUNX_t1KM6j$2mwuF zhe6IYe0}Bv0oSnm`B}9NFF9HT{$wc)xD1HyvbOtW!kKe$y`bOx{TB=^aeom>3qD{s z^)s5VwSKJWr?0Emv*z^WW=@v-dwYE-C;)QiWLX&mVO*%Bro6VDeoq-9LwZl=ca zA6DgMc=nxdA0F(Fe2+JJ5N6K#P2GT}ySZS1m#y<^Dubx};zphvrF(n+&Pa4MCa~6( zlCR=&zK)|@Wi7Ot3uwq1-PJreSqq|2-s_-^vf!r|*~#(F;r`xe=yQg>XipgvrsZ@I zt>PvrP=V+8L(T@OwW>95{8{e#Y0*19NZgY03=nAW;!1yhnO)7RgRy(C;~(tr?e30* zgNV9c$+clcrGg+^g+-wtZAdDi=*_Sa>j^_(EI`Ygt62Tc-&_ubChk;Ou!-ql*K^Z3P0di-K>xaV!k2IKL7v3|5rY))xjr+Jm6d0sY2p-Kt0Mqe3Y!5F5j1s~VXQ9R-aZlJ{5(ZL5e1*Y#ZCy8ZxM5tQop z|Mq<7BJ>3VH1vkXh=lhd!aKsGQ3~stidt1INrMVGU>~?^sf$fU!dg48gJxYWp3mje z$LV<3MB zlks3^EHDFN711nAb;oirz@>#%Mfr6z374N=HOGg=?#_q_Ww9o0wQR%8EJ7KiDwiGq z7;Qf;n;Y-rtMY1f=`vH27i3MN`aq>#p>T&sYVyzKOGPdtYcIAv{TW!o@j5w~HN&@()M zS^{bSBgLv!J4xFwX4T&A==kJ#G#CnGU5q9cMX?H&Mb*T)E{(9<;pkXkj3a>6C^8!V z=)5>vys|_!Im@gzAbOkCJL1mfgj|t-wbH-+SLd8F4bD1~AUxACXwjm;j=@YkxBjQ| zW>#Mf9A;6pz}J-qE;NlU6Uu)!A8^}(NE#*^B7V%jij}OY-@sM*yK&6yfS*@ZX zN^4^5N-iBYF#u6bN_Bh;dboFg*8%JknV7$p^ykmlpIj&IFqXs%jXmbIfQ5BFn`F1^ z++`$5tP9XrkQngs=tdeazE@dm{)3gMRRF(ycr|ZP^JWgO+MM>tVHvR0Z9u5$} zDBV=G8yC!)?Bdp@g+Ce!Lsm9WH;$p=c45+TJRWo?0QaQl^rAT_eHfL3us1-P))*ca$EKQp-9l zDpv>~W5@`GO3Hy!b#WFuq~*RO=Gs3xVk zCf5RM+9sFAb&zw^+G*4tk(EI-4@v7`Yl|}Z2jZ?HNn+iG;(?_4=c<1L$DOlp;JuIG z(>d;(cD{hq%GusJrr}_CLn_cz zmMOq2+2wMB@ufl&9#Tl6;mI`I|3BfgU--^TcYK1lFNAbJ$Sl8$7QdBR6wp0K1#Sk_ zrwuh|?lHR_Jwg;B^A90Z3MR;SfeW**(}keu?bf`Tr{{o7Q>8+&vlwx%yEq6H+(6rS zt!!1aNGX88-TO9rAMTZC`z*I$iwNd}OMrmFRlLejZc8g~y>(I$pws&3=X)KaLABI6 zcZrnR1nPj%rx2zb#2r40o?F@8SWZi8LVB2lvz+^>k7jNJA2$Bhs$-1)Kx4#mf$2bk zC^RRZ@FLPgRB`=RCFxgU7A8%V6=x-dEUk%1389*3`VnxyixA>;TCLieE&*l%@esVO zy|0&U1Sfnc6#yBXZ_P!9YJcMmTNcy?`di!f?z;Nra>2B(ps66O$Z65m*xH^I&bPMl z&f3Q|28B~@O#CfDu3zaYT$FD#J+;z`CNiuE*3B+`bAgmrYdxJ#s&&0$*K~9DW2I;! zqc=oL0DbCw)W?aViK4o+78N`$Drc3V0jLwtf~*gbvN(#-ZT6?^S8j7)Qzf8)%s+R! zPYL5_rj8@rkLlv(1<3$%c6I)#Lb7yv~9kX9W3 zFXSAOAVZr<(NjCM7NwHOv90bt3uex-GREN}zs$tn%cRJu<3S1JIYm>iC62dN{}S^z z@??JMoqdDAin6p)uQi)nDW&(ZMJ19QS^~Mptk(9{o4wULfeP!kweuM^K<*|0$jlM1 zrPlU9D1Q?C!MwPYG1^0OFZ3cZ0E<(lz)k!WY#YSlwH|i3L**v@6NkNRJVu!W!Z|&8 zJD&s~>O-DMt5x@z#u)eK zGfAywFV`#%_2?Hf&T~FTr3;N|t{^cNQXpm5%-aPLaGC3v0%CA32%7rfpx*$KNHDK} zIajk!_!j1UMDh zYq{1ac`%7fzZ$~>9n!*qLw$?TG?2UE(S4MH-52w=(g+_X4LwC9F=-|Xy++AT;;yUe zt6dVM<}p9W*T+JYc!p;RiQ_mgE)l|AP2q}MVVQROn7}7Lp2|EYiGRi+-4P&AMGJEG z%eHO9ge!(2`jaZkN6*%#9#z%1 zN?WuH8{KRbHweW;N+>`LzD2lR*69s=8VmIC$wIL4WZEErJU*?QcpC8cO&|WJ-~Ra9 z-}v6wzI6B7-xpmVME)v2p#b=g0ow8@@M8#s$Egr4WgIr?{(i;hKK=2}f8s+QJX!Gp z8;(L0>=@w`CTPN}m%$msMH66>M%YcK0wo#*=5BOzA3_t>+O+KDEnS?F``8rNLk3pp zDN1$wFitN+r}L0X4e*d3LUG@*A=iXdkpd;bBqotDh-kI`f6bfPvnoH-n>;1g?@B$UiaE|2OmA9DTIS{mH7?cW- z!QdpQYN-zmTGB1eBlk#f*wr1c>sCv73w5zQG$x(q=7apr`)}i;AN(205C8Jlq7R|e zn+kPv5H@V>eT>cBqbDB_BEJ8<@BQjmzuLwIy#7;<=iz-w{#T#b;>pX};}3)vMK}@y z+OAZimJ)I?W=1k3OgU1X3_!z>nuv@bHxC0L=((ZMNgLz=LQSJ#KO#qd2fz_u z1P9Fa2yo3^xPJ!$A1kYDV~gBTAH&Y)z6&$qHWng%i*Pc_oYdn8_i{e_{r%YWoD?Og ze^)pRWtjWfFBhs(0HTm)ybFWZsWu5)c@P9=oGP8P&iif);SYYCf-==+yvHj&QLuHbO zl}|uX6?#6;aJSwcS_dGw5Wpcx!b8WM!VsmCJGa0g)hzoR)CvDKsv!_Yl#>T>KOkKY z@b5*A}A*uv!Dx`Dv#Zg9l`TCGT(#Gh*R;mK48|jJi z^EdXcfVpj)$KrvqlkVRA|3CWOe2GNjokdWPXfqzG+wM+x%4jmRL#s4J0(ig!7+xg+ z+?asuuw18zqf!2j_PY_mb%9%7fM{h8R_6d^K$^dJOJ5iFVUbJDgjTz)nrO^MW4~Zd zwJzHhc%ejS_fkbJs&A>f8m)X5uLTjsMB$mY)5b$K;dqe${%l*m-VeR3O#XV0uLWq@ zJ#&Cpay0LoHc`zuN7A3&)|CS8F@JjhRBg*`zCnQkAZh_lrvla{lWW$S^e!pHX6N2#FxSqM)zf@XU{{!pK;V76**-Xk%~gM8>lGAJ?8axy*iebVmTbv zZTgHLrDNK#uM-FO@DP?Ne;qb59y}JYA*kA76slUoSqgxZsu}5V4-dqW_NC(#1nPa# zgj$Hamh79XSjkIXos6VE>Il9H;5ZgBo9JO}-!Rndbi@4KVq>CAsLk;?Ya1Ri=}xXr zuZKq2*6kumD1}hS8l_qpX=_|3X**tgndhbW@o*c6Cn0FKnit50J4^vI-BpIpWvTIo z>DJP+2+el$hF=vS+~ea_0tpb6OecZZKJ@|OY?IWz zpBs?g_y!~Bv46AIh$je-=2CJg*-E}%CIIi>z5ytu>$0#7l#2cK%FXm@Aw z9n!K$fGT6A^-=|3g(a8Sxe_Ehj(HcOfbeSawyk=tS1(^c_grt(iVHEpo9G6#N+?-$ zAW;Z#R4wTTayZd;TZDeQ5pehOqxDnJxL#n)M306C?*MR($l9#gK-lUh^qbL4dq+8h z3@y`}y-#cB^gRz!4?7MA_80Mf>`pS)Sh(ehQk@rq%RsM;HrQCgDClQK``vwM?y+5R z=!oEcfAbG%s>AWBg&TJYph^fj>@tBE`8GL%wD8LTLoR0ab~{U2Q5F=snK=0DD7`V5 zZ2XOB{~4?k;I=ledM&^uT_kbgvf&RlK3B;AR`VexvWCt0gPt8JPa80x<$RkiQ-!s2 zDKpJteVV9B(v?8fG)2igO?N~aw|JLAE~;efyzxtlxv!iYd3ZZpXA^5Mx}(2L(^QO8^3-dR0w4@A z%6``uCzOwG43TYEG>mt8!gyd&6~O7!pb-$KAwXrCk|ixhZQ1E9FJQ-y9+qu$5+zQt z8xNzMXQc>DH!!2A`HH!Pq*fRLWsRt1$#r->UTbzZNC8K=_W-p12pxv~dU>{xqvJtY z!GaFT+1u{B-4DgVsABiG(Or+$eKe9geuLU|T4|4O*HvVzz`vA><|XH1bo5UicSg4t z0R-mh1Q1E%yv(|Wzpf;2p%OFpJecv)ha#{g@qBgSWI_2Ba#pW#i{7jndkXl#JH-YS z^e$Ovw{O9ov$cye9{>WYjT?nD&+&wbpftp1Ae$mk_>q7)f96u;T*EHmh~U9dXfzXU ziL`F?7Bu3Bg4!|v`ss^zf$mfW^1)yQ9_V5of$`xm`odSirkTiCfe`WlgD@~mfKvv| z9vH7tg+-WDGmuu}D!Ck9ayHV~FlXmIxJOzf@6i_P90LldI-sp;&tE`kmW#`D%jS<_ zr~g8_7SprW!xEOhfJF?X{!qC#56SDJrUXe4;iAHQc=snY3SZjL*on#k0$**6M-z)E zcwgYNFUTet$XIvs-3@oh9RY45W*Z$%fOx0lEYh=Gpj}wl7sS&DXvgGWlSqg|xr}kE zG2R?6U>(p19XP+kWF)Vq12AABv-; zAYCrapjwPOhtg=QIk(p}a9cs6S@E9Q-S1EdTSOGSz=qWCC) zmiHv!-Ww#UL8u~0v7bNdkpvDfnrrnj_Uwc1s`~zR_0UUO_#NjgBzLcfd1CPlE3g8D z0K@|x=AAKyXrfS3EJ*^9DDs}9FuTjiCo*!o(8vlM4%m=F*em4tTaYvosBarY`;HJ* zP=MLIV)a!7QHNbm-TZ1QrlH@N}K?)R~XuGOTNw3$*^8~m2*t{%@bNK?w zlJ}GIHKn@Cc*(h_E>qI7%Gp%8yOSoAY)iJh#y8FQJ%T8h6ww}{dEv$Kw`oV8R_~p@ z+b856y77KVMWJcsAE(RYQah1Wa;aa#ycHT_pLW(Q{B7=;P<<}YzDk+PD(X|x>*cyE zODW4<@b3A1xPn7>gw>B~t~bAzj;8B{$aK<2ZyCj+GX ze|~}2LKbJX`o9e_Vc7ZJd$nFaj#PN zv-H0o-_NA2*vaLxQk1KR@zAc5^ov>coj)E0$u0t?0`)|8O_RDaHvvNCVoUWWx$#Ep zCiBTR#KAObeh^`3eQcn%qh)$`?o-HPkeqOc3;fN#emw6&)E^_i@aetq@t7B!#Y6l@ zij4voXgyK)u+hwe!2KG?^*6GOSevgGvPN!~>vYxYRmi273Cw#se<&{`FVjj|E@rEMbmN3X zAOpN^^zUr|St3n9#oKkdsMCy-+FWx;#4=jYPys+ESvzRuf~`gl*(-zBU^jXI_pgSv zVMv0zmk~gMj^_#-FvS^3p1ZG(@c|HS^zK1eV2`W<2OJDhKPEvnMTQ^+1cE3;^x;|D zdrZMdHR)P0yVavgsS-$2I_gz2!YKp~L2Uem+)fOac8sl~P(*8P8bDhNR{)k}uDyAH zi}azkf%+v-_se=S6LqCPc8*#^)Oa6`(O^=95d#LC%I8Dz`NBhqH09wv;-by7{$&R<20mo?roJ5QuOJi;lboL4q-1~7f3722*G z-s?2!oU(k*rJA_>9*esdYyIl(K{)%|MU1}{@Nld-v~qhc)eDdml{+IOZ5?=7@?x26 z6Bc1d+~X3kW3)bTDp9O(Su&vCzTFA~y_1;dh3EDue(b`;@f;k!f;G{{7092`8|dlY()2~e|oYU=p>`ME5a{BC|Idx|L)M0{(y`ksXN;{JH$ z+=u76uJEoKQ3w8_bAJLqmqOU9se?u-rR5J7V?6=OQc6&7NiJe$)qR#l>O1~z`?-x> z>h9DOaava(WKH1ZQf;Nz^acP|4XJ=TT1}IS!K~zsphA*Xi+u|a?FM1B2uyTNvaB2?LSB^YmXK_g77Jz{oRNi z$VH4KDHN}```DSpkjE4J8C`(8B1w!47a+QLnHHrflpuTC2mtyK1UT!28_C|7m{MAD zzMa%~y|o)kN$RD)Bgv#IY4M`_7ec;n=qlWawBzU<=_^{A8{#b}2#7G3d0xCJG^th) z3L&uD(C|MgR4CaiT%vXp80B7#Vt>6`S;go!FyN_$I_-d4$Y~N#vXNHWhxX44{OZ`q zFx+jv*Fuhw3-J#jTgE67w3dd#jera==eZugA!e;W)F46s#r6m`;IGZA&z55IQZAQr zy<96j`FfeI>r%+K)}k)MNW}Sumqyq}&-g_c`jV!aE>h)o)f%_97V9-LpHyO;q&Xgs z-sL$z@G`tEQGS1yx9gi3J-|f-YW=ctmlqvO!r5c=y64Y9RmXGSmLg?)sgQJsi(*dV zx9?c?w_+!LAjCM9>RtOpK7~w4YxWUyDSr*t_74th4>d^j2&9DQH_TAQtIuH&wMQI{ z;B#U4f$5xh`R@oV#g2b#A}nUMXfB$R5J0`y6M$NG?IMCd{`k|&sFTOX6?`M}@2!WpB^LwZMBFnefPrmrj~7Q3_BpZ`&~-#dW(>8-Oo+&Yhy(S0$He{=SY?-2_;M?0R5O`5dW@TB*JUgoN zsd(I0#5rd)M%z?3{Yi+d{FFj`F7+6XZ*Ske{ed*k^R3vomsumvX;aFn5#Dq2(Ht?L z(P)4R=UU#tnNgDgu+L2m1>aT4y#9^h3;hX)RIDOSvGM`!=fpMTz_sFd>Y z@o7d+v+>bBmxky<4ZGxs50X4;P5WxX0V7*Cyo!T*AEEzFL!SjB)u8(n+kYBCjW~XQ zi2L{)gChza*u4W@R`=gUz|Z-=b6$w{mD+!CVy%G`!C+Al!=UcKaT&SRehnAKL$0Qc z&!X^29Hx>EbSLoKwCY#OGkc#y?Gou=ApKU=j53RwX-!q7`lI?Wf6!b?AM(#B+hg%R znU^15H^S#fF2#D&!@Bv3N!FLEkQhVDLLMeFNf#Cm>Y+42-A%1JT6Dz z-PjEigW)&D@Hm)=o21E0S$AqFY24W_obB|$$GNB(Lg4@tPh%sTsC_*eH%d9tW zd%a%fnPx@+jOhYtKhEp&J3sd9RN}FQF*+FlqP@x@#)5^(Rx3rXs@0?x7LeQu?G!6q z2t}%FB}|^eQLL=ZK{6G>FNYFT3p-0^76oD5Q{!VP)_;3-$d zjN9Jflj->q&=81^{Ivuu?yOsKC?3JH?tp&3j$tZJGsU8WFF-)mW40tn^(5v%wa==` z0Hc$KF`|1%S4DSAM7+!xaF%jp#bD>)cCph=Jr<)*shDLmBoxcI(9GF_n_Zk_1$iibW@x3@kd4mgmou z`OZ-JolldSGEzuh%JT8K-p#BsC6WNlx%M{&x76tCdZh#;B{d0|X=Y+w!l=88pv1%J zWfX3uzUZ%~L})gn;&MS!O;Zv}TlPEY>*Fy3RCi^>qQmsU7aI}4qi$o|djq!-$0dJe z%+^S>Lw)4iw^geFlJhcGhPJgmYfiA%wN-Md$$S-&)5$YId;}P)1cPFj=jU_F`sNMIrj%}! zKIgd}r6kZ^ua?oFT%M30Y`^i~ zwjldwGypAwz{b!$I`Ep4d;w8#{LcW>!{>uX;dFPJA6^V31RITG!U<#c&y^tp`nyHL zD{(CJOZWK-=0+Xw8Fw7QaC9OZBRLBSyAt2y`NS|r&L_YoCUdmUI(gyS;~BvN@u6ar zWp=U@#pZbt6TU>fIh~sSyYW@${}7jPxDCQJlSM!}!B}KXm|D+8hOTh2WXcY1DFy@c zGKn8^Ah*s;iDt!Y_T003Unc9-8jdM4W?0Q;^&>C@qR=Rx?zFpNcI4lI5=UlKnXg7$ z>Nul55WjZIbme=81yLbC0NI8DxOZ*hENA3Yf4-`tIi!p!wpd+v?9U-Ogs15hy}*b} zFFXhU+<8U6JPNGch|eSj?^k$IOzD6BpS`Q#avbS_lA7$nWq$C%{r?|y%rH!e-5}jv zyWKC&Ci6CLU{S*mCY7pGwYzT`r=6W&&cfxDG8zDs);eee6;GXpOq5Ch)jQ+$Q*i{b#55ImVMH8Dn_}3$`TbIen znD1LlQvzLd$+URcuyA4X23s z;6cP%Ypf+g@3T;5lH@I%l86lSm)07QNN2sI&=`}(D-HIRgOp2nxFRG{MPDid4?Ie5 z1|Q9;TOH7s&R+RQ#Mxyq1L_qb$#)2+{ZbV_ENDq1M`1n?jg(Jt+;QK_ZA8x?V@l10Cabc06$VU@ut3G0~I9(08=kI>qzLv#r-J z;nBW6I0{|&@WnH=vz3D1ef26ux$*~&bJr*`9bF4U#@KmdToB1qW2kMdHKz9=gxGamjEtn~ zJ?pMx@3rr?8*De;u-xkXsJ z`LR`DD~Kuig7 zkvpl`jl?1)G7Z5f1w^8VImzU?Lsn`Kjn;%XZsM_w$C?r+5t&FJ23U(ks0+(mU~NjN z+71c=Zak0sQZglPSB3YLq=uR;12SFMUQNrcX~s~NZ=Xt)mZO!0PG-E%A<{|{100o5 z!!h(RlGU))0Lkws$9{IzH15gs7sr476*)b5`SR%a2()_q>=_t+d3x4$et&<@Y;Qx1 zSKoXaudd#F^y#JSl@#x#)eA(n=y_=`sB39kchrB?Sc#TW+D@FfLm{Fwbo#*ZW^nIe29JD zdp|fM5(sY4(f!Tdd;hs@WM7-lxdo8ymMo_k+Y`TWbIv*M&-*uT&N=T+&NB|m>TUv6 z_q-zcvO&1J$C4#k(wI-fK?FZ!s{n#Hb?ct(0)ejKi2t zfo)zGUsdoT)JfV*_G}>{cZPoJb=^horfo<-_%$S0Wag&KSjTtYeRsKBW-@HmS^C?y zILshyY_6=N6iAaIHgcw&rNm|?yI-{QupV2r9Uu@#UY}#FXA(ErBTe4;IQNfXGqzk0 z<6^mruFG0^O-SspXBJIw2}fpD9LD;#t&O!ayxgr}a3tR-M*C)y|Na%2X(xB5VsH1e6F(-=k)}S}W6bw#*97G<$2%);#&G zT1X&)B-BOpeHRhVY^~MpHg&>FybO+bhp^cNXVN?y6aDei!~p8=*JJ- z8XtWpF3!{iAHF}{z5C$e{G*RQ?pLc%|M;g5KK$U5zxna!pMRkb-aET@@9~32AARq8 z?~cP_wdzh*?c%)Ldgu07KfQY&7>0ohee(T}V{A`8zxUk_KkzWt^L0rg%5Z-1`8f2q zPI6lVdCJM1+j{?fmW!@*`ym!WQs4K{cP;I7I*4vb*w9if_ah>i99x!7%e8A3Y$>!| z5ZZM_NZ5oO&aT$JR%Bs=Z6r`Bi^bw3N@*>Dh}Z=u7AMXyVFTJ~V0FbfHYJ=jp5-Z( zQZhm*Ng%$8blT=w5tA(n5h0wm1U9dLO%3w&bQz&xs23L-S=(AdY&Y?`CavcM7i4dP zKy0^ci@rZuol57!Fk~h#gTwpi2ukcqv!(BMy&oYMYu%Q#$5t;F9t#A>yatS=mYLbs z>xc@7Ojx?I zQ^qdv^Q7{kzNe-Z{B_TF%eF57y`B>x+-N;;!^3s}_2)sRG@9T8mXzvbEL8 z>fP^t=XAOH)1N+la_|0kZr|>gi(b)1wDp>?N?l2}VYoOu`|YoPbNa!1w{M+rvTAiK zNPum~%$C`Pv88QL3Bs_OwsC8+gfKR)Ic6K}%(TFSa&)CNLXt3KBU`rSY{odfe=iFd zx^?UNK5=GZ_Wp0#a52RoMF}=rDOj28eiK^)PoK?eoQ=gC4xVXj(6j1Vsg$q*BME`w z)hOH5s+Y9siauZ0nl|%iwRk;b7iMwba8PJ5sSsiYPke$KC{^{7-SG{9#?J2US#pD z4IuC{G`p4(ecvya%Tl_F^?H3V)Mn6rJ34>9i7NM|&zvMnX>6htMYy%UtiQ2ddUu~U#rxm=E)k%IkDr|Mix1y#8XL6@>-Fu`yFdKF4{n{FK7913 zwaTLZ=z|Y^d!(k%F5E4;TPLSoKRh`*d-$ki5^?(8ySIM$eP``YmhawqXB^hI-o5jK z?|niSXP-XEVOTAST3gyDnJ2B?4o-1?_SwZ3_fDFNC7hYAv~+bk4T{n)`WVOcIM%EU zTSjODL=vr*R+VX#Le63tnn)XPt*$k*;RJ(cy{wG2G;P}?$*iqrOSdLmpr8VlRKk7d z5)BJzNoMR{!#CXedU9wrSOP3zAEYSGQyQ#meN4&B#bWW!JMYw5A3c7Q=@Ma^S3|d0 z^k+|x?xtGXy3zyE4dGqiEJo5nR5Lc%SsM&PiRk-{XN)u3`T_upFgpVI2ASiVj5b5D zU8uA9*V7*bz&MUOSD{vG8iTO}C=3$F66OY8wM-HSsI^{PoI~mR<-!2N;Fc*oOOtxT zFcJvOOwvKm>V&h$LWnPmTR}{FY|y^F_NlDxx^5UYW@)XN8#7M*S|{C)Qbhb&r|-nr zk<`qVSt3G8-h@i2p8pl}oXXZ5c^-VjfF7}DjqbHJG<2okHo4@mUIXZDx>g;LRH9%D zv;(JG+s4TjJ0f-wvl%Vh-m-Yor5Gl+gP2m*?Ubh|Pf0dH2w(eMZA&VO647;?z}R+u zN~F3?;(ua*;#%#mmM;F-^r5o%ApM742;q-Jh+*?2V;tP%A`cXdo^mC2%mg1dK zR<*SU4@+kX|g4DOS47T(Xv*xESgd_Wdvb|P*FOJ48s;3HLwN{|mk*2k1c{vGHP#_m!yd}XR6e=?51Td4^m^J|OXvqC0VdE{Z5BT!_ zY13L(q1m~anO)z%_x}55XJ-!{JZRY>dh@F2E-r?%i*XzgoeAUl^}{c9^{8%q)Xb2K zj6E6Ir<_7*w^rA~1>`pHb|q;T)~z)#P6*415`+$evN&{|*3=Fi5&b`Grd>qr{O;>6 zm3EiymYHcHEvTqPbQWN2r5w5;!vpk^?gPw?7-Y&g8rBet#j-@$jj^`e%SZCfn+lac zMkHV)VXUtcV(%-JdJ6;?;ZPW36q`lJr>Cb|Gvsl-Ubj|fU&haQJg(*o`I#(lb@jxL zgPOIN{{05vo6UE`D@0e^F!klHaRnG*+nHS#B~+adQp}Gkeuc|fbz>Uy9D`rje(c3p z%O2|*dggAJzuYRdZki}D)LPfW`tjdrJrlgd}`mYic2j znyrnFPDMn<3(%gUzC7m>ZP~=ZJ{}BD+C0{K!Qv)DI^7ZOqzxhOP94-A&Df@ zc7jl0$2{Ce*jIwegf4eVOivF0+ae@76D{xm^q0|+{Xt^!noTJ0Ef}Jow!oAAWI{mbCWdX)*8aJ==Ax9!IMPuRpu{ z#e+vJlWcXpuH$h3-h=gO5k>tll;*7C`R_lK-HrA9?){60kGy$ojG8rVDIyFu&iy!> zbQ^PU8s`*wAH-A=bsmg`?Izm}0h$V?So>|3Ndm^{RHh+qV+6>MvV{T_#eV2>f8eqy z5lsI`mNn&++JLvreOl9Ql7xiPDXHH!@gv4@)3!Me+#*V}TJPV#H>?LlL|`P@<`vLA zE!Wo3P^56qzslur%z0JJut5ec#jOwuY}v-G)-^WnyKUvxZh2uNWGNwWL&9N%By5Di zU}J1SMCtpoQM8LiL}c?e!>4YU48)8vLdCR@5hzR}6PKHNS^aW`%iupdZ~~Y#UpKJ^9}~T#B1&mapcpUiA%U%y zYB0p~ztZs&{G63r0d|5VL&MD*$h4(EA!S-vAP5xu@lR}03W*(-g^eW(FkneSArf$6 zGZppWuu1SR44x3elPm)Rf)r+s5D0f^KRg2)uor@QZs;J*Den7)#@ana3d7@bN?7!D z)RS{Q{fx8o`sA#WWN6JTTdmIAT-z31fY!s)t=Mj@%d}H%U7V#vWZPw5c|8JP*9w3N z#!7st+^2HpZcC@zen1R>u#8DsYtA$vWDAfT$jAT&he|1lGaELtzFZUZUMLyu-x-66 zKq912NZ1xK0-1<#l*~4Lo~?N_lcc@=6xssV*|H%5At{7ut!1VUg9IN`Lefmak4tuRBPAk^NaIZn=OMU9i+dgd!26&#!1VR zw2M^CG)ImcU@++GCWL;wwC+AKeLbuZUDuUXa!fEs-nQfESqS_rNiE$h(ON0dl`hKW zr-VSpD#hMaa+3+~Hvk>D9Z!#lZn3Z+y&>H)!8VE08$bs1@?`6gS+I`x8pqBgZ4;v5 zY{tTt2!t(pDz8a~2*3hQ^X`IR@(bIGC^wv$8485}SprYu)L(D5QB#|)!R(Ro{&j_s z`mPfyl0Y_~%mTgL&d$myK{bZf?6P9>Ew zwteXxZ|s9!bc5}jt?d;^#yd}`#0P&BEX~6RP1lzaw8}W9bHB%ChioU7Nst6WAscOp z?a%btG$D26uUSMAgk#g1y##DIv{t-R@; zvaO!?wT?SMyIXpkX@jk+-Owo#Q0tTn)fHLFYS4lvnv7qw*QYR$`BbyDR;81zNm6g; zw*H(_0Ok!rNQuleJrxpPC$Gp~xi^Bo?^dg2*OhS`o(|DCj-Z3MY4J95B<)N~iYO7f zs%(!Ov7UP&qxm*5Z;{BfYXweEPy4Rh1>JGvtq(4i4)+@D*3vmrB@t1&QY4iwLWL-4 zV_&<9WG-kYt(#y|TXSo8daJaNBrK48l(Mf1OTd2ZsM+f^V=`f1Zj+?L(zpl-h?vxl z1PWt2ZEAZ;mWTvwoO0SmAuJzi8_x?@#y2%uSxguMVxFmaRovPkT?|7PrR)1e*VP14UqDx7;8*f)ba*;cQvgLkWtAqP`SCwwkRv_qq{mOR|l@?7P!F3GgXE z$xcWE3Td8_)mpPHYcC_2*#+-~ql}`9!0c?48#a54eXvAXk^&NosSj3~+G3fj*9e8l zG-7T)Hhr8!;Yc+dxP;k>G0tW%AQTdkv2Dy1zBzxm4hn=1t~>J!)(i6N3#ko8dugq@ zTIE0a*WddmfBZef`8fXUcMpE@>qn2*!t6) z?D{UPu{O}x`OSooK{|92m#|O?Ef-8FB^OkZ%h#7nBfSZjak#=SSGRtAW<$h5k=-L}3V=^-$x<$Xw z@@d!{U||9BL9ocHSQc?4wtyi)Oc-OK%L1D)2qFrSAui*KjSaYq0=vAJN;-f{gD|!Z zliaIVbpR#iPkKP3>I`ELEhI+ zz;4PXPUj>9_ot}OaKLKX=rrNVKkjb?*_mFY~AW6RKvV6xfr!8=G;2*5iG z(lr(>-j9_~jN2B;tj(F&?3q=wJ(W%mMO;WmaIjAtBI(*QN4~82%Ag2sG1$g4i6;aY zBy368*cSL^i`Q32XNw!KeJC4v9?}VbWJ}`N*vnd(7G?O+Cnx{$zy9Isghy-sv;XP# zFMs#{2~v`6B-=9S+UhofyuWi2w^rT1@h|-Kzx6Nv)py?4s^`H4|Lgzz;8(x-L)U4s zSS^<)5hc?*Tc}z$qBfg-xI~`oWi22{$*SZh?=Ak-f9CuD^xwX7f8>Ap|IR=C;}=hr ztwFCfKNC-1TWJsD2&`7i#bR-KdLn77;qzt9S@=a)5ZTnAE7avUa&zt|B7SLOV<3+L}O$>&%eMTf>p*+rf5bmQobr$dMy2jyH_s z*l*LncL^;Z{-QWaTp5D|MtjY|SSuh136K;Ky7BuNNv*XKp;8dGu|4-Fvvh+Y;mhQR zplpZ(e;L{A(Sf`a2s24sMNDZLh7^*JksT1o9OkMT3#Bzu zhm$hKY16Dw1hF3*$!QrdX=H2aMURMd%FMBqhW+acyU7w=s5xvh6Xr*6GA*CUH%}4M zg1r_qhlCUnLcSq3OcJ0=4)dxpaQL#GP6~sk0=D=j>0U+H1rUl-ilMdkY_(j00Yn%aRLV5t1+ zxB1hb{PE$#_3c}CjI^En*rvQ>8mG5lG=;+0HyPv&<``^v3t?B#O#RNM~hda)jAt(NT@q~LSKvCJ_R&rafY=hXk$4?p!U;eL8fAfdmfuI=O9Gc?m&2?nZbZr&8EBj*6?`AZ{ab4@UU7@|oY-eX@ zc1ne5<%rPR8Gd=h2HD=#Y$FPcygsBi?uPA%Kt?9t+Evsgd$rrL^gFDQeu$40VB5~M(j6j|J(0>{>w*W>x{;+ZT*$TajZ5FW{<@Tfv%SD zgaSn!OTAe7Y>{ond#gAr8azE`{tC0nryV*R1z+nuJAXCJYqP%I*3$E+CRu5FJ&c*n zZmZSmL^T~hmAN-Sl{Ug%+@|R6s ziv<)RqZ<{>4fatw7<%EZ+sWFM&{G{d4r_@`348gJ)%H}#d5DF;^AKcn_2qPC zXoLNFLxy!cEU%3lLbJ5r-cAg9W#-5?V#wUALsn?9ShUvG7o!b&^Of3>7fXu@i2w|? zC0R;lXezxqLRboIVgLBoTI8kY+i^q({|4<_3ixFmGu8|PBdS7XY2(4DjTI)CtT^9o56hIgO z?6tV}CpAoqt-+bLOYHn)1KZ#Y`*De z+7=cH$+B_gEQCkWylZ~{;Ql#ZncXU;9N{bZUIsXLwV7hJ#s23?H)}}gR}rsco`FsR z^bEmt<~k(p+=(UCT8CkX2n8pbBH^j=*^|e4Hrh_NzF$1uKX!TI+;T>P8=+&lmJKlj}qeM|=rhxosLeeqxXkAM3AfBo>` zSy@|Wb!@e|HfM%0N{|neMQux-ew{ebz9}VGN~xKawimP$L7~%LMgkxqxrl%Y4L~F$ zK!8LBhGJqrvI&QwM1ra9AcQO(x>zA2&~*}2!hn&noVB*6N1+bqr>F0nEc(9O8frZ| zJ8!MoKJ*UI%r!ny18~4@)2v!6Pi0ye>-w)S84S+ZN4NfkzvE=dy>mJS zoEQlV5;tHFo|zs9;ecXVX20mSi37FoGTXWi+18ELnsaRv(e*kx?YL;;fC&?_rzD~v zF-KkjLWpPYC+b1;+qxHG~X{> z{)?N>^;ay_ukx(NMK>witHv*ufXv}yT_To?W!4;P#g_2vi2=A~lHBZ({O#652JHEY zL5#mb@oToUFr;htr{|Y$Bx%1lTLO04*;7(fWH92A!y1zCwWUYhD2Ht%9UfE}=|E-M zyt$aKiKMTN1l1U44U`HI&B+YcLagy=G)Dg&Lqtx zlJ=^hRU*n>=sFEP3NuyJMzB0@;hQd@ggewZEX=|O`)fyq|aZ4AZ9>FusxSbF!}#re|%c5#6X zDKzs~pRQJ(7irO6{mt)=^2S<-5wrSE#am&7-pr<&C^%=UAj8;lg<)Kr!xZzglGtJ^TvxdO8AXwQq$@Q1Ga7b82!bq~zpxFts(+UvS(@ck%B7!79hh90(Y+EWa z-hk2kVCw3^VcX=F*Zpbo48DA?Jt;8W%UVqeSX_%_u(9L_1yk0}K~wjv#t4??zdW{` ztpO2%jY(U!u|NXE?55WT9tJc|w9rGC4S zQHbJp-nWat`l$Q&|HU8whyTjo{$oER;Qqk>`_r@k=70Fv|M8PA?mk%#$=Yqmtm#PE zfwPfSF-i+!ftZs30va?im0=HB0`S3(%d?g+eMlluCJl$_6O2)n5;bUz0tFJPogu@d z{6va0gm3^#kzc-iB(f0*0t+N2G<~hfipmmN4ms+4wIUSD#R?2=wPI%`p2abBespOj{|%JgCdL#&6$~!6wzu6g*fsJVq=#Q zo8@bZuA_5pq>+7a@1{^FAZ#otf)J7+cevujM99Wl=J6?F!jq~6$)4>(hjvF2V*kvU z#TX<60)upOME_iEZn2#QuZ6!tSvj4#=}e0)+1l*o4*G{{Yqs}>Tsth#yli9ejGV3H zmE^d5P15Y2U;|hLB+fKWOlB&byQxGc_)12baT1@-lZ(b9AF=eYmWjkdQ*Veohv9;ESusw_eSW?R@r873v+Mb-9AvU@fM<^mn zDcDYHHl_-6<*h>q5@$9qx%l3P@o)X}KmL#Z^}qf5@6*GhHUH1g&;Hy0`Dg#{Pd~f& zNM~aV&XKgR!)}qF27^^noxRz>H_TqA)z~6;a$(l;4zeVaA`1x=Ws!2F~Wc0d83i5fJ>DA4vrVS2z6a-m2GKQ%?-GelOdJMqChE8R3aop+B-2d z)={GGN)Rm*fo;Gbi@2fkW&jkkKhAPrBnkPN%A0{n+)j_{+O7mK!x*ri;YGSl{zx+arA--8XFo{@G4EbFoWjt_n$P^PmrjIe9Vmiua| z!OZa7XQW92*^NvL6HdadGn3;Hkgx`R#_m1a5Qbg>m zp=xWb)}66LYps+`l8Qo?`lr=IvJpaI=P+!B!g6K;5n3%5r>D0{l=a2>{QOC+BLWGy zB5Y~dhH>npD>I7L8j+GH8qpj02HN{FnR~CeBC4`Go86+wpU5L{P6tZ`bSblIr$xq&Sh3pw z_{x=keD%`R1!RWJUHtC-wO{_>*8PXwZW@n)WyuIR2_XPQU^};|$zpmcM=`!zVtu98 z?*gDG=pZ6Im}O>DlboB)&8)1d%`M;B_EN-1Zhb_^1L+`Q8|hRQrl<3Z7f&yonk%y4 z>Zeq$Z>Kx=w;rx;@Agv?7>eW(gcw8!fnX2o+I7+g7bdZLvP=vBVFygZik$*UL6FHCjXMTKm6v;Xnk zwUy0&zaLYXf_d_;*`Pg9oI7*s^r_Qh9T+_eZFz_>?CzCI%WHS;FL%qbSEi!Xj-JfI zD4Ja=_T(r4 zm<9P@X1q9e`piVpo}X?ZxzP&X{A}liYqKd#?d|#MR&T4Hq6Lcoxo{Zl60dHZ?sp zJ~rO&jGKjCx4XHuwY2u<*Y?up?rvoaEmsya64r(92TmZ!lVXYmTOkAtK}t@X-~=c5 z7DAcIvJ6>P6a|1@uh+2VoZzJUvB=RaSCRon{=I5?PC$aBWh^XsU@9X-j&xgswSqtb ztRqqhhe^Y=MXNj3fPkQ;yWQm=seK|;yTlJi3hih4HkAl99fCUcF5&_@ygEidB28{o*s1e)^*e zQ&ZU8#lQUa{_ozpQZb)LI%>8b0NE?;W3+sF*<76=Ub*xkde+h;zxHT&MD zi#PAB09h6?$tVypAkI(Z|M-&|PhLk=;SayxeDe?Y&s>~+?%5}9+&DcqgFHvD(fA)O zFWi6F`}pR@yC2Lp|Ne((W^r!HTe&Hs74X6j&d!}Z8#CN_i2wI5 z-(6eWlI4;cVy4{-bCb_LdFk5K^JmYF&Ca3IL6*Y|RfS#;t82JM=+ zrDxDcmH+U_*-y-6P!TOU@!=eP!vU3RgxZ;i~l&E;t9TOG;-9M z&>_*x-7N%ufp0m2o&diATRstQ9j1_x2yhUTP!L{vp|C& zm#5^(jfhogpydczMj^#G+$S_Z2ko8J+$Yk}4n_J>gpzyGsAaT)9otK{ z0W~hX!z5XUZ-ien{8A%$9g5yio*Nh{2US)1;hbhc-#2Dkk3GwB(#mqsJZ@|x>G3P4 z1DaDNF{Qy^V63V^-%~~Ten=AKkh3;DfCLd!v|ty`bbkKI>3@9Xi^!<_ky`uu`%C}N z-{1e}=2pL&8V{KntY8-1qeCDBxT);qc4?t_`Ki`K8+XR|{shD?Cr>0GW3mv>%<#EO zn3=`a7C!mV^5B4k;D8fJ2})#`of`Ym3uhNjpTBqs;~iu`Oi)5zU}g;G=5hVP?B%nQ z|NO>>pDk|hl_f?ldm6GJjE8t>VeHixue|ib3)e1jb_^L%Irj4jFHYdf0-m|bC(d1Y zeF}eg@7~IGDI;jXWg4`#eeJBh{0s^}aooG7fBx!|PhG-X3jpTEFh7eyLaxn1d+^ZC z%4?UV{@Y*N_~8%cPtBv90f5Ikgl!6!r*U!Ke(>al`E!%u_jc#*Ru#=vfO15lHx_m- z%ypl8Vk`;WwcK~7fBd88e)!DzDnfO7 z%6@nS=VsA{8303Qz@JRtqFZj5;NE zfH7R1$CDRkPoJLn&6~FepWLf<2DGAXj2+q`T{}DZ-~IHNpFKNsY7Rv}6ad(8oEdBk zS5D>6US4?e+QRJH4}Sm7t%vJQGQkAm`+yVFML*^)Ng>dJ#VSfCVPGdX!LgxVHAL_m zn7*p2D$BCn{-V8YjB!5=1^l~l$);39?Qc-?{RH1Vxve26x!b5YYJl+d!NX@K|5EIX z{%Qu(30NN^(U4_EAv)Yeu~~FmM^qq?#-QO3?t&BUwS=hs6$l7C@K_2*pu=>Ck8tSd z>7Yth^$KPn?XYqi4%e|ZDk3GSVE~4{X|e_dNm&-M5FmL<8imdvR>=qoatGisWm!f* zF4i02G1WoAP955C)qGP_U9VdQ6F>x2tE$bjnsqKSQHy{ihV)qgkGB2%OP2vb5sipm z;x9pUpw19-xhvRy3GryaY{<5^Z{ObB{K5#DVkBT@!wvRE(e2~UYz1gPdbAu2l6#D? zK3}a)HO60Qn{;p%kKFoIRqe?G&13QuA!krv^W0~>uU%D{p*puP_p?_{|93w*b@epc zhK+5!@$UWq@cO6k-sx4C3E6m=kfmsug-lFNnE@dSxsxgjbSFYNfnt*Vf&(4yA1eri z=wQ{d-c-;e*iA{*wiE%1?vMixmsbp7;q=VHso9A!lqDW4ZY@9D>F!jWcATGY%`MDN zO^r{S!~9fccBB8l|LNw#o?A{0=f;@JPfg&p7cTt!pIl$r* zV(k2dvt#4gg-M%w9{gRS1+@Dmz)#;!aC58E|1H&Hv=-CoWsd&>NIc zgP)`!|NZ~GdFSqqlso7&AZFN|%y%a-Rub3Ghx4<~JaK)zQ{drpT3LFSs>+Co ziSY~PPR~rBXd}OX_>;NY%e!xXbhiQp^?SRwKHKe0`oeU$bYUDsSz==&t!%9AaQxGU zy?)uXV5D-U%L|jg`02Huzi{!1IphKT0hU)bH#T<%WhK|xcz)*G^vvwk*>U{n*?g@1 zq{qQqAKvc{`m{C#qt8#~KYH%sPhXh1G=n6pZ>H7N&5iY)7|Seha;i8rKR!1*f2Pg$ zH6TB?yVKv^eK07M6mWv?2Wp>EZG0kWcF@J@$Una)IKc_{^_YPU(k~$Z!zO(3O7fH(FeCt;qFkgC`QM2pSq`7D6zSyU|G2-F=44)7=ArVZAYN0FAO&>-tLYaLXed^)gsZ z?(1wRcNZj@)k6{+r+(dm2!fjA6Z-3|Ck&7gq(~mS3LAT~v4|w7L1h7w&C5*C?%wY9 z_V#c?nvyzcfH!R8j~}y5?pX+Rk~>@unY(XoZvk-kMyQzxKyIX&g)9r!EOL|NF{b2S zpl(Md0iG1)4ucCqo@JKhm27n2_gl*b6vPD6?0oj~A1(aTt7o1(jaCL9;L}^X|LNDC zeRO-POp|6g5#3Q$$XNt48&XiXgd{~!GE3$;DtUs`uvI{#cZVk>!c8s&%R|Upv`kD= z6g*LaXau}y!2=g}@Xk9A-+lMWzFVpaed8q`rRVu{8a0iC(pn7{N?M{=T7DL z#dE~=jsM5a@9%H)2A)MAAP}vftN=3LsmtwtY;Okk+2Z=^xW-@Uo{aOL&+PW9jYvwzdZ$Z+Z9Z*x(;uc;>%*?c&@x60x+dk3L_2|C77-mp8lJL5e;;7A{{}c;)$P zH=dZDZ{yW#=xk;@1>x)p#;ZuEAgcX_&iFRi)$r{$^UMBOmgcwk7;~JW`$HvZ?vuz7P^J(4eHR zsNDXJa5B%kANnaF5lId*WF|K`9RdgjAdsNRWJVaBNQrRw{bywRQhlnLDysh#r)E7Q zqv}s^bNM{`_OdOBntu;@fB-0S^w^dcf*6am(Bc$8MfzYz1KBav4wjZ&~{LJj)i@LYjtwF(PVa>-LlU>))c zNUVD@Q39aHi`gDTut>wit)asNLJb}$<5ZDnc^0xf&mN(sy$xoYQm;R#Vu~?=6zY?f z%CZcGRuMwpLL?(FJa~iyu~P8<`I-C&PoMq8kIq~_hZY3U49vz#b-IHcx3-N!iIHLz zLLdM+RX4u@93*Kl0SR!Zc>xGgeTWlWa)SfG30M$Jh(L1sqn9KHy%n;R3cr4P={LXr zcyY1UAK0KWA$B&()y)qEz49M_{KTb&v5Rwj?T2SSytTA`ryFTWRT0W7=cj-2(&cOC zTW!Xzt({-~cJV*_;lbi!H&&exI%Qd6WpC`k?&|7F8gzg1^40mt@fUu8TX!$sS=qj` zS=ODY4Kfn~<5WBE3%_}D<3IoQ!#nqudjlVv+8CR7xUpUCbgNd&p1pD5r~l^F7oR@WZsE1($Cj6`b>F;gRnd3L zVxOkK41fuRp)36H-MjyvfBE?4LmO0?VwMNl+F*6}6Zy{n@Q*K@n#krlc=BBP)VM9K z_FUOo7kl0APP*nik zE#6+**{y=R!7W4e>4U`wOPhnAUitYe*JcX*@Y=+u&t6^K?l13{G-ww%ztFyP%38q6 zUjNMx?*510-`(0OvoIbcdX;f-dv(RDYHplYu3s*Q`RVq|WanYnj&6YCdxR5+fKp15 zn~@~Mza`$_1SdGjjXqkc<IvwKI)LMlTe47bUc!UHAG6oYtilb1Wno=z6uquA0BgE( zqz#Xb)8xs+ACVLUiVYfju&ju>)$EaOKVI{yn%91F&NL*kl1pC2$ibi}ax+UQ5vV)* zTJq$cKxCO!(cliig0(n&4_@GC8A%S2v=9(1 zcp8ODC8^Q=^3Yf7@X>Fm$6vv0n?LH9viSqvE(k`F;m7`ZI!6{0GyQQg5^QyPUpY)@tGH%zjoyU3IU18fE?$~*^L`#KUrGa*cvco20={G zQ%EEU8C)Vk?w<(_!5zoJA(!d{dE`*T1s9a$K&XIs-s}GJZ|>c`(=DqUR(NJfMT+tB z2Wzc2?_M}FJ2zdlTltlXxO{f}v&GGBL>XfX+fSW4f91@0$I$oe_aE*4`ppL)-y2l1 zWmdrDW>66m-@Ut@y>a{O`7=|KH!BB;xym6R5TWr4%I( z!OScr$5xv)TZQQ0*PYH$Yo8ifa`(+=r6gnO{mXsZZq+?`bfuzL1(8>!l%Pf-w$3EOW#^_R!XCYV|R*L4f?5Pge`V z5l{!d^dxss;MTT2{ABs|VzoVJB{vvo5DHUahT?;^TX}pWa%zeqm;!lU+WC^9yqy z+}-Py(Lf$F4zun6_a1J2dS`PtPF6!!=E;>>?KafCaBlqi)oB8K;Lc+ChqrI8ZAsa5 zN)d!ehO82@T)P!N{B-H=y@xZ;oNW;oE_6fBpfVb-#MP~^x$1w>!~++FT2uDe>N zs}yB~g#@}IXSuf9U0UC*E{`X{9b@CIkRjw2&?-E3I-w|v(Y0jA@}koj%g{0lKuPvz zCwcY4^mqqwtgP#gA3XSMakr;Q`lZY^%!yt{PybRIfmXXfqP*{OH# zZYw6r)1$|)JS&QJ>-4F)`RU!YjlR61WmJ)zG5XTh-oL)H^wHuM`)YMFF0b~6iiBC_ zBEAne0YRfdu17DkU?l6z!4sU|1m7kE$$Q=I#Kc6aXp!uf14;PWAbEt`O2mFIzWw=e zLO23_IS-j+9h7=}{p|_<%V9rmD45}U4t`GuOwx>ms4u|UUkZsJ2py0ik}1Wq9Ap^` zE>B8H@}pvGGt23C!LKP0(i$3~c+dn0prEf~($H}UV}kIF;3)~nLg=*G!%Sn=JE>VW z%i6zfW^Ohlx5*RK*wTg&I;6W4&8#`V2FX1(^GO5@s0Gh@k!Fpg!OXHpG~3;SnFS?x zSW<(=LONpX=jbSZy_kWz?7CUT+#Efj11*rWsi~=n@rm_pEs<4Kwf~GxCkvtf2!I~% zY-GXZ2w$MxF67>0k_3YFwh|ccw5BH7t=s~gqtfA{O?wDAK^jNVb{*$XD?k^+UULa;oXQf zU;^B|$cwxv`u+a5m5OR$CI*ab992RD?mgVRb$5BE7b7T>7y*Ep!zyIEU48o5-IeE0 z%}!4g0aq_BoS9!<-Mnv#*RL#Gzcf8H1_PE?b?e@Wm)?pW@K>lc=%#i~I?`7!C zPfpH`g$~ejtgmgYt!x${7hqX*Rwf3SC6d;*v9iCr^l+o9F10(D8N>P03*)V=eq^uA z*SE^OUTSqPInGy}JiT8W;7F$f#7%IkvYK$P7teVY^)=Ba<5WS9gq3-^L6XeG6`T^XlG`C z_C+2xnJ@{O0GC68_DL;-#{ws)^I^aDsHaS*NBI7<3ZU7eS1^oV5HW;n_z5M+1$s*E z(ahzJz!1#k(Ip6i;SkvPALQbGL(hE?7%@d-2!UD1MuKp1_Y_9Gpu~M%*YsFORg6WR z13b)`*=PdfH8DO`r?q4B+ZxH;jG&UoqY7(NG;OsCBV$#8^xFE>NQge<`;?SMT8NG0 z?`AwZ8-o$p2&{#M?+{0vpS9pV{EH9QHiKYLUgYkcNI3R2XCCe;-JrQ6eV$+Qlo2$1 z)!1)&Girte&}?pY_Qs7H|dn-4a3A`RzG6&Ej_n#}R+wbt79lRI16pWfR8ELa$w9JD%r zf?w79KwoQ$j$n`(K`8q(idH{jbGzK$?Nv^ZFcA>zi!KJ3Vn~s@3cEqKqJb zEC)l;A7FWHuUDz6o!CYi%;-6Max8CUqivm@9DnWA7oOP5oh?HJ#>nI#Py``Y+0o|A z_(TQ>C<3M@rg972!N9X*Xj$E^cyqhIva#7q?u;RjkO=?`mNOKODzEVfX0V_vMjC$jNl;1GS8?$gp9rM&fwC8 zi98erv;w9&t%+89(5?D0f4I7L@8Q<$TxY^~?QAwRJ^kF3>9zI2(o%0_tyI%0q1GOOU%0k|j^sFoMbG z$?Hq>Xm~F7RmPf0j!0qyYPyk+D@UN@M|uH{peE(2Sw=Ikij9D(U|F&>VyzNJP_25) z8l@8Zor^rV8EJONfb2CBhW7U*2|=vq^9Z7DQmV#iqbCup1(rubV=KliRWU`Eys`CP z3b8_CSATsp`TF64jyh92&-1HSuA~&Zdwa`E%gIw!mOI1~rLv4t zOew|)eu298sd0R)-EL(@@N3&QhIP3`9*V+Ks>D&gv;#3YN6YNo12^4kk@BWT_B<3%cqZP_}wgH zKfG|mWnr`o0gmq8pd54!2IgQUP~d(n9hvpIu^bS{0??j-he4J0+?lrv<{1D8=T6~& z_{G(fj+qq`kqiJ%wKO|{U^L*&Y&I6k3^CQC4~77AdstarkEx_eLluaikkJaTV3?km zDsrS$S;$|wF>~e8%vKjLC*gT`kvP@Q)XY?x5V83@%im0lx*Y98dbqNE`|k4M($3~iOoOgvRun~al~p<%55SX%i^t!F zl~Rf^6$SDXLlKeMke$T_kE~|0@B)g*xws*8J)kHB)x^VHxu; zaVFN&b`nV2Z*vPLZFIx5EacIu5@mEjM=Ox^*TUn!KI&r%BPh|qDy9qb^JmVSf$R6b z`~BM5TDRAG`|Y=Tz245wj=PhOog0~7vb*aMsjaFCpjoyrtAjAArnRb%^H;YN?X~)KmYFLKfiJ7vj^;IDpB{=H{STu{j+CI|MaB`dBC|T z{Pd;y#r5i~w?FLn66JuC08PjDq1XQF$e8OUV3ZmQH#Z=q1js;m207tCKu!69;kOfJ zuq?S%m8T*k!8M?%l@;wy+lt_T5z}KB8$*WUM_B;`3J`z;LoV1ULLgX=$PMAc@Q$e48Z8~02;CiV1`!S%8^WP=G*=JoliD0zQ4D1`P!wkv(v3k8{-8A z8LAw!V|eOv-Ye%<*XBRJhYvnm`sB{N)!km6wH^iH4N?+7){3C;_~HZvNV<;-WZn8wI~IjYK?2(EYl*j)(n!JiIdpWf}4;Z?!C#xz#-sbhPjf5^7kmEQCDI zvcO=Yeg*P0^jsu&2a>4KnxL_sv4#O0dH*cWb0aMaAd8|%F{Lm5=T%j)`2;S2Cx`nx zgLwFKk!peDsAjH{!|?4vO#u}Lwgxj8A$dL5!!JlwI&~p6$NHr!k_TpKQEHe-o2;=m?$-B{efOW49YYrN6>;1m zxUb|H2{XuzCI>+?JN{PI5OH}d7nDG`t`ea?rv;uR8_UH zx!LRWk|(oo6e1;n-+I}5cH9ai!$@sseX24@pinn5mO&F^V?}00=Ruh<|N1~Fz|E}P zE~=Pz_Xa+)f*S!j%u~D79_v_^86bf>J%96qmH+Vjho3L@632^n5j_q#@z$M^|MLFS zeCx*5Q&Sn9zKH+!=Vz;(l|Oy3CM8-7!Ds}C+gRG@2hRdB;s8Vi zrl{Pq;41rrt`!ps*@w6GzK|9*8Zc{zgAYe35`i#D4ho*!i;NaXz$|y?;yQNw7&t&= zfH>B|dePEX9ucIIb7D?{6m5wjW2v~m74u3bCx z+jsEmw{CCmmYr6IXzg_e?oOB>3eF=zcaKN#<_Q{s?KbRc3x;Ni#D7Jc_5=cX{QR}} z*Aq7XZI~xX@LQV^prHD_A)t_i0Cs!*@etarc1i2?x<*Q>*VKL;Q}SeHSr)SV3jiSK z9;>Kf&wELNbg(iHhLcc(#*saPK)))@!k=MZfFXoDYrA`qXXy*L&O&yxV%|cNWWmQ2q@F@~H8e?07-~6Wk7wcpHW)Q6L)uE%p0gm6Uza*wf zM|i|H1=-k{A{_Ey=(Mw<$T%>#k9NEP0$pZGqmW+h^Gk|^MzX4l!&M%k9=+JOCIyL5 z&3yMIN0K26R_jwCjILMJH3JSvpbndb8;6pQ;1cAuW7=IH$a;Z&vX_M*KU50KN09u; zj#s^t_Z#Y_VW1EJhYVl{6hOGU)xYvfT5GdssH6s`708CtN7U?w6j=pMBHFTjm<$_-jzJL z9D>7>e}M-N9xSh{SQgry&eqmef6yNu+HX#4_1!Yt<~pn=L>~#?j1^LHG*0L|%ZfZR z0~Kp6q2#O(?b=vLmLK9;}yHya0$db6##o}hzu3ANg3?fmYUB&jt z54Qg0%}*xA#xKux#xvZwjI|f8Y_9tkLB1?t2AbfYLXh!*+0q3$ z_QSdf$SbBY%1sCqNf6SIyviS4*m+W9fXkDVDk9M=kI@ZQWCekNFz ziw7fgq5|9%L`$qHMP_5-*j_iRZkJJ3>#X&F0FeSABLKQn#L%XNLD?5zU=n`yukU?y zb8VwrIS^DDoXP~lD7{WbQBhI^5T;hQv$A5Ab+SxliV4=(n?VSUU`PNd5(XG#!{d6` z{B43srqKnY!|HjR4MGyhAR+umL6_U}Br_x5{t7v@?Er#cr-cP^Z2pP8RKGdnvo)ye{9+9=MVvNNl@ z@sIC*R=Kx|Jf%k_>f{grqlm+Ih{wm|b^@V}Ru$Y80?Yt7$fi^7zY21D0s#d7N+$ll zQQ4OG)+k*Fo`ozM4kKj4^qML@>Z9+qOeKuKfBOXYJKNT;?h{VQ1-L6EHyWD+ zjL>(`T9=PkJhHyY1^BlfB>`l!u2Q6oFpi$6MoL;85;PKYD7e>WO6{v{Hi8kFXy)*$ zIczmGEq(!`(ZG?wco4EH=twHmk-jwt43~A0DvXVpIv$d%YqsKtb!PWO+Hs*jjdDMU zAW66z(W(X=ACp#pB#DHP^fV;p3aoF4+8-&|)OtxZC*yfAZP0|f zsu*L6f{6V+b-*3!J2OF$5_BAmqyT~%?N=VLzcq8Cwf_qS{r=+O{oS3N+qZ8oFE8)x z?heXD8dP5uNA&qoo-<4%M=>GGt<@^V%?dTE#@S?Q@RW#btdkXaQ9F<&3{N$Hik^bJ z=;W=j&}tV2Q4SER%+#(b-`m>tL51;Lz+O4|$(^{o)=wis>uO#=W>Sse?|c_Kg|0RjRLBmj}Bj8UeP zN^&a8I%f3PFUvtyRY2fKB8Fg%05%?Z?Sn^N$5iK3paGszcW(=H}+vg_i)b!Mu zx%ul?W`1ye?)2H|BDagDapQ>#pWj-#x50LvwOWvr(m)c4qj>yy(g`Rjr4+J^Mt<85 zKPNcB33Q~PPFa?ZymKsy*1&tC^iCjzu&=2WMGivlRawOtJ*DKXLnU-PUSV&7jjtsL z46kWvM>J2^2`F6EP1NG@H@l_ngYsF zUCju0J9b0*hAGDvGLX@5bJKywamN*WB2kwax45 zE5U=A4j)#BlaNNFF{RY&b=$Gu3$ZK*Y}zZpV8I=A*IWXq)VaSp_|2Q2o;uZe@#zcW z0hi9;<>ya5*w}dc^DPHC;+Z^h8tN$71~lZ6IZ=cx7zm)@UT;>o6KJ(CGuv*r3+|gz zB|@}EZ%`|@bLUPkoSq382EzJ!T3OjBt3=DTHn#V+_k3<9Gn_fQFgbJn;m+2;J46Cp zJm89gh%_}}GgCz#n1y0_wZF3$A?Lv@39=Q60Dn(d;5$Dcd$vY3qK527h z>I=z{00y)m1~6c7dVaPjlHg^jy*-tKvM4%v9#V|%$<@f_XwcNJQEpA)c+e~(O|-L_ zsaC5P>+h9on>%}hiWuvsSPtUWpnrR1XQCKJ!va$C52Sw;!KV^3~RP(SH%^i)lkpB$MJoZ#CNL^2rODfQRK0zqE~o?@JK|> z@;rnfd5W>Bs+5u>9CaOkbC?+bs+shvD#o&kkN9mCGPCfA>%yvvWm%rwt-ovRw=MOQ zkY&STBspmKE^%@_))3irViOwL8a zPCOcfqz3(_bRcUU8$B%@)H(=|YT9$o*hNJwnsLu9|;-ZC8R^F~yYpK$Jy&aE`9Q(T0*B zam2FHP{9t!4i8gXXcSft7~>$Wtge3g@yFZSTaQBVBOjqAP*LLq~d-O0kQUp#kyVJ-)@2l({P?TzlHGvy&H zF0I^ouy*0n%viwH3wY*<`TI*ty>3Y`%L4%-L1_*wUZ$Lik2kNSx6l0 z%#Wx4<3Ihw6HiR{N^I=n*T4Jpy^n5XxhF+I`fjL*1QPC^%(^k$S=##ZYkq#A2spoh z8<%G`msbY^D_X6XDuIAWNik;DpPjUS`0>*(J-ZNy%?H}(QTpt@1r<(<|Ci3P9% z3qyCCyXe}thzO4>P9Tl$EjGGg;5)<#PH=+yfdN1kLaWu1r<9QW86gy{mYEF(gOuV> zsue);uU8@RYunfe9FZp~Ptg+-hW_VAlk(wFK~Rb@xjW5Be!CR+1hs6XebFq-^Zb$X z9N#@oppo0pj3&YJT53Otz7D>DfvOc-vp`ZVmzRUVUU#q8>*@=jnWFvp zGh0Hcs#sP15CRn3eZPlr-RFu&tSo>VsCqEZ?}&;O9F_pEO&Qz%d+fr@;|>87UpMWw(j3wbI*Yw z8O?!edfcA8JbUqM5s2h~_7jh#`ob9HZv%64;pwL@zIVU6bN8XVkVs}`$zy*a#AmLY zefIk0@jOewgO$PU2dmpXUk_{j?wZr;1M^VupqPnFYJkbCS? zgCZ|)TwVCZs~2ARA#z}I5AVN+phO5FB?1(1yjW^=YcachXYJ3g^Up67_63)I^@XeV zwm$lwc|wp!pbBD@ zg2QMuKq-1GP4XBAgF(9;I-QPL*6)|cmdGD5x}V_h4|h)~nOPRB@^sw6B2I9E6MTEI zRVk$^R&`QxTN;MP7=XcG(C_!69mXd{FA2lgMyv^flA{T-;o>~b4@}w@aE%O@kn|VsCv3*|7Z{W|;$-T5u(~>u`(FS^(8?HeVgKz0}^(m|)IecDM$aY&QM1yeP7e zjl|`YTuIWGlzV>NlSKy^J+*8*I)eM2?V5!F$nye1s*k(7<1s__6;HoH##=X1UTTqqQ>s8mao5c^TJI2 z^0Sx5a?E$|%2Q)&t7li%dk?xjw?ag(-+%C6admZiVtPE!aQW)=^(RklZLRI@CbQt~ zKzDosFWk8Jv!DE6;S_+BSs)>pBRJ&bek8iP6gXbsg_p)Z+dSVNY;0_#BoIN~%dLFs z((FI|`1%i@m>F-O>-hZ6&gYBU+f|+;SHfmDd*_q28#i`lrY0u>p1*AOUVCa|kQVQ6 z8FDeCDrLdP+qkmO`FF3Jd-m%16p(<;HQc{{uUAEQ&m_;O2uD zpM7Hb^3+(s)7P-|>NS+N?>}7aRgq|sErv1_gDYoR|L&EuFI~^(G7NyVmBGr=GTbs` z-W^n%Yn^nyU4WEC&-s(QdchHK-~j7t~1CZ|0;6j|Wa5N$wUxW?58q*k5xp3vq%I{4F@X;Dn^y zt19Lp&+`_fn1(4CsVoO6CA09&s_t~$!u+Nql`t%^rF)Q&7eyEjq^jH<(6`^K`~-v^ z0pDi`YIr+|#~s{fTaR9&!K<#pByInaAPtl#1Otr@x(x7`hKxmUs3ulA5N~NT4nuMe zvj!v0hVLNs7xt$sxjR7-a@DIJql1tJP$2h5PD&VRs>AYqBV$$Vb7n|~9|Df#T|uK~ zc!4g_NBP5}YGj~hm@!63d7ib~#Q|$4bA8TfdBh6hc zSRo~8wEIa)ltyZ`0Dkjm7J*IYJ|$2jZFJp{3?5t|fYu~VHXK27gc1lxYj42M?r#77 zy~Wd~PCa?P(>7e3#mi5hxqE+WclW{OKuIW**0=i08-s-e1)MvBfBMO#=^VFjuE*#= z7#p8_`k4zqzH#cx8F&I%!GcBuwt6N*M2aa201$!M3HKd3mR-Fg7-G z@%;2ZJa_Tcr;4!_1|>ebj}Jaw-`H_y3kWnsPJDWQ_Yd#hnxDCTWo~S49KU$=R5t#T zxBmF)@=|wDvWh-F);T>Je){6MA3b;O^aR|~?jZc}osaK6csNKMG8gF0qVxJx{&Ktq zkW_Tz*dN}zcjL)3KYQh3tH7xV{2yMqI6dC^=#!P@jqY~88$#&Np1wTyi$Al(A5M-3 z2)J`^@Y^@uez?5e8o%HORpo3HsuQ4ONCpoL%@&Y@iNRq9b60{vY|I@xf)fB*Ii9?F z?zI=n&FvDEmbZILyMtbT@ALaR?|gLk+}sZ)rt&jW_=lIL<~z^7_u=h(50`iL%F5Ll zZ(X>&@ai*jKe;hK+lGV(%X;U%&+k821ILW{?pFEPC%0d|K6&Z<r?!r{4eQ z*6QkZRi)wWY+|zW?3Ias_wuDvbL|q?+`*j(yBph;Faf6I)u60GuQxU}(P|e}ipc>D zH{`MNv7JB%&s31y$Pf%esF|4iZwmZ71-_o(gxog!g0Suj2-@)Jb6CnWWRD?)=A`zK z#}4k->wcY-=40g`;#eHG8DmQB3Ly-ufAMIL$^AQOv3i1f;^2D^5`4_BL$Ga|CY zIvCHF)mnfOXhy&uL4uKKcs~(5c@_eIc|@3KpG!0Uip zHlO{EZ*0fE*xKqy*C{s7c-0Mr4w_X%QvH|18~S)BNHN9=M3HBIMoKA0-?z3uN+cQ- zd65Sa0Qhw?%{6-S)Br0VWl$Z2zp$KC4A0P(f&I z?DpPxZ+U9!^QrMC7iL@Ixj+9zv9on)bF2HuTNPA@8@?2IV7vtZn=$9HYCU`PcHsZHH z1?b30#)NDSWA}D@|Ce9Ax19$6{wLSRGrV{a*U!%1erA4SCvNZV8aX@BIe)r1GY11I z;PW;7r#BYg{baB7_H?IHsT#!o?#>S2i6|XG$blSHR&-*SnkPaTQ)CgDp&o%LQFZ*@ zqXizWY%H!;7fQ`^xGx|?ndg{_bdwK?^ zpT~_0S2lMp?ez!D!t_KjKaJTLbTX8{{Vn|RkN1D`?)v&(4q8NKbCA9B@!H(f0I=jZXi{4D?J(@(9gRgc0sQ_-^NnX&UToNJ@1u(pdoyt(w&yDRHmc8LHO=&oLW zAX>ZK?sVF5chB98211XM@OJ_=is1xKqGvm@i+;hULH7{5lS2^Pe6cEhx=xT0Ca3qQc6GjGCs;VegW*G_9n`|{lB5A~- zt5pYowO~0JW28nJEcV;r9O<%QK$c}gbFic|{4yU+Y*jHDX*T+{7$c2>$OBf-<_)Tz z@&;s-T**@i!3ZC=s3J+LVpZoUi9kw8?q-%{83}htkG=~C7KZGa`#qJ|=$IrD2FayE zwaGfvBv%KHOO3*G<#I3>1Rm+*MY1l|rGioBF3E@6H(W+_GB5EZ@YvHvj{pJGKBzi2 z*lf7az9*BSpu=6VCE{_d@5je1{Y@_v;sh&5^mn_|H~WisJA=z z(;ts7pcKl)vT_&9Ab||Bb{6Qut()uX8~3kWzj*1=?1eL^&LCDG5HbW}*YWA+_|@y5 zz5dR<`|ByuXfOnV5h23vu5SMF_0K}|e|YiA+^Nn)2NQERKLb|*AOsYEfZhO0o9T}q zF8=fDpWj=Jdod({@Z4+=J+f5}cR`L`zif2uJ2)~<8?Y6Vzjtpt{`TWer}NCE`7<-E z@eDt>fPpginJfSa;{gOVBHp{b_v_!?`r}(`YkPU#>A2u>0hYBCIstHl3{`X@RaMU@ zfi8iNWLXgvLoq49fjoyNBdk7sSh+1OZT;z^#mUJl<84fhVR9V4fTe9r{wb$NN4^`g z&lY!yTa*I*aU&w*Ha*)V~0`P?84ZQWi#_!+0cYh`Jy(o!X$Uy;~ zl6pN7d0vcl+I{pr`MzjC=rJ(2o}i|+6ah+*8EBl)kpAX=w*1a%ZSxz+rW?6NgVv6q zdKrNRNHBm5pm98Jw^IR^M@M00uN?Fv0VOCUIL(H-q6s8y*g>b+Mw?+c+GzDg<|@ zg&3S(c%Qc{#}R-o$sjeYQKuP{>@E3 z3?Ay`!YoiiOv5AlWd}09ew_9ZsA+H(f%?T`X>1)}{g^V_W~Ut%UrfKYKzO(7qfH@1 zmj+ermxD)f{sw_x6$Qb=K4(g-s#um)O5S`vODXPkyTJxk*}u26^X})$a<1)m7dHka zLxKU^w}lvl1`>;I!p#SRfBD^gaq8m2Sc$yvXirbhotod=+^h!uK@9g6xBulg@2{<_ zz3}XlPhLJdKMNzO3M95R{ocLBKYnoc5AWStn3*_#Ic^6mZ4Fj;Dh~xiqy%M~Th%8w z*E2umv3T#}jraca;hFcBUwiGvt5;9WOd-!uRv7GIX&di2bE?Ncdt)lXdqpt&j+z0{gKtzhL?Q-nnJLUiB|MKP!pE&#c z(^sCndU~vdRttFxazx?YCRSE=Kfbl}hfkK?`*@|>8-umNpFxAdz>CD<<2zfe@zXBc zet@2e%DVlsKc2S;rn>k(7^7@yt^e5~sy>!CvA&%g89=}Vt5WO1^4`Dv_CvC`@xybi zHbnFiR`#%)$|_V({PboSnv48w$e?is*N-y6KU!5=m;#j~iUN02_yCeXZNFq`d+S&4-dtIJ<(Vg*I=?VC+b#-7Lv0zoK6ZEU z;NkAOAKm@a$M+YvyzAqM!9Z$Ms0<;-*z0v^YKFpd{m_cPDc1W0Cm@bTp69Jc8OUW> z4F=_Y(@05w^|6YFNwSf{3OY6VXp+{Xt2MuchQI;DW54Fh9q~M=EX(}Q$bylNOgt{> zyU^E8Ff@RBjQD=x`_~6yzXc>mq9G~5n?c9>Xr%s1A^OwD zb~n4`Ffnrr3wUlySoc?L+HNw?#+9*7R#!PM7NyuFwQ)=JDZ2KF7t@mdpY>>QL z`DTCY!DgsZw2=1GjJvVb9i;#3 ze|(Fr&@KBbyY23v4Iog-v!%8E|M!<~{k8?e_|8tXwdr?P*Y2#me(}Q0^z>LOr#$rg zetUWE=7X*E?O1uIvUX*;K){U|6dZJ*OdS;M2Yb6a@2z}#cjerC>+G4iiHRb)Tb3nR zztZB;-tGJA8{55Z34|$GZpeOVI=!8&&DtVc>wL9qb@X77vv`E!|yJ70W>xEEGi!wR^pOjE_&F)DyK9 zgaBp6POF$4>yR{~y9)|rFv~5^Lsn%mK5BPa#)uL4UlA7({uZ3z1Z?(>tx=3eF)Qi!rk-1T&MI(3e_eifHx;ara>>AxXyzY&^(-XgZnz z#OOdA66st5kkYr=#Nq^!hR2A2nSEbduyF`H_8`O>bzc`r0Ldm#C+jze;QsWyC=fJq z#MVZ_Z>3ap!bGR2x^Zu-YeEo%%tV1?z5YQ5UH#Q4b)x|z3E7jARP9f0Lar}ha`*j^ z?IO=dIZ#rSVtuB`rHmxt?#KVjr>0+gw56G4S(ej67ILGZim4hHkrg?FL0L+oQ7+m+ z3>BeKUmsBoNxZ3I=6Rmn1$aaQJh+b?9H-iBY$Vx~YgHen{U1e0bw%|szM1=rAATtM z0$(rK7ym~3Mr42WmelKY2W4N7>yTlHWO59uw7t_!9dz17kq5{RWNS98+Xo2~lu(vs zw^x;Ak`oAkfJDL!MiYW7k12n+9+%gXASZ;Lt7%N)(XcfTBqQ59&O2Mh;#S)0RfDRE z@`x5h2g_thZ6cI|YWsoTURolfp#liVJ#%kIv?5yT-L%>3CwihsnKKI#APeI=gQ~Y2 z0}|1ba@VXXEib0`A8r^o_RsN3p;Cw#cNPguZ~zc1!G`-2Wa#vhB0^bpw%7YNH_GtY zqA>~rAOuI2lt<4TGh%?7K*9qZfH4;*@2(6MSNb3#LZ#dlXaL|6N7sf4GMNAYl_Z-^ zG45`p#m%h@9w;!19zln>^2*05wB7QQn1SQt)4{A;?jQ~4jED3-JbB2nx=eV_+Zf>j^mjhyC(MM~bEU5quLSpEP6~=pm zd}Fz~wX&Igycr-OK$#$iCov%;6e;9l%awr?q972ZTg4VRTg9rsrp2|&K3g)T`XmpI zEGm!6B3h1?khubxg2YCe8kq?`C`%CIow4Ho&)%Q+sf}dazvy@E2tnGe?*0vT9^U_3 zy7Tbo9nX25ss6EV8BQSkQA|}wKtnFM&`)|S3q#*Uxy%Y zK!o0IH^01n6S6Rw!jh1{D56DEA5Dl+nPvX6*v2eK0g{hYStkJ7NXU|mxpU`RV0&q+ zc&J+H{gi~f-;HsO)>hT6br&_49@@0t;}2TzoX8xw>at>XS^ZS@)}(NFYQ0>U`6LIG zE=gRdQej=ps>=o>MwaHWnKbn-XvzDL9F?4c#WEKBJ?g&DJ01~f7FJ*CV z>r&^&OcF9ODk0mJ`=&{up!T(&X8>!8Y121zVLKuaAg?>gU=}VtN(&LuT2y(iakYid z7zs$SeW-K*s!A}%@JA$7l}=`4gd$=-{V!%sl?6Eu+e@mvB!gX<$*|+x>!Qjvh7GXY zZY1Rxm36=;E^^gnL)kg%W8dtM!V-X_79r(GKmt8W^t+isY^^iN!CGu{+s<+pAjAqG zNl4GGxC;g^cZ`KOQ!e8t0NX#0WeeEOtWg7yx=#3n6y@5@36BVA(?SxGToxc&G)@@) zC~TR>hY!bw?IUFZ1QpR*qcxz7p(>dv9mQlY7SUw+mN&r)4B%AXYihg=^ue2b?OCmq zv{Hi(5293t6l3*;G@q8FU^#vxT%y2cGqas}df2UIxv)TmkdRzZ2ty$g1;vmNvYalE zEYwU4BDE14Jmi6j6LLWKv_K#lB@!$QhDf#IT-sfT&=u(dOITpSXs8z+)UF63BuG;w z8%&tybLOz3_5Hk?$Ig&U<&D7~KbmA2fhl2OY!m5TOdpsG<+7?|yJV28HUrw0V)hh8 zca~ySD})@_l%%2+fhLi1MIj_N)2%eafP}(?c_#606EKNhen_-O$`r*36ILrIFcX@D zgux=kD*P?scv3!Vii2@`C|5ppvl!yCU_);NIfP|VtEvs4! zQ-59KRuU%!wp|4X`pGt!#%vets>;@;&1Tc$XyKd_zO5?n zc6(f7%$pvYDSC@V&%qR6lUXMlV1JM}nn#V4-S`8w)_-~TFl}P@IOh4WN@%jQ#OVVy z-@GIW;k9<6MRBAqUIWGk3nP#)w9~SR&5>f506_>fN}fR`hFYqdGQ~=kw1LbCTjnDo z&A`&4xlW(}WOWzFfXWOsC(TSIAQ}X|aM_Tev^xBR3KbU~7hxmemCT$)0=PpMM4^Ff zH`Ad5&4z+9!wgGCKt$}uJda1hxUWy2Afh6$RA8_S*jBj}F{>F%#YIt-B$>?9N8YOk z-E=SzI1FG*6xg_oXrkBRLjbghw}n8YkP%>5Z|%|-)}Q5UhzydI_O=+1(4b#VhMZZ4 zJmgS2NJ+hW`_BG-p7URu#Bs+cP;335dDnY4b=i^xuauKR5nUTk)5wu~@L%29Xf2#6 z=tqtB?KJhN_ZA_^u6Z1&54(A{A6XTWZSLIpas)D3g?omm@Ctm6Z+g?rqILo znC=CXYgFdQtm8kn0Y#Jn43`?n-~y{kV$N#4wQ7PXE8mIWJT{a6c))o7_%5x=V znx@v;gmSkB1m?SNKX}ad;*QX{!K{eDg=AK$MAA7eSr{X&djmZ;qimrNl8R`S=kgdc zik@n%p+8x}Y!|3ct;)8!r1d5QNV=u!B@qfqfa`=U^gK#8hO-1qN`?$Jb{vo@lLD-z z;ctX(aJgI_W1PP$i;vjYE_@t!r=t~AwuBTmW+l@sfvpRf_Xy!QAU*G3=I1_a$^3z- zs(GG0l1i;K%WMk*khqPEB>*85BD8#ucAX=(KcQs{JozBDn@wvSyx;Ha0)hCW;C6rb zn|eHK`*yS4P8-nji&xedc@p6HupbG5x&eWabShkY*lc>`uC-AIkkHu#^<^vL=NOQY z47$;M*nlrAKox=jtqK$hDs1q_{|a2IjmoG93)x^U@C*Nng@pr&Wi!Qzpo9t;LxF`% zg9ZU58!bgj7Cck8Ffy{J=MotPw3361vmeGBu`Ztxfh|BO%DBqWy3Kx`_hT&k0pcU! z_5Q3rqh(^Tx|d2Sq4jrFD21U!62g!I8P0lah4{FQW@Kd5%FPr~Lxq771A-&s<2oQ< z)txxR6NPKJl#2ubqL48Lu|`Jo$b;0Gq=$#?G)?muoYA$;=?3#Xxub~9_L$X(MESOB z(oiB=i>XihmWEnhFXK-*Dg5e~mT$k{8&}v_dw&$` ztKZ-E|M>0whsRNNDa;dsy6iLlvj~0bb>HB>ZO`XF-M#jF-%~%(T(zI~m1%Gp1PB}7 zI-V75p0t~utm4e9oaebJBf=07#ri6Lep_QHz{c2P%#kCcXs!37X&Mo|33xJYIU)Rs z&s6q252Vdz6AEmdSJSmqSVsTG?o4wNfiXyYq-WL~`2_ygRxBDLr1f}bJySOV zsn&gfj~)31J%}DHON*DTU+~ zD%)<+Y)e|ACM`n7Ue@v$!{u^6DNLqF$BrrDi zx^=CNAc=n&27MdLR;ahOY5kYCZ?_L`e*f*aUu*34vp8Z@F4@ABx!Yxn`QhO$AMuGg zV!IRFE+Zoeg&4Aj4GMJ2WiHC~-fl-x>)G6D- zU_Gg<{n=+c*|iwL(OG(j_3m2)*5 zM#6sCAu(E39lE`##hbTpqqk#QtdSrTAkG8`63m_NhOiV*lUm``IL9-$f-tgG(fWbb z+C1t!7`vWzD7Sr?{QDds1U&CfFNw=Fg@V2Yy?;iTwH}Rjp>OX(`u$wb!5a%tgGLe1`q3^35CfL{X8P`XfF4*-M*QoI>nObtb`?z%mx99^&y|O_j5$sY&P5NX5P<0VN)yv zhrNvCMm! z+{M(Qg(RTNYGb!3UsRMXQN#ThLhEUoAhdEM*_Y#~>_WzPX$Tw5>85BsYGhG>uwB%F z49@!)(Ikl{9rhNA1LbNh?B_+M#z&wGAfz}TSCt(trkMO>)=CIt+o3Wq;*#aJLUv~5NLK-yKFua;w%5gnGIAABVJJyjEa_#|B5=@;?!{)Az%mjt zrkt)YF58)4VeFjuc7p#!X05+j%od)a-g`-7>|LqVB`I2qv#Fpm?;i&*^{PQ?OVBYQ zs$4F?JkN*p&er0|ePSCz!uGBF@=FXx1UfY#iYKweD)RzeSpI@r#_xiC@k|9^ZllZp z*V-qh>wyFdVL|BTuwj<9uWcWHX7Xb6Vr^j=FK=os)lVyrQj4rgyDlZ&bs-v#&9f9= z3$SqT9G}0BNzgtM=F-cl)y3OLNJiMtMpxN-(ZT$xS%jW@9zcNcjq*m#na8x`7Tbqy zRXsUVgs(?#?|e73&iACoEr@-qd0dG#^`281^)-o4rW?d|r>Cbr#;618d;kH6&H5M zPj;;8x`X%mWmNoo;;$>56tvJZHK^m!G{(o0IoSq>1PE89q^hN2<~ZTrY^KVm1M#?S zR<-E996c}1+Gof-#|B}$(yloVjBLGcrtQ?H$HxyNXP#qNC-G5`Rb?M6_7F#tyROT- z_+~D~J^ZhiuPM8-=+z=@r>#pp4C!`J?qB$_B)<1?a;3`)b}e72D4ip>Lf95uRoG3daP-b$% zRhJRsY9W`fDN9;S?~G-tjHHkZc9k8{YC!iBQn2KCp984sn@zMpA}HCEp^0c=CsM%} zB_U+a_=u1s&}W7w0VovdoaccZ5!ePv5h0LK1ZGrKj;#O4t)zcx zUc{#j*2m?ejpiSA;Oe{*GsgVuum9Ov8#yvFT9c4)9Z8oi+x4QbEoAhBv3-_u3DJ9G z7U1%3x0CdhA5U@8Mr$$8c>tR)$9yIBjd56C`?kJnLJRz7Kc9t`Jr)aPhmh1x`Ca={ z&;7ze+`zg_Mf!+76I1z0^l`b0`L39G%m@AOcW)1g0uPz___*^Dy;fYb6#Fr_V=R{x zAvQf+Q+6Z?nSumtVPED=0X`3YPm%sh;ZoBPKxJ(*Ctg!5OScm$Gil&~O(3&*bwG6i0T{VPjxHf3Fxc77-#$U-bg3dP9W zj|xSfdNtK>*_VxTZ1WslW(Emkbi@=9p(Xc3CX7vmnnEBfKY?*c5rlaSmehNzY)_Va zJD`vhasI(a=4WM%z1&6^87tzPt1a0!_z(VsA><#r&KCTnrymiv5v}!*w%cuH&hx%p z9MQADiLtAYLZK!5A4TZx+qb>&>@!PEwr6%CdPi`lr^m$sqoTTt>~tJa)Us?e@*#bQqb3hrW_KVeb5M5R&}4 z(?!@W7p`FoEfw7rishfgMO2R}3=MdxCOxyP#J{-kOM!v9VZSzG4wIe;NKDN60^FPm>teW~!4-8mxJm8L20tqT` zdA}d6<_xno-hED6lQg)J%x+dyW%fB$|Sr3h>&sC5=V9B9Fb z)>?1DGb99x2ubS>1)8Qw+DO-}YrE_VEy$5&gN`h&+{G9^-~aee<&hRTgkg``@?CO5 zEm_mPOw^PRW#`ni?GacmV@ce(^J-kxw+~N_=`qih39@}PhfWGW(xFG%X1m?(56n+J zXCYjuMH%U9TM-$|38_GKK;fF_%yCrT*lZ(otm_@4^^S1eDo7$T+o>T$LC(YG&KF@D zJ0g&(3T!VIyzkE)NeW43uW9vS5jsY$a zB2U>Jc13HFY)ed2!**HHMPRlp$##T5z-c8#i)c|kDHSJu)1LOWvM7868zUnWNQ4Eh zl89(tw1hraXKnk`XlBNK63i2l#1l4U1+8^dw&yXz)q9t)CGiq3-_T_raGP@Z1o&sO zY%dGAjdc1QTWj4_Ist@^Pbs3Z4oFpYM4a5EU9jKpWw3oJ_Nwwc?_G84n77TVi6^Hl zJn1a*b=+xjM4_r0BbO+XB<{xT8QF|H_P&zZG)_B;-pMMvYop#_urC5J>2WK+ALDgtU4jLI8>gA<=qo5=YdU3_5|d zsu5alYgNhXvR&20-w$@}W(o={>Poiea?fRt9&KvTn=-2em^*h~6D9?5I%r!rei|CC zW?G2_N3`ws*coVyaXdrEs5C@`w9KCM%@mE+8H8m7uH)RpRBLgtu5;-~C~*zrDraUK zXib?;-`lW$ecNqt9H@Zg&OfW4t=Im|Yx(@PUkJB=FLnOC(%%!UOCg8A;0Qe}wtR#t zSEX&0T8m}Fwh^G`$~HbHdSs3)^Te_Sf&$rv1N2O_Q4+YW*TgR(M_aN`)=vu z*03!F1{;u~HT2XXxJpQ$yM7tl<`(`LW42u6+9^U+-XCS3K2+st%y1k~ym@oX$=vOB zkB{%?xkQC{bI?kD`>u}UJllATaR`-2n&%wz5c>K-t+#%7ciUs+0b3yM{8_T<%`|Pd z598$X=FamjKFfWcG&##3-s$1tNlog*2ba0hhU$B9r(C=JezTd5%B!tc8j4FKfsCta zKYYv>DT=UV1A3mHW$X(ridz+W3P~kCt)6TNV<80+lC+HTk_btw1R0P&u5F;T7LqV1 zBzkOGo2I^eEHEISx5*P`(R}ZxRi+j@hA_xna_7$Nl*Qhs!(ZmHR@R^Dk7b=Wo|2o5 z?fq_d>g!`5B5dsYlVxRV3rGGxOM$=s?!KX6(Bb;Mdby%PPJr zcNRS%$?J^M6M&4cut4b2m@@@|uLYWH7Z7Sw?;DSz47jRPDK%T^qGT7?M!}dwvW=LgE-&`HduyC1TXqPU zm%YM5L6pn7bY5Eq3<^oIG+bj08NBMhsfcJ=s-Gp0WS0##lEAM!i6UacE_4n$+OG2t z8qyz2mT_efY9U!@U3MucjBW^}`iZiuynuh6m@Se((ITYsvAi~O*dAkcSuGCqK5b8V zN_LJhVgztpKpw0xXTaV zkRrl9P6BK%T}HTbQ#^eB@aVU1wg<)-`~B>7xueYY;f|43VcY%Gx0`PCvS{8A3fl#v zYGjo$0vUslK=^DJD#Q~m<6Ak<5I28gv!KO0n`Z%xSZjbES@zs$8AJ`100|4!La+}S z`L}H&#B=oQean_9I}|ONkWg{w^W4o^El%X#6|KgBvH*N3Z7a-q-t36TsxhCGkgGru z0Z4kC3WRu$KQ1zt=vh3-Wd~RqIW)6r?PG4&uvw*s>73*Un5xRmklKM9Szy3MU)p5x z&O)?^3hDd$HSP$7UM3@B(9PbN7sjoQTvzUoQ1pmiMCM$!IT0!ol?C*vl?z+ol}X)N zlU{b6Ff1Z;vP%RYEvugcwS2Rhg2EAE#6Rcw!W-ZwUz?J08Wc}`OR65pM>aqF@lcRPf=oqB`>24f>5 zq$so>`M8@OcZ(Oy_Vq5V>xT6@@u?m&{%k&;Y`~JG0879{+Z{g_74)(naM9GNA-hU` z%Bh9`wbrZB7(iFL=^BJ`)NVg)q4=zJ_no_0YnO3|-uvO|c5>vUBJ3(#0<@|}b4WrF z$L!xYNht5;eU6j@^!ka1uFLG(*w0(XgXGnV6#|59$`*`F_PfI_sZY*K7xuHB*IFfG zj;yKCTU76L_EhpLmBlMg_+^Jp zX*CksfYhh9)4n_#L8~gr7P)l%T6)9S*hYjTEgN9Lcs+zlM1;$LLN36zFBEsJa)E4t zo|(01F{dw6vUL$=uzfjdyRak?czH)l^wzK?h1$L$mD1rcxj1duv z2)JyszgUu(kI3d4y1b`|Q~HC;RR&41M7fJ6>~bBj3nQO1w8{uTHWmxEEt!=hiJ)vK zxG;eAQ)MGx^HRz}HboEpcl*5}9=0bnso&q9RQSSR%y;FE1+F^QJlnOI z`ZTrPq==Bq#($JT><(t_em~N-o?j)bFIV;Umn(w%d77u>SGV;yCrlL)I#}r+9`_G@ z+k>pK&s)Aon!=LG&dLvuANKow6}?YsWA5C!ia@JMq~5#j{i(5D*#_w%<`7F$3$dAw zdC0r@k9h6#x+aNNVXn+xSz@-nA(t<_Uc*qXk=d&v^dc?RNUN>OD9cfKQdSOyoPOTf z_Rgy*yDF!Rrf%grsOh z1fuK+8K)hDgh9|EwAr|vRZE*wh3ty}HTVlJ!V8jUt*K$_tZIzFE1TVmP^E}aED6s# z5+a6Jj9Wv2F_JH@w_Z`U@lTMAtn2OWms?|5$aYm_rO>)I{OMUmh%g8lzLge{q-ol0 zw-0t5jOxk+eSKj3PPFKKs>(6O>-(kOHKp=sIPIQvHQ$;;J$Ypz@twJ2B*OMc*W>=c z(X#EmwOGQSRo?IOK$ZQk%&yn9hcbG7VMtm?>DtZXxA*VgZe#QI!8Lq2_bwyJSP_P* zdfa^&wO3S&sG6lP0(0li4I&~(25dJQ!+t*ctEEtgOYGuEovLdmyN+gUrX;>MXKUF4 zjGhaUQ@R3A zyCBJcy*hzhAXzQcn_8eUWdzPFk%TCS=b}WeDk&-y*hT_uqZhtj31M7W0M^g}6o&d%G5^zA=XrV7n?a)gqdrRB66~J5|*{A-+HNkS*vli4b10omkL$bp)2U zjPaHoQftwBADYkqW1CXRf;#6-*fvOE8+>U&*e?>9Bq2g1E?WrTVhc}L65vlxP>qGL zghK&XGR7$6^0G}ySjN7Pel!NHd*N7uP(z4_vso*JO&%$`&YwhBUhKzZKkeJGXdSVQ zuZsy7uq8XdkgIiVz?5AsBSA<3wl3rlh(KXnF5hTT!(Wa5eT3ZwV&R$)R7Cj1*BDE- zd+%@GzMZD&U_2ZuY}exvcm5Q%!Qg4yocsmvfw;Ux-qE60>kH{x8xe?cD)~MfoDXjv z@>Hx0_IHbU+veZf_h9fv2R=#7s0wNsPU6BVDTMsP%|)JL`(`SQqi9y`#`yL357X5D z_U=vP>`E&nPRQ6OiQR|!_xF#7BzbFk`|g)%uHQdAX1Q!z%jy4YN_6Lr?U1I;rccx3 zJg;VEzPK+!A>3QvOjFs1fw>5?pnr;nvv0qbY{ooKn@#KeR4_aw6j5bs@e1jj&sQ}~ zy|uRA&)oUyuYl?;i}A0^%<}9wAp~hJaZe{lD=Cmgl%)(Tjb1)k?{wz%?)fc>+u*ebM+A~W zt%W4pijYDG^EF7%vcJtgi{4uYe>+Vjfxer~a7GZ)sl(6~IJDJ0R@l0Nu)pS0+;K>8 z;$izRk9}rVmDITNd9)Tb$vn1KmD={3*nLTet5Ilwxa@HOw(pQ zVRPrsCAJ+>G}%-Ym=|c}&s9T;QS{+#qtAt^-iHQcF6m(?YO zm6+Fa=QVbbjBVi%62kIz<5oh8y&3^Tgg^)dLI8{LcTRPZm071*;`S)AuX4Fm5k_7a zWs=Yd1aR(5Ymfb$cI&M%MGNt~Dqf*xWx2|k={h14(lm7`fRM@##j={0-9~o7>cb1X zigQY#SZrOu7$^`1OO}>=Z7pHs#kjpLT`aZMY`17?Ves>OZ%a!PO|7F5Lb~o?ULN^s zt9y;$mcfSp~ zX53P)^cW-d``+57_x(KE245GtWp(r#3EQjZK?cMH5%D4$$M{Om4C8CYo?hA0NMLlu zPW*xKRU=o%KS!@AO9qh2t>pd4|NL$D;c?y`)qAFC+DNLZlQM1n_F;@vxfvj`T>E)s zO;18weTs-Dh54$oIiXhzgNq*ugV61vLZPZcwy*#0VnHZQ9q(-f z*DIC2>e4Jgx8^1K+(wNvvt@?VBEW?KdD(vq5;l~pjKH_F6wRI2<=q+C4k<+W+hhTX z?~KJ0w#_Y20vd$5z1d`CTc&Lpz5kjGe9=)ZQ z!F52@6h^{QEDhmpyNt1|h6tIen#ZuM2*qM38Y2W2GOj9D^~RV)n(J9)hlda*`~&9yGL^wEU_R7fj~k6>xF$QXc4y2Cz=N9Y=S;B7lp28wHo{~ z_-Skm7;_JRknozeVt`{?_mpgjQupdJvp;ykR;oe zUN7@hvd;h0f25f1}EW0`Pg>9}d^Q|Lt$P-R}4IzgHUR%BXQxH4+qM zVRRl%TQsQF)LQ4MR7sab8WNZ3cUhHLPqs>C+7;2tHnt4{OYwXcEu;J{+iHjI_DVbD!g-jbVNabmN<|o(lyT#Lt86%f7GXgrq<1 zlc-8GwW%E>Ey{kjwtzB$K+<-*QAC!r>b7NT0R2hPeIsM()(Iow%QqA&^f40Kr`~%T zANE9u;Djzi8!89t(?(!CvTR2P%;&K!iGM>RA%UL_(Kf;YVUVw`czdGwOsQ0>*L>E;q>H$P`4&j4bIlh95|+@j6D)9GT*5+}m3a+n5(rSJMMFy4CYrDT>!RPw zTA0mcVS%h1^F5l~e!yT{#L`+||#LRH5?t!?PZWW`{E>%f?U zz~vK^dxXoBT~%Wcl7bZioy|mHgRV^5q{T688B@5b00UAyb&DN2&*`kGbwHA3w8U`H zRS%(%5P@C3)`@SzXSd|u`%w(J$9}&% zth5u)SWemERmh8-9ljq)!-|Cx(?c?LV zs$R7Fm%#{R8>gL;ge0}zHiwmalaTxrUVC<`FP=0lepx#rrViVUq&!f^4AiU|nTsTY zWX||jfJ@-3-M@Zt1kd~U3G<_3UXLslAPHL85W>O;8C`}gEzU@1qq)kilUsa_h~8Uk zZ};0_TyrMe)zDhufOVN=Ig7543Y>8<-$D2Xw?liqGeUtM*dWg*k9*Z z6t^+@q}gnYt0aXK5oIKdSYae={PBTqv`P_oUVA#)HYOy2|4L_!u9e6wx8FOo>|0?c zalzv8l#I(&E-!98BMY+_IDs%AEm7Ue?y@TiY)Vx{ky&G(cGnYl!PzjaDJGpNudh4cOA^y?PfEN@&5hqhy7nAVthS!3`E3s zv(3t?vRzf&`5fj(^Q+O(4nDyo_<5Cxe#(Q+Yv(*@x&I3F z0)OPU-+tX}jy1Zc`rC?eYCw6iZ_S;5U!n<7uBHRw#rG zwlP^m3ydW6EX~xMC_5yBF)Vv1+fEx>HqNSuXwekPN(=#((+JB#ieRw@ZGo?3x1OJ~ z@MqGBK*pag&#=KlP)h{IH?o~&Y(7VXB*5TujjGBTWAq53@dBT~cG;L8n0}C6YVD{m ze2^T;I{sM(gzO(D6lW-@dT$R8$MUhPI_1lyk^F6xtL{j$4S%#L+r8g$=c^*O%ah7e zZ;9UK?Zd+~P2)&{U!>=iki_PAQfhOml{8dUjybEw7*#%*qiF-@i;OYmS#t6`4}~NF z{S=Hl7FD)mudF<9-0J*zJeBM!zihdQ@bPV zbC62XPp;knFFn*lM{JiXt0d7d;syecWudi2xU#E0-YIHS4aPomBqoqAsyvL01U^x7 zmcZqJu80{dXC^0jBot%*2W=9K4tgUq+bg^w) zMiMr@&>E1=-UhU!O{`k`2%jh)0U>b%Nk2dflF!f|ceDN5+qZAFo2_)r?>&^QW?2HT z{V)13Uc}h!f^3b4?eyl&o7RqgzTba;oaec0DZbTg^^RS(B|U5(>JK9-zdBxbM*wDw zqb#jC+N*ZGfB!zK01=BWUq}zz&7n(Bo5HRGkGl^aV`f!$*+!S)Ss`@e98I7M2Ikyz zfIz~!;r+F4Cbq@;BaD$_ ze3<%vGfms$f%4FUF~)AcuLJhFf$*Jw6MFALybiAb>C(CZ{~(^+%xtYKnV8(d-1%1c zNKKpVDQcTl6%lvi_Jr-v#%vMd6+|dPy3A-rKwS72j4=ucAyXKg?LdQUyWr^-Wz~K^ za+J_^yR~X}cxWroM;wAbK=(#MrK0Y#k4Qy;4PS55;`ziF zV>9{QO7&9;bw}Fo4=DHNfQCBJBd z@6S1Nz7Gmv8QW#sD&rGXV;%>LqP0*M&_%35PayMzY|9Q4AR|EdcA)h~w`@V!C*$^L z)%^0yFPqJF-p}v(z!;TF>Y0{6uDp*l*v3FrG3%p#rzmZ zdZp7&%kqUXL3a#DviV450umvR{WGV4>zisO>+@0zOERjY=f)e@>t>}g7SdTsO-3h1 z4OiK?z``A8Wm15iNUB7&uxz2_jYAR%E*lGFB__)~kOD<{$V3FDw{DNh8l$EOo6M@L znp5U+KV<5p*ROC%4{s+cqBXUaRdChC>Pow+wH9%nJQ<$M-Q`LJ2uZAxsRgbeY=jVc z76rGzA|`)%{nYu9Z>!T^tO7<}E~*yBrYw7EO)#FqBf>%gOsqGtvg(A;W!m#!H)Km+ z(xyW(o^)yc<3HZLdv_?qi`K>%2Xk_shh1hNB<$i$ae-~Wen?qIw8|Bs-j0*(&5=oH zx7#0fV`NpjEFo4U0)D2Y_Ep?D7TLDee*675?tBAudD#|G&0mK*Mm8X0V_i<@q-DO>d-pnYWP9x}xpcD0`WKpp zOYo|-u2HW)#e{5IfL#&H9Hh7qEG!;ywdx~RDlgP zHs0qzdfd<0Q?#i~t|M95dN-g5wM_5!^WiT{FEukVoz>73U6Ntm?GqjCglH=hUKg!W zPRgCalY5Pdrrw)EIcl*UFxcl+%f_WD5R&A8k|lWFZ$Pr`PyFuwMCW>2a!9X{X!2V2 z2*Lyc3d?0ITUbINN@-K(N$PoOF-?lFFh;Tf3S>(1a-=|<8zcUO>8r2OSY@Y<+BUo0 z?(N%OoP1zX&BJDTnCEd^#+*48ZOPBh;LKl(JGQUJ z|1s!V$lgL2ixn2&%c?H^S|zOFo}%z46-|wdF?A#(#9tNvP=Vj;9hQw=b_KA@h-xjC z6VS$G#nRff(smv2X#r~~+?BOze+x|xwW)PQlj4ZfRFo+Mvhl|P!`AyIM~-<2O)Uf` zO{ugkiPOXF_Bgck7Ak9)EJGoxY_JQeOvtDH4$=EEc|x(Cs>nzP^zp!>5P<~(ANh=@ zKwg<1x=7Z3iC%f280ot2S6^09ebJW5RC!d{1|*=Ch%Q>zmk{Si&wJBf`6T+wS+-)q zRC=zH7;F7HO`Gjza|)mK{Yg7DPZ+2&V10v=T!d&%)6@?im-x4ND!f-s|`dJMeh?;7+_?E zZS0(Lj@H_?x2ifkpfRf4Y%HLRKsvn@Irb4x@|?8Zd+R7{OTxLtaHrmo0HdmlOP5_N zJgYYlOTw<7ji}3$x;5V7R~9$uQu*t%_?#$XV}z)z%=Ac?v9T24h%)8sgutK+e}*M# z5gax8K_;o+1RyRVZ zwK#0Nw>E9Mq^jB$zwQmlkq1W)P~~HMY^3eyTBMgq%H3X}IMJd_o9UDd zF}2om95@Es>JJQLmPA#aPV@2Z?c444lsz%h@YhllyCcvoZ=m=rUafZ#;kC-H@|bsH z)Mhh1`9OiacygC4@{_jpGZ+PT{OQJml7!EC>_U=#wTrU&Yofb|-q%iEtqA8pRMlmY zh;0AJhV5&)`C=`%m*^GiUIzHgf?t#c2r&5kejyn!#s-YEm^YXcc4AD9JQ=oClh$tI9^;F*- zH`Q8YnkrX0eWaFj;E2|Hv>+2bi6&)kyqQ`o5n7zWl0rPi5{!)IOpxTonp`-`I)f$2 z=vD{X3n%dHpWL?`$3n6d>(juu0>MVlw`t^a=o#B1U8R+>%emx31{OP3&lHs<3xvU+ zRrI#)oBN4)Jrdu8&b4O!Qh}Jaa^o1me!OW12Q^-aTwK({8_i z|NHM_lr0ptA_NAv77uSW(^0OcON^28k&#*D!Wd&fpRVgSM_jH-NPFLQyJ^}*)6;pk z?VFxqd%rvUJ+3!s*F4wn@87?9^Ds@D|MFk|>u2m$DGaz3`#Q9DUC zQUIpr{rr>yaI|ix>CN`tNuA+Cjsdn63S$t+Hh)d-_(Vh~M4_AW9?$dqB6_n8wvAqDlv-kiZGEyEr+AJ(t$Zr% ztNm;Z_~MBp!bVaYP(uL*+o1p=bi}Bzy!;lnbBr;6R1dXm5JEjP^=3m?dzeB|McD*J zIplKRw9Uh&g|GqR&33A$8kzgTK#i<=%O@ylQ{f9@{y~R@}#i}%N8FYH+$^rusI@i&-R)JFqNtEl_4`(Qe z&l!WhnaE%psRCV}$=P1{R$_@Q2vb#AWHy(pT&^6-5i5Z{TY}a%>Vei$ekaNbW7}Ai zRHAT!g(M^_0r~?B!uBm^k8S*~7lG@y-A&>nhW#;jao+DW_3o`(Yllq5dLpY#mF>!` z1KdE;li?((MNHEMfvL(<*6}`b*tk}n2qW}OJcVDnns43z5h{GXTcJp64XDa5+JydO z&W*i#mH58=tJEoUzIE@%w#?IZn#cZ>fn2M`XdYwVnoc#Yo2fOQ%5m2Y)LVM%2EQWH zLsFT;HZ{mG`gHhf(b_57ITRXWRN3XS!9Zpna;34}ygdTs3GV#c;&pbLjH;OD9P`M{ zyqWrTvpvW&oQ|BM`e5ZyYhYLj|NUjTXCsG{O91wkA4IfhQrKmemZ-!J7qhh@s$5l9 zlaqB%MgDG*6FyQAO{iS_s2Drz91vYlaKw6T&{m*pkz0R$UDeGME^S>Y1X66K>0!IY zq(e*wLm{JrfU;y;7;Fk`*|sJxAzlJ2P-|^xNl}xkD~&D67KJQ1K$_JDLKR?@RSvoJ z(S~i4R#aCJ4vv&vRV3iDS82dwSa|@s*)>3|1;~`7kf9_SOh}?@)5G35bW46 ze7#`}BoH7W6-2ZaO|3t3fe1qxCECj+89pvH6qaR-Rb|6G(kK*VgCyOuAMLY97Wflw zYb5a{*ya^p%zZ;7HS~?rATX01~xVjl5Ck;Sz?x?vZPBXciDhm zVPA*~WSeW)NQ6)!T`3_lwgu=i+Fv7`#nQvJaLL6evLp~hbVV2j;0=MtYfcZ?PaG$%|F#;7EZ{ zT&k&ASRrhTLZN68fU9KWm*1gmAsMm4P%Dc|&x<(f#b>m!mk<2xjf^B05D}UbBn@-x)k80o=`9R3IgG2>Tlm|H`8Vw z^C89eaEn5wI14_!Jz?wp`w#owzA|mu`kH9HpZ|i6*6aXho;v-UI-x&oAC85Rn+K_h z-8}Z#wtR$>bX*B-Px0jaepgktrQiqVj=znXr`G>mQ^*#=1d6Pj$8KbyXwlxheV18} z`(yBSmA2>GJEe=KrLXD}q*tvY5I-IIlp$%3=*y2K9SKOTY{AB$a+R%6i|OMk@&V4V@Z51 ziq`)AkN@(&```cXAAk9Kr~?RH8)JneMZ}-8Y;~iSPpZ{x#B(HMVQA>M`0?*=^#A-% z?fx&nI%^3o`ywb{z%HPGb)OQ2c&h;of`6=hde zRtsS$+Xie<0f>lZ*8vQaUNm6^BtFX>9u( z_mpCLOpNWo*wj*?+yw$k=14Sq&Qfy3A!7NN zkWq$4v=8=`~UC#{`c7t3L%9QLReL`R@{p|{V)I5|C_H-i~JE{wwD)q+Zbb> zh^!QK?(_S}?o75XZv_|^7>oi*NUiq=DaGEV+yEOp?E<4R$bx2?F3cx^AwYZ*sagZt zE?aAk7)~N4RZ@jpYdyl5vk25>0ue59m?1(`jc%H_iqUobtZN91Pf%dn5-Yn=7G{Mc z1u~TeXaXo?6Gl)1@W?r&*1Di7n|kkUGfhXCB9{dmLZ9xKwv?W!jc;_ zQ)o72DUfYkkOikj7FNg#6bNIhw`q(KL0}6LLa*t;P0XYMsZE+V?yL z7AK4qOw;sK&U%tcKDN2aEQ~^cF#3Xy&6mldm}_Be`Ijfe5D-E1G;ylJiZ zz4!YaXtgmv?mo00+wIn_Uw{48Mv~1P-INxv{g>}bA?Z>G(iXnZ)aI=G{e5d~ zv)T4j>B={3bS>}32VhlfyRZ8j|w0&I{ZudLRx z5m|LeL_aveB4no&m&R;?uKea-vY!@Tgg`bcBw8F(&lHhWWsKGqxK;~6Ytf|LQ^CeE z-G9|LDs=6z_KjRD{d~>Uyl$txx{BbkU^5+u``3MHO9Fq9aSKS|Pa~rDU;h4&|IPpI zf1m&5zsn7be3@?QS9{5B8Az{pkN6eM1ug3T_}lz{{cT`$b2VRA{_(9-Qa(Q|8NgrT z)-hs@{AZPOwV0rVmnXkXZJ+q3B0j^kT-^Vlp&N&WbrCw1%?cwYPUin%KmNzL2%iIZ zx)p3|)Aqmr&;OVF!1S*v8$2Pr`a9w;j65&m0kOpGgHik6%phhEG%2Jeg`v*L@;0?Z z@K#DF;#H{1FuIaVOhc~W8as3 zZ`2e;rL!s<0RcjogaU-VFp#0CZz7_+vkQfwa-a~wN2;oI9Ri)x)JM*byLmqkj3mG~ zBBo70P5q}akH_7?tlf`MfD~8&Lew`hZ*-;TSW)FubwEj)dOKttw-)=|-uMJnIff6N zu7S;FbFgm@=66*|!JWHKF6eTilI8MI&-*CKa58eHsl&NXRaHfkZ+3mw^N_0q{+d40 zT&-l%{YQN5tno=Tvh}Y09{W}d_+!_Qu25^u{GdpIG%`n(ZQm9ZM~VpBnPr27^o^&k zzHob+pA3sDj8idMi)ay*waW7n`1Rr*-xVVxgTdG2UyVVpw`~8^{tNV3OW03~z>)M> zn?Lzg>D~hCE@ckbe@&gd&YwZIp7Dw65}YTh3?M>r?VOY-;Wrc8hI&s43`o8AH<|kN z*Y_e+LTLu1mu%ZF#557xmJb_OZ9hC#L_NHjTHEY)?|1v1vAjW>#y*&%XPAyYwuacz3vZYOfyv?fwC(W5>5x|{!FH;;1bVP|-@jbc0XzyGo|@gKhp+Xl%{ zqF_h}g|yvlsO&waU*7$H-~T%A_vz7LDo{(sL$AM2ZR*?Bwjb=@o$vqieJd3z0+kuE zOklu>K)k^CQlvpRK(6^=QtzAneiX#iT%0r8Y>8ted^1(mZnvBF=`t%lmE&Rk=I!Bj zw>$RJI&P-`A-+H71J3rCv(otYL#Ln3rf;_oz&->qh_kaB_|MgG}Y;zQs zx^oA(<b)uA4|%>6MOIB-^O*c|IW-gThIvh-bodz*C@6l$8qYf_xcy! z-fxN%J9g|?wwl!|izxO2km$R6%3KNy$dq2me&aOzz4#g*3IqWf*j-}roSAu^_Vlp> z1KG~B71qWBmn^KMN+l>3%UfIJmDRk`#zLbA*sQnVzD_LUud|CKjy%Z&y=}uoorMBD zS;#wXZECtB-Ijf_xbSqTs0|T-VC{-Z!VAYI$NLBRbA{TIrFCJeQ)AuIYkN4mxVBl- z*4eM(o7ZvQU^>9QVFiBq$U$Jn7anMz|G54b3#<{;%)xOf=;@QKt)Oa1FcQ27g6?XQ zhbMccCi@Zi`WKJqm&#Q`7DWB076Hgc9c|*o(TRcn{z5VM(w_gd7`fAL#AL z_FdAdEHqfHQ>#@xB%w$uSI!+>yAdjTfBBD@*uz$L#^|9<*4aT0T0pC6lfhe9GL1SO&xx!aDm=a-pB;)>8?-qYH92=8a`yH7Qenl;r0U2Lu zDumxarnX9@-#S|2_B#l{UmtwVYv)TF?$vRVH(1-A!gt>s{*V8%{U>jA{@`r)#j{;+ zoN9mjeA~bLaqoZrHjv8*E8?F-7(mgVq5Ob+Mm5O z@w2zP$NG~lDF7s*SGYFgmmdh2?LRDP6MiYBnv?7-rL=YS8A@A(P+)0v$bI(*qcano z?Jf&2jKD89G(|pbeeY!E-~QD9H$O@I%b%R+%}TPmVag6xfR?s&Xgu-n|Jm@r`jPzS z@1zI&vcT-rvD^r9YCI$6=N=6Ent>S`9mPhnaXD=Bqw}Z6e)@Jtcbg>8@e>iq9z{JV z^X9RhfA`~=fB)0zfAwzv_fMow9cnv%FnjK3`_JDR_^$B09nHZ8}9A~fDjS_pgjc_&W!z=UmX7Fn_X|8O8w|m>filv@IU&xuVB3|Ii$ zn8Iha+%yHY4Ii^OulczTHh($BMwS6|)RA<~pBz4YbU2*^T~BSZ4ZCBOf_&d~Jx^;J zDIJHCKLdc-3Q3tv+H;*S46D_eu?&qt=*Hmw-#Ev(aR4JmQ4~D0ZVsia$4!%<*p;wp z6I!X1gCG=A`o14Oszr;h)j3M3FbvA&QWQlbgr7*HQYp{%;zOu8vDqAOY4I(qa6UgD z$oe1t^OG#TzHD1Jgm=G{ZxIc(%CSKx^H&t=5E?7;b`O35Xz^#ljk0Y^t+i;;;xC2# z))+V+#13_IrlzKcj~(+8iNkDzyZ2VKVT=c0fQba| zcROLkmn|Y2wo}MJ04d3bhB^;U3~iP2t82@KtuJ&6ik|EthvU+s6xGg*J$-WI;+u#3 zBz$qBaO>vn-1hEJ%JXbla7J>7tPy}n7}nOlU}cNVe;pzBk4HQYaQes2&w>_A)x&B^bOFHV01WkImvb9+ zRcnYqFsFTYY_PL83vCk000f$u>v9Sm=>!-87$F>MY@<%OuT~hZ?pZVb%!)zdz)3c9PSJ&5f3q`}+o=Fe&b`K2>92iLt z4~_TV>iYQdot0cjY!H;hAR)n(5fbkqsF5^5go4?XYHTQT;p{|L79QNoEiSB)P=^kW z95~o*&K;^1YmXPp1Kp_~TsUyzL}qmp?%bZu<%-#iJTyHxHr9=2x;OHrvQT&p?j*kO5FB zW3*)p@f0p1yVCJJR|pYl9R#7)hM$4^hQ;yk5Ov**&aYQuBO#>gdWcvbuQklT&G0v^ zlBg^H9m8W(MT-_KW!u!Cz}WN_mk6#fzNm#k=FY?I%U2heQ&J*eluYQcx!JegI6FPn zefHec_I7!3We3<1t|J^Ng>Xp36$v*JO{ZGh^f4enM&|Ce)R_Y#oo#se-qy;- zb}dSij#4IY5EP&V>$}w7y;qUzdi&x`!s9DfpMHL2VST+4DVHp{tcXDZOC$!?S`m@& zdqRj<*;s9?w#pjI76CoS6;f)WL#++iI4elHu8@vaDt?-lktDgU2h2p&uv^jET4YQ! zx^s`0pN!9c|IF0tfxgwklWG8naheT34~9hx5|J#_sxa6K2PVgE%;tiKt<@4qFm|A3ZeO*8~6W zd&|pfIg1HInQ%>KLcV!s^6bgsTn?^Xd-&w((^_b2iY*AG!J3`xyw%(O;Pg!IvExVI zKAADqkqg;wu*!O>FKHVtzG!#M|bYrSqvku%_(lIF8}y;={ zfPw_3-2_N!0ECcX7zIH90742uLb{I9hS_Ek89~BWt5l@oQDk7tu(MNKUVJn;Ix{&j zbbqlL1SN`h30f&d9RIb4wG7Z05EcRwN#AuGrB$sKG-d7UqnQ6@R(y!hOZ7@=#2~h| z+WT6qR+(+{UKrIkg8vM!?!aRIF@qT`TG(cuvf0Dd=WiWX2%(hTL#fDELqyNLA6tBj zU^ex5ViH6`ptbn=#jLu6gOVhC*E3ohZTMRw0qgOE(psyO{@UUz5(+7dt_N}QOF`}Y zXBrx{;5v@u0z}P|2Y#n*v|6-i(NebDI1fT2qZxtYUd=zrg<6ml)*8V^8Xy%UsG82|lSc+x&zdR4bdCd$Xh4TZKTgKy)lP3s08bI5Tx* zX87vEm7PEXOCZ7*1M$#86u@pSpXup5e7K|kMy9w@vS=Bs#IU0++27-DZRFci>13Nr zV2LA%m84YNeeOVCe@AE!{ zp$Z~N36}^WtrRV922Z!t!==i0)fJeOT-)3%Zfc-|yr5rJXFz9V==88Kz zV=HuauS)$0y z_4H{-CjC8BU7$C|;z?^$sgx1G^Sm&qX|~Kw8@-meZSgY5FW7OTjh&OjKpN??vxo0X ztrP+)rJxZU`tniA(Bkg{?~@_QD3@F*1lL!N5nt^&4T#zrt&QtQ*KtD=#O`?rygwk| zo6$(e`P1F45u0{U7#C~}!|RKeyh4Ka19T0ynUTFCRfITC`~KT~xO5 zdC4-^FMZV!ETBaR0E{k>r!_GXfCuc9YSp!+o!rvG^vvn*u8i+9u{}DK{r;JWq3(>! zU=V}{!tporogZFVxO981P||E0%9c3}!mYMvcw)NS@1TVTD_eQXm;i3ryGw>Fso_pK zc69K~?@xF#_~`Q7hga6}o6N?yL<|uiF=~ro0Huva(&_J=8$Eq$)N7-f0#|_NKwx^F ztakkVli9luHw6@?Clf!on7nbLZDoDA9EiF^jAq81fA-_-`T?EF#Fzq`B$FgspIx;axwF=Zi zM>~*ACyV*^PcOS4TzyhfakHvL*3DCp5mY&J@aW{Y{Ot!TYa0b++Z3CO8;o?Bx4(DL z74W-1UVXB-($$^5_|EZ}!=ny?F;J}ySUE8=eg1q$sg$wWq*9)c;J8kxl6USTKe+m6ZY!`LBXGC#+iNS^-y0r2 zF`8LkuosVFYE#p1SFt#q`vClh|9QZW{6!)cj;v1K6I;#Coa4YNRqr)n)L zrQ#7>*L9^7DJQM>P&$l4W^O9nZ%6)YtF(q-*+x<1O6f>p>g_8e2oM12r6n!POaNg$ zA1L8_z+4S#%obsPW$x?2S6}}VH7$c>OGJnyr4T~a8w3AQ@Z+ zFASitT^0acUG2U{9qq{zM`xx-Gn*Ub-+yxFm+yW0+xM@{&2A3&!iCclgIy_5H3n_~ zptLr+*4ge34))ZdV0C@7T-C->)4~aS!Ts&-g_Gkyy*M#B1nbLdw{AUN-UZ@?e6Sy$JlXi|`=9>i{Xbs2ejfo&pYEET9t1K$q%3;ns##rK zOC_PNr^9oIfQbQ^_s2;(u7?743dKi{9yt=mM*A{J0UWW3+S2mK%-D7=c(S<8j7Y%3 z0-0l89{L4_UzhLv1Xl(f|^??uWd)w5P_L&HQ|}z;g0^n z&Po6~JG+wf*ih$N=Z_pa+*T^+&p*HO``>?d?ebk6R8Jl4I(le4nt~NY;Yt z=T{%T|NAR9ZY?+jXHT>rJ}{W@qy>yvmaBSwWjn0ENPniYJ%PlY>$r{+ce^!gN8;RI zBG+?Bpw>!hWtm~0eHbEO6Zq?bFJJcG-n@cH%n*iQ5Y$Ay1daIRL+TyNN^2#BaHNb) zmgD7>R&}k#YYw@WZ;35rqDTdGlv0AobseuB0gbJOYi@O&0 zeTSC87^@-$mOb|wxQ}oCp7jl8`eQR>0LPK8>sni%@nMG76IT(16ohC^y&0KfS9nfcxjL3leQYL^$k*ns`cRu|5_In@S{Nwe-Prq3B=(9VIXCJu2 z^mfZk%6L+F!ngt)0RU_f34xjt02zo)#y5k#Ze-x@{MzOFn;(6#@#~KszW?!~l{HOR zm6A~Uk#QpBY;NwT2!;o{GAUogn{Hq1E!W+TEJC>!&do0cH5eb+d%2#pT39_kGLUV9 z)wO(nH$W1G0UUIa9wO!Q@X@8x|NX(WUtgX5@aEDdpFg;E^=_#Y_Vz(%XF3SWhLr)+ zSZxIaMBj4<2Shc1t=vvz&@xC=Ne>Q8j~fHGZ{4p}%hS_+r%!iRD)8Y)H$MLC@uyc8 zKK}gv?7~(;f{I{eb3I=P$`NjEZKo5E^bDevYccz`ISJyt-8*<(O zlvyDXWLZ^qJmRTgICIE9aUk8-1yKk)C0#a7MfjnUF|s|fj!{;!YV*|8Xm(FsXReC7jAMcz5R#-LkYW?AZ{QTuq9u15=E=`Y zxuXCOe9uiJ60YMAwZ!eW0z`ODIy z0BCKjwO`dr%of;EJw_ji;>Z97Zb{j{I9$hCD~RHRe#@=!ukRu)TC|kyH;%cr*(Pq>+FV|>BV*aqCuhcbQ@+FSis&K?l)e9W@edzgx_9q^Bpw;) zK0GmW`pD$iV5ftsE#;*Xo`WbD1%qITK>Ms{8JYw^R49~EKAbvs;OwcH>8TONmk;M= zKluFz4<6hPf}l|+1&E?hE68Rc;Q@dKw$O0L;<&Xt2qVKlPGWtdl+WpbUKr|0BxRKG zae8Wu0UpgRmujdC0)s=4^!!S>_Th(rxOC}~($UyZ@7Qq9(IbbZrzib{=eXc{UR}~l z25CXW8$tkN)4u0I5JIIESrkYhQKgg4KyOcOtG2e8a}(ad!$Zjw+_=4d=izcm`9@?5 zWwW-nU9CZ>#D!v6IKDPqs+5cdS2_f)V3GhsJ*juzIzBxPn`^sYT)(xMs~RTcFq5R? zM<-98nkkp;#~&~J>b-lv`QVGIx8{Wl-#dHgz{G&>FcHMYoe7BA6MAYS`MnFrhbQ_U zKB;|j_0d+ol=Q@4UuQN0>zldR<>F4wtJ%!jE_{A><-JcH-I-mfMqruCRd~9({bapR z;*@sUcLUDlcR`1lq@PLnNEiVdkhPF&)oQ|nY$oCPk_famN*QaPeT}>hQ>V*zlG&BM2y947==Oz*NLkxzzi`;MMfJ(?jah2xYae<;+w#+U7m`fTCJ*$ zmO>H! zi^kfR=Se@z{1wJgBm^NbGjJpS78-El@|U4ixQ&~=nS_AGD0Yh$En0j-LhgsMdt+@()aP_YU;8b#{85?>W$whQUr)&4WNBY(%=mmwFrv`@VHuP)dX$ zSe5{$c?S>#`K`T?&PTT&ZXTGn-|>WUND2^wrLZ8_Ll$vX53^;<(13EgwJ&Zw>`0wHIGP$6J+xJV zt*!FvMs9X-#pXlH0BDIUunK|-1NaGWTn(TA6U@Mkm)U5OHV7KL{7!WL-u%xd4j!56 zd2shZchA7s=)mT-SzO*`N?Q;Jkdm&46a`v(VtQhHuc+IWP6%QBgiI%?s|(5z7=s87 zlFuLvATc1bB<{>690AoDl&g`YBqBIM4-X7v67cx`dNtIeV?ARN-Gwqdm|Nbh38pp^ z1%d!TSPSdx8~J?EaXjCrY&MGsRtv#NVoJF%J=v8DK?68KNZ~OeAtz=XUYML2bA^3+W4+~W3HwF)xj0JnS2(dnMIE({Nir0>oZ|8VL4 z_bL;t@}Kp5YllRBxGwn-;75TV8E66C1#YV{OlZSu{5V1Hf(puHBgX?B@J~_2BZo#cOwF6FyuxHat1f zga%#wax9FBBLW6pAbAjF6*AG|IQE2 z|NN&1hlVo-WT|2wJzBhV>s~Ie9S`DICPJhy)e8c!waUiFk&ujx#K1-%WzrCaMj4Z7 zZ|~^w3pKsIQwl9Hf>t#$q31Y^kk40CU>&4v+Ue@(ELWjawuFKiHA1llcOP#pud0#Z z%*nlm<0OKvO3-~h*`A){Zi(lXHn$3@W|HO5-`Li3vx`x+)Yp}1OL-1x7j$>poSDhK z^XB29!Sua{yTAM6o%>I=%ZfA$B8(F@wZ)osEGCc=V1)$f3y~mnz&gMPpsnS4<)17| zCdopuL>2%7fB_97b2GY_2!uoc&~d5>BLD%S5afE21S)0fF+cG({H}mn>nIF~>XB5< zgsymcSS!tzUDu6|jFzlz@y%&yA~e=UQB0 z-y2=m^?YACjv#>m`0^}Oi*G{gdH!XpFt*mMzWcOj(Neb9>|8=_4)MJVA~V`&mxSvM z4D<{Rbtnz%>v?89HQjr7Mg#$T_|dKZ`TzXg|M|;5{OWg~UA_5Wr(DwjmL!n)vb(bJ z#WJY+H-doxa9`vKR{~gCFE8(^b?wh@hnH_HEY9br`r*Qv<3mHkj+6xZ+Ns!frPR5@ znL}f+y14b;dsqL@Utam&e|_oydjHDfg>`E{(jFpSxsIa{faCE?E2Z@Pgb=_CfWl4p zIN7oJP5b}+;r{>l|NP`T!eRO?!F*i8@Cr?jzb+%c zY{3Fp6e#?yt(~>CTu)bOYOMdz)R?t!_uj)&HDD52$9~Y-6Xx(l-`f{b#T|J6gX{n6 z|NF`R`peIL{oduvmv0x#JCOnaTYtp{v3>;`|0}H{%K!ua)3Cv`5Z=3(dsj*+Yv)3LD5AH92 zeCg)>#X=#Jfh9%}Tbl?Wlc_dz6WS6R;aM2#%>MAyu^${i*q`xS)Wl$H049_=aEx{> zN*2TjAQ>b85NHGgzzk>skVpyyV595h{l>N;DTV6@M68>^+D2bHeydw7_a(1@QYr`o zMD#pQ2-#eQA^?ojeLdGhZt4gwoD#N)hZ z4PdAN0n(mMyEctBMY=mJIy2H00IUJ^V1M_;x6afm!Ka`6VQqPHb7gJGs|2O(-m#yh zQ{X2&q$)7f`*wsB7*s*4Bw!YkdQTmj0(e7pb5)avle z@oEGQX4iI$LB_R`WE8>jdSP~XH(#m@4Yti3oqprQSaEfEd#jq;E##|_muMgE=~!7S z?}Pk?VSnBek~8Za!Q8z^Z&H zg&<>izd7nFesjDjTYNWRgufiGKCF4)^;qJD_2?|;S;Z{jI8q8y^?2oGod;hhr5g31 z7yW2(U$Zq-AJ>y5k`BKaPNH0wk z;XY}K2n1Lkutlv+UC$cJpe>^@@)_dU1q3ODI&Q^^|3i3H5gK3T7xEfu{+-Zc{GzBt zeDTJ{_HFK+*TC5LO3--KM6|!i0EqwKH!#A>H{$*)%wP7?v2ASWR~1XX(R&{@z5+MC zzKYfoM~fD%Ia@>kVzNk$H#+3FQPdMK-0oCIx|B*=G%QSe7f(%RXAVql7R=QvT9i7volO_|SB#rfSoIn1|Ie5MZ5h+8@&m$HY(P9+Je4eURXzxsAvngJU zn88|LOJp4eut)^pSc@WEio83IHu}3Bzj^W4#k0v`e)P(XImD2J00Lu85Jr`1&B~6x zv9XC2Q!F{{DW0C_K6-fAcL11?OfA%#xss0Hz*yJmL!+rgkQAk32YU~Wce?}t3Lzx5 z+1D2JO!-_pGB?}3vr|eXt3yLmY#_HC?7?+8iD0_AI#Nkky34x-7chYc$d(!>6eN^F zxP)MljF&E!!`X++Z=X4O{P2L4aQ)6^se%>=&^jIhfsh;o#h_YX)jQPRGdeOz8Sq?p zs?Rxpda|vJN)Zq_Y=Nv}VUNfnTXZ$>PO-R?(>-mnCy`pRMaw`yJNbO2YI+Acl9{%` zZgFdA^T6TWi{}nGTB)iyIFfzy{BSl6`6>(!W`}zE4$mCTw8O`jw{JgQj0{K=5>OI= zNYqU7$;Qs*+jE1%lV_)syT>Me|Jm&7R{qxP^0^c3KRkC7QOs|L2`8K$N*_9Wpj3m| zh4oq(PK?-&{7%8{4PqUy@OcZ@+Rga3kUlIKGb9XfQT9l9D_} zQvFT9GTY}vs4pv0tl!#f@vzqQymUHcw2q?ac`23oWg)VEXT`b&5MRzyXxpjOP4Rcn&s6(PcKFhuZ+zuXs0zQdI>9Y*+oDBF*?wghyR{T*2sHKj zaQ@8Hp#xJ<1Rw!jNTnf>f?OW1U3>D`^(ULVsy&HMmbT}Y)pU>Z#+&EH1Mmp4J~(XG zwk$i8kVyewb`2qbKmvAlcEiF>ZLHsIPkR!o)-sBOrbI|dg%E0?EeK>Jhf(g9YPasL z_4ZF5ok^ZOJ-N17K-kd62MY*PESX1h+kL}(p3J>%$=*sCJP#5+>=a6=RL1czmF}?M zFR$%9oLxJ4c;JmU4^Pg3N6_X&p;+>LR~v`XU0L0_a^>dRZ=XMXc6fMXxKe?RPU!4` z?L0hs_;fqB%Pg!7Tz~KOtkSTZD+CcMV;$LefdoK6kwMK)s6iQFP=bRi>$#Je)^Lit`zBVy6JU!BP@6N5gP~5)m-V+Bp+q?@# zd6@)MRdH(<+A{FH^JmF|GH~^ajazpgYo!s50b7n#)j-5aEt0h$oL>oV-F^DgH^z?~ zn|`8CTP&D>Ugw`>e*W17(gOaz2Pf|#*+b93>T|Fpjl z{+s?i^1hXGj8paF09#S7T4>FCG{T*YvBn4zL}bis&-aL}-q7zh41ZZ(r3U!5T?<+% zYqeIf_HcaPbEFeSq0z=doEdFj5^rB8S^xj*1%&VtU!(D{AqboEtAZpV;WHe^0R*kJ zg~w1_f6Z1bJ6`KB1 zs^ZP_D2*Cr~gq_vGNz@DI)&$uB>-dvn2rc>LrwRYc$i|K~Bfw zlKJS9+qtQ>zTWP%n?y)#tl7E6jXMt(S2v5vOqW)A>B$x$TLvlolr)S2Pzt~bfG1tg z?bM-t@L;1>y?*GxXtuvAnY4wHS$tghVs`z}(~=8`C&ikxpmX^$phHr!wVGjTw=%o5 zwYF)3PysNoNhZbK;5Q0*x?C<4L(ASvWg;QP9>!W5E8Fi1FIRF<)p1>)2$>Dr#>BO? ztc`IUR|;v2fp1ES77YBlI z#iFuaA;iyaKlL6IN+qQ%v9$&?qKvGw8`ztVwiefN?56UiDg4#DwpJ=*6x^RJt0Y#32ZlR(vI|SwYipY-VnDLi)~kX@ zfCQisuilzlSl-$xmK`rm zW0KZ?u&^^Rn>#Ss_5BOyu3WkGcz)^0>az4wRyHdeg|HSdGb)fPTeaVP`p{2q6>DxadZx@DF6^wYuckfMm##ut zF}PdPyH#!3jRJe^-pb-!jSz?gBC%zztNsAo8at&D79*CJTq%?`?=8=+L>C`ODYvdT(7RmyoHD7ZgK@;FE0J&~L$}#4LR5@6fBj*gr-E)>8 zX>B86BmlDox1`Z5fN;pOr<9~Qd)C72FGH5O3VNn>Z4}5ktu2ZWVs>xZ1U>{Lt&1mc zK$4QXJI!NJ6d}@NGWUf-q#oi|_nqS^-Uq`4m>_^g5C+JINOqIo&4M6_fIx!7%IY!_ z9fpio5?5ALlFUqVfPl1|IQ4Tch~OO9V1o@d%=UY?qk^1EDnLiIU#E}1+yz`PAqqt% zGANWl>5Pu(20)HU#&}Q7;a<%>huZo56X(beb2ykqF(Z5}7z!vQ+`hek<mvv!ntS`{-oc*05GMe#J++h4{hLWZwn;Gx zCC$wD+O&VGX>(!a(@k6LW!t-(dG{dG1qwlS>e}trabGN8rC{oJ^aVL%ixb!H?Z5Tm zCx7+VzxUefm#$pDwOfy>-M^>CjMH(S*_zr)@mEqt7)lfC7;X8OWj%l$eClQ5q^UgZ%D}^L_#Z1VhnS zY#19&raQIm>=Q20B{IyUoBO(V(6kJ0;|Slb!sQnp_}EW8VB?=%pS7xnNn8>-Zw5)9 zLjw5=C|`*HxZrJ@6tV+fsol1P`Q2go_kPvSg9p(;y4b5J@T&(B(2C4TM$#yU%^mD!1}KoxESP2Y zl(UR3ha77rUPi0|DPsOKm>D}rB!HlSD5>Uf$ye7F*ihUJHuyfV@!0|rRs}(wMZ(zO zTv(w1L`TR7I->k2lL&!wm@py&5Nu)0JR>Xx90(p2Y&4as+e??XPM&?`wbx&{b9;9>YY%6Kb#rJD;}esUj0 zAOo@~6E3*s*BunH140A@s$Vc05Coc>FrXA%a3BH@nL)8?K_Ead)9F*&7oT}{f49AI z^{bu3Mxn~??t?%&BA5xvIZFZr@E0p9vfunBgn2LA2qS&b9cWp$fe*1}P4az<&~C7S z1edFIch4z=*zvBIa!PGSx{gsBbF(4SVVDaqf`pkFZ8WN441iqTrZ#zQQ}&)NxV&lG z7(-E(lIo_R3s^g#)KLdO zfsjigNJ@*@kk+V#3ma^(!G_s>Ye*ynI)bCL1$Tvk1SliXLk6cnQA|J*42VJ@6j8ZQ zYAM;OxF&_8!7C^HFhEe!5oQ5WI!hR2SK8l0ueHmC^`$kDL{Fd1Pmu3mju9KQRJ*Lgr_ExUw`=} zgJ1ddi&SWa>ubZUo zxqFTxfuKw{8F~v(rwB+8iW|4?|IJ^;_=NXbF|(E39gY#;zq&e=U%X3nuJ%h2Vur`#+`wn)I^m29|tmSBW{4XqXC z=t6|&vBzdHW_jB*DW~K)OIZLA`dr7lu8nwD(Wk0FPG>cw$9dfW4X+3QnO!bq&vlc` zXhs5BCp;&~A(yiIN(~q4p4JD({KSe!BoNKYva}ep&t1MbyE|w-wO@`1*DDcmd65!4 zxjTZvg3S@kB&R&L%U6+)jf|%NLcfsYmV}1;7DK(U7=m$u7=m~IX62F~mpg~T_ZYp!WIQD&qWAZi(9-usG4GwY(A4InJLcZM3!Iv&gKG7lSUu)&7e3d}u`kdOrl zG!Zn2ggGEUbie@+*4r8aWbdK`-Ju2N5!MS(03uwfJ>I-OYyacZ>-To<@0KPvY3bA* ze9<`s2Y4h?mw@%*N)q8qqi$)l8RhAZlz@nr7X2~BSOH0xREZWRvrMQ%1lTN<-{vGPtVGfpyiiE=%rm4AhegBJl_dfXG^JmYT zetLVdH7-M0D8@FYW>y~@)^~UIZr;BZ49@|Q!jpo~@;#Ls%}Bw?AgN;)kT?fcxdmDhWBmX%Q8(B? zAS7vi@YY>%fr4gi7vEz#nmejT%>qT~S4tI`>k7>f0%0^7%d+h9TmZIBopMg8&F;S1 zD&x}e<;ySUftI5C37=l2(&;m&;kf#Iglk{ zJ;X)xB&_groFu-Q*+&Jm7j)79kOn}Ygr;d-N;zfs;misUxh6Pxb*A}cJ>pi?(j7ni z({4@|jWLE8jU;Gu&}cG$2&x05{>dD+YO^c5!3G;P1DrPFv z0Re6>$V7KIVTJ-C0MUikfkZht_|DZM))NApC2{KJ-Pz5%*M-V4TF^Ec6uP?whonBV zl^W{Hcc(5Ot-h240I_vA#eg6hUZ~y#wuEq z^l$IpXxh4|JtZYE%hh$xu>gnGw)-srwj_Nb30sW2D7nM);S>mhm9-Z|IjX8r83E6! zY11?%Pq6INvV!1>BhiZO0~b$#x&shEK$OVUy@L7^ z&JQwaxkcP&e(>>EA25SCmfK?%5|?8yi4+L;EZ0#F>JJJnJ3z5NgM)oUoUpq5B#_uW zlz=4vx=Vx<^z{dFRW4NhWo^QeH~1bgl#LNYc+=+ODdu_Q+aku$rqHJ33ul~`Z>4}o zs;>SLzMDgiuwLt!f<%xch%l$Jv51}P+dZe$Hcgv-8S6_(#L^H-1Jhd9jk_KW{%FLu z&0WeWY}Fh%w5~4d(qBo&7?)WPfG^o7$<;iuzd~w|&AKuM!^stRw6-ouGsdE@SS0t< z(OTEz9IAwlFmd%A9$JSbB!@F2XB7ouAzGNLQ-$o_1@-uVFbM+?{KZlD7DV-=QV`r| z!OX~_h%^J}t}ez~a}2smm(&mLa^859N|Ng4LY(NTQqi6hgFx zz_)c}3lg%ps3?0DF0rztu=#mO7*G%;kbn$P(16wO=w9LQal(+tKv>)wr9ea=EPaMtg++&jW0n*k z4Hx{PK+;G?tBN_wLdvtIIZWL?c=jxa4hgolFd-={V1SN>U)d3dqyk{eo-|F)xlU~= zi&zwa!Kz{uQjTJq*fesP5JU(RMEN4^70~L3L^r3A3zjYiR*vs(O!_+cJ!=Bd+ev6; zE0-=}Cc*(9xKK^(?$NRV>mwGAcxtHrs3p$@iT;IJ3+j;#bQ9G(gShn1cU^fjlUx#h za18T!O)}8LWg9>c;9|^p?B_nN93ZiR15S|Az=gbc07VcRZ1BS*_el^ScW)OV$}z^G zD2t+KQ(9!-DoYyVOR{HrzY^f`M@PRPjUj|iy~&KEfks=!CgJSbqBQ0o&Iy4&B4`Vh#-vs0)v?qF`AhL z%Do#}2PEid=N1=8tzocS*Ii2e5^H@j?3xD*t?{yklX8~Cc~Un3GqV^7BV3*#0mzWJ zFlPZ9Y_P%cuu+hr5CN?shY*42$UqBb(jCk+C=dnE8X5wGE#r}>C8bye39YxGJjzxS z1p`W;BmgGBmuzZfZ1vc3iw5>qOudya02550AVC8cK_Gd=E&v7*-nE(lr7FhJXjBv> zp!wROsb{mCla9y^+M`P{4hI!jcf;4N7*K$On(oisS$mt5V`fo?Kq7=uF=p-Vs#|xk zcayp^0a-|bgYJkff&r^SF-CabrsY8(!F5I7K_7(4X~55kMI16$mGFY(;y1`o>g!W4 zK4LaW7>5~EYoN!GU8gYUip2R3mgwKfpAN0{KJq#M21X~fQuH(e{O$`SiGcb|C9>~g zKRpg{)#t@2;Q$1~1{-|OAchbnxiI%MdX$G4x=gJR^pY$D0Lfi#3#ce!QJ7gsbKk9mwO(;ak58Z^IvnM#>d(2ZF-IM? z?Aeo+R;er=_S7tR^5pZVX+PQsk&p40YJKxS2uGmq@7w_>DVY>xwAfYXi@~xfxu;*u zD(6FyEs{pgQ?jG=&ePN;_fA~`A_e|%ES+4hVWoA;zlB9~C%G(vYi33iW&%hh_bds> zgbTt38*H#)w#S1}5JbX2$X&IgfteE$!9XAiP!2?38&Fe$iWwn;phTyRx%n=DBpl38 zmJ%Ss^pk~lG$bv7q(DeemYf6xxY3sW#4Ly${I#1oQc}}2 zv)MGKxd%c+Ge4xYe57s1B4`9$nuS211WdA&l$}{_$)lkPWn~N{BUHDI0Ph(N2rvL9 z9E}>twdbLZcXl5kJvI;Sn8Ns$qIj7D~+ucWd(kxGs0wB11(=>8NS(vdX zBaOZH4ueopaaFT)J@XNcmpQ>c_%wWj4K~;?TM`6lfCNJ}B{P+y!pdsD)~s$JqdY*S zG_JTcDo3TI#-@jU*ve@_7BXm*;W!`=0!E`Egwb@G+tvjmOi%jfGHH8Mjmy$=Iy^j_ zc@_vH0#zs`)g;EENxnCm&H73(CJ%upwo5S0W??o%U<5>SGZaNR9*<)P>7m3@Q_t$0 zGaLemVc~Ln9F3QBOdI899%Fv`sgu*09UdkyxI<2F#WrQ;%JtNV6QdJbbJ5Fb%Bh|^ zL}qnz#4KCzSc!mHnoJa9d3eCOZi8tZ3tS3MfAdy-FDxM=j38yETryXqP*&lv^}Sgl zDExdtdfa8rF)XY21HiD{Lx;ojxU5EH8LX6aIP<#Al3bz<;t0M2$nHQ4HkJ%t6&#I5 z)2TNtT!HMf&$wAAZ17V`E0qL5u9V;|Gj@eQ#^Z5LZPV0k^5mLhQ#RRxrTy{u{9;kNpg-cR8`sikJ^;lw9tv@QvV^4=!)@n1m1zS?AX0SM5ffBHYc35G;^H5iRzb z^A5Ov0oW};_7rm-dmK8q=RD@$J7x>`DmR+L9hGm7RaO{zd`U$}68fgkF_=in1t^$- z3}#g@lgqO~#0DE|uwk~xg9rx#5h)MUxwGTn_>CVSZvX3#uUx+|P08p@6*VgJr3<5% zU%veG$&;Ude)oe9zdURXxnQ;eWRIqT2zn$mnS(vus=_O;Tsm|1(kCB(cIC>olwhF% z_+%Tn1pMOp(Qm%`{O)dg`<;((?j|A%Nob$m%D?yK#dGIhcu*AQU*GxU=1wCOP>5l6 z_VoC**Iyiui}&BV{@G{u+O{prA{J$g7K)M#b=}VDx@nq}GGRs|*oS>e(AR0LL{i_l zy{e;5OD^?uQI{oNef6bhp1JtrAAf#uuoFU*C%{eZsPq#fyLe{&;-&NF&YdgE0)|=L z?CtLy*6n9s-nqBa+`XStri)SnXH~3!|F?hb)Tw9w?O(4xsD)!F^oXNSkk-Wk>M`Mk z0MvcZiDqkTufO)(#fzuj`rxy-uiO#>jsy0CakK#f*H84Z>Oe~R*=?g+zkUUfszPu4 z==_Tp&sL+cvwY`+uRgl+Wu56h$TH;j-;PK>|J3-kS6}(^t9zeZ{c6@iu%DH}=LQ?N zyC8SC(YpxYjo1~xT{j0UTh`vG=HBCuz&Cw0C@uhi()uTUHw-2E zS1B)9k|z^`F@{)#2ou~}sd4#oWPXl8E!w)-sX7((Xr2WPa7jl4Sh@=xs){dNKKIh)(ZT-Y>eW_GSOCKl33AR) z&pf3+`Q6jg8asFH-P^BQFGPW}7cQ26|7WM4d$GE5_59Axjs3k@(*UsG2hX0}`tv_J zUzE80#aGucHDXkZVp)b5A3AuahfUqMJ4myVpuq%?SWneE2qc6xumKV0koJR8E6mQH zKl{gj`l=lN^v_e!D4Gx~Z;kEc7oWNO-03%e?c9kmiio|vmKGUeD4rIJor6nPuI|43 z-kmSM+L_j^BU5{mV)pyLeg4vO+h2Uj&#z9Y00v=QR_%H$Rf_?D>OLh(0wHob8JEBI z(o>gT+`4`%_!Sp|zD-K10eqY?Di_W0C(R9#_@r^h1oT7DS}6dOl9Xxb#TU;1!{5I0 z!a3~E9{ybaih)j92+`@HAA7%CPb2|lN)t|h{kf+uUwZmr_x@)$cO6AP>-yHg(9?V(>9$n?k{%=a+o8=`0!|yckZ>tQSu?Z9G*i2_01U`B#su= zS&I9qX;RJ}f`xg^YB5j9%nP{W0u01DXgv>ED`+GhDYwQLqXmm08X>u7b^92}GeFV+ z;->`>>Q6wqV3IqO9Smg*oflWqLdW~(d$%RdUn4;-DSO+tE`g?{@&ZKwBz{4lerMX? z?o`_{gd$oI&|-rPHrO!RCj|$hG6Y>pLMqvuuHX@6h=6F?D$)xV%0KC5TOKmAf#+`K#~CvGePA*1~Lmy5HR&3l@aZUV){pKzHsRr{_}sox^w?N zxTV}iOt*^J8$Y`AXMgzWv(I3EA8&tf=k}c&_xE-sZf%WDpWS|NeBrrEzxSIbU%YVg zfB)mh?|gFOux{mEF~;L47ZY<6 zO1R{3Na60?-RqxU`}042@y6@VUcY;9zwsg#{C>1UHrRki(;3&sU5KFgHQ^X$SN1ub za}R}wSXd0zJod0{8cEBlqC)>4KyYm;sE#y((7Glb%@@`B&aA+1S>;7swPh$;AObX+ zu_!_pI7+f<+j;nL>+Zv`)H0d}I@;#`jNhb9J%F?%$qp%Jcgr3NS=WeYAr_uHI2V5| ztWGG%*k4$FlC}&&(o&#qRmqJ;(rB!zQW6PCZQII~bM~$e?0(D#MTgb~{t%T&Bp)5~wbgIHJ26-lP7o#j4;p=!3P$1adu49F8lYE?F2O-W8Efg`5NZC{F zm7*XET0ws{J}8E%RscxUev;4$>^=`o^`JH*f0xG>W5YjH23V z9IW>y#sQx&s;n|D#vJ3JIeJlC#n~pIonQm zKfm+UsdJZ}J~g&ck@oBCZ4ugwr?*Z7eDL|z`#bj(C)H$9l@$O_Z83^Zm6N1!*yO## z8Z-iBu+t|_w$p>u?2X1japX3hwwl(tX$6qVkcY?^xHXv=4Jr<%Q%60SY)=n3tFvQD zh%%B6fTX-+dgFzs|MWL5e04|fz4Kn1qvt%LPh$JpOI!c&w{M&tly%?xK{;bYfg>jiRK~rfHpLEgd4fEOGK=HJjzx6xlt*0Jss8Q86y7rnYHae|+t3 zRh2utbwnN&VZ62Fne8mch_Yl6N>A85*qf#rD1{;-7it~ML?T3t%JS4?vOlw(eL0kn zs~Ap>Mx|Lhoz+ct1R=-hqwyqV+^r8<0Sd6lJBXwrM=&SdfOpsRPAR4@RbnDIdae?aF|~ zB%~&V{li=*3PnJkltq{n2uUe;!t7!8b0e~hdhTrb@`Z6_xPAN1ho9cMdRuo6b?APg zz@!Yp;;1aLcu?ov7Adn3qNifS?GxM6Lq9l}Zf}jPKz3yL!Qp-gYEx@ggraDhS+I7p zU6yeRVrpxp`pnkkFr>Zx>?o$Sefa(dfAr?X-+kl!Kfd?r?pJ$2ASe(UZ1C+s3hO`< z^o{O59u=~)?8;5cltYnXh((NLS;k;pnxLzD-m17IxgsPI#AWn>1W*KzFc#`b3%78c z!0+*O=ES~#1SD~p2uWI(Pg_I)?w;B@XKz!IhB2d$G;%=mG#EpYl6sOQk{IMLYDq$e zPcjoEsT*4}3sGiP%(oR{gy#&&=g=JPPC{CBX*fQ{RRfnDu#V8uI;k)akbCF;oPP=( zD57gC;j6-dGz~c<^b-kPJ>_scu`#xrjxG7L2hfPJat<{^6UWF>O%c_2$noSs zc5BDq>A}x`m3y13b^5}DaRea_X_83eA@4#m>ht-gZeD{nTitT0)u@jcBjJE+X>qyR z9vmE()DcAp6Jk2BzLE}y)2ZTBtzSD|PJ}$6Oz!`sT*1^$ z3h1`e7J(tz@a}fjqrk5Y&U^g{AP+BaBIqHXQlruC0^;L|KxQSvPp6*?(7Sj|LcUJwGV9;;%I<0Obkr->n!Q1yHW!tNXMA$=-VS>RJLYRqI!-zIj{@`7? zc+tiP2W2#$a7Pm!n*%cgt0Ph<*NYdm?y&FkxF4*~FUQ>yhGf{cyJ5qp?cs3+BNle@ zi|K$b;C0CoF-fOCE?jl1^?^Dhl;|P1xVN1c3?{W&Jr<9;J&GW(;dp#{eo?J=+Y`zm z&W5Pdj)IY3G#qfde3a03r#b9(m*!)gzR?-HAE^j~z0vF(9lhIova`Gt?~cyLLjtpd z_&eR{@25Wufao@V+^DRJ;5IH}4u6>6oIPjKk(P@V*Fq4 zQnpsdIz>m=x^{Rv9y|@-GOQ9NtNGS?^-t04|3WwE320_z7kJ*VitmY4S;P>!P9?^O z-HyNd!!vnd6P0*GS(%0{$m)HHx0sX}%htFO&uDS-} zZGbT_kOi7ed%tpjV?HWj1e5~8pc{@-BK&N>T)b#GC4PI&&vbvI)49y|fG`H&cZ-j= zlZj;?dpa!l^tFsXopJa3o$f_9?)7|SJLnLh+q6j{Cr)LvORjqv_YXcV{p!utXhZ-B z*W<1*eWmHN*&0t&@F;ed7YS`@NdNz|ux!=9H`sLI2y%Bk|m;KVCB}E2ptco13Q&+Jj z1!KvH4j=#Quk;)SQ0#v~Ie9->oC#%Z~?u$$9w4gFJ5O^)K z7=7|!F%?oIW*Vv?+GHKPCl&w8gOiI&J0zngTY>&4(7f!FD;>swM|w~}_qJk-3yRuB zlOEbyiL5Nyk>UKb~am||OusH-QP!VHU%4GbX?K_{pDQF``fMdW}{}?ZV zYGYd1NguC!YF&I;Z4U-2$MeaM{c8^*p@`!%$6UGHt29l?MzV|khxe6S4%JFD5R5(9 zo?D2@2!U=GXnG-RLjzvsq-O{H{@9HB@YlAznCIGEiQ{>z;twR*m5zG>8U0rmo#EKy z9N*7|ws#g4MVOhZSu#wta5NusEOKb%uerV`|DG7xgHL3~w)?a9-t^ z>Q$uw+e>X^0uIpxqVm`w2vT+-{Nj!C#j9s-+hi`}2-um#z$Xdmh;r2!8{Cyxr#Tlv?3WU!LVoTO)nqanLXC`Ur)3U6O1rk;gV(btr*_gz}50$Ae)F z11B=Fwh();84b8$JOWAy%td)F>G#2D5%zOL(|}RIq}*Adu(r3exfXwSQW_5kqa4EI z4sP%C&z)?vg#G~fGRvV#ERdyi%D2CM;dx!30`82mKETfGEcD->^(_LKLddL(Wy#MBZ`$R5e+{m+F4oS9)uS_Ao9QtrhspGDk=g3~ zGYxe!`sXm>G3F6sgb-_1nSC(iW9vrxa~}Sk?sTU+{qK?OjqC!XV=nK4{Aed(VEFiX zA@}ZdJP}0EP5@cMqed#^`0C@u%$!s$jE_(A{n5nhwka+ONZ=3<22*T!G8EcqBevh18iUfUd@V0$kzS6akC94KXKk)hYPTj0Ea*-b zNCBu1g%_{Ry8V{l=X$c8-C5;}wOFxo2~1sIOg*|cFWceG%TlFWv8kt}>DMjYc7ZSHN(mfSE+ke&X*Vk58zxCB6 zw*t@i8^!ay%YoNd=l^dfDJhUU0}8k|){owweQRNHE*^aQ`Un79bE6>OQOwVO>+#y= zy1P_nPd|T`&$monT}U{;`FMS4-cic8UL2i|MiWI*R#r3HtC9e7KWwxc)%>6tN~VLE zRG?9-j}6RYBjR(;#}#5gr_};#NCLR+z=5ta!jS2bn9mDiHE?`*-niig<;1fb+Q^8M24Gwqi&1Iq_V~ zDe_Qn)DcIst8=ME`SLZV4+;98^$OVaDDAzRFUE6sGU2>^E{VMhJKB&ZZ`-v z(_~|v0b(3xVzblhcDl*9CmB|H-JWIwyW`qB{Zn+CY5$zVR6>Z(rbHnM0yni>^42G> z(|q9HTi3by);lv0nw|JDpc73qr}aDLZ}2>_UjJ^B%^yyMM}O}?E6S!X6al7`Vn&IH z@in!gsiudPvv<&q)fteb50=vpmxj>mnN-sx@P)_qCc()1!GAH<0;hJj%}dvP-UZUH$yoyR%Yr!W>Rp zbvU~6_`ki9L5HWo+}Xu>{&Y2wSxkEOb5)|M4!a{2arqT?R36ta+8kj#7>EF-6al4- zFb2BHawo-KJgu}^Enrlg48#82Xw(x9$+cS3APikk+adr&m{@O3>Q}AgYAEVcBvf%r zkchce14>P^JrpmmbgIEZ#OZT_I;N43U*NfRe>9wOBqPSCUcD-vbw4_&ms%9zvEndf zo1RGq(oyg_g%;9Gm;i=$kA3pu?ellF;i&HvO*f~ezx5yzblN!LlG($JIlmh|`Q+%A zADz|P8ZVkBrHaSHe1Un!6P+0p35;Rmi>e>SOb6Ch79t)ew0r%jWfx=$R77er>Pf`l zvlpH7%ckEeeCsRg^KtmuFDu{w`Eh?Zk@-Qd(^yY^+rx?NTBqIaXbjcbO>Mjw_PH=% zVp3p;#}EtKQ_;-sa7FxFp0e{W^b&Lu7i%C;{m7d)4~oQp6XGy6s0qiNl33C(4 zmtojsJU=kFT(QqHD+-J12|rp&Cy~IvKCDdlHG2I;zwPl_)GNVH_Ya@us+}>&XizM- z1OeUOO>D1jRmz>0M;Fogg)~69sqG(?J0tDha&UJmzyrKGIC=ZN*&S(azIS!m@hGm% ztoy7SyR4pwJVBW13_pK!xW8W?4^5u7OlHfj-A{+)|7Y(l{NqZl?ccSl>Og}SEy%nj$)-y?>PtNR>FTD@woH5YJB$-B1~ z?_9%g{xJBze(~W!BT`1~A6NWPZ>+A1jxxl#3L!=#NHR6-k9Ka(uP=mONeiTf{(1h^ znKFf5v|S1=Dry`%XP%*-%R?};JLAPB*?+=|mlUZN{j}yeM8wR`M>Nlsh_Dbth-fk` ze94@0LKxfEVKjSQSJWy)09PMTP#7-DTvx2mK1Nn>?$=Z)89$LXz4zO45~c@B>< zJ707+wpUi?{6Vw7zFwG_-*&3xqq{oCXRnGgm|dJ5Kdc>BnuEdS*81}H`uwnUyjUvV z*trJirw0eE-gu>?LE;1qIDsRGU^s}6E1haD%U$SPUa5DR&5pM^?;EFyt$^qvWcT@D z`cYJCw9V_Qt4s5<1#GM>oXOYI4(jbF%1o=9BoUU%<@p&u&at$zFiKpb9}jXb$@t{7 zGwN20feX#zjjaXamlx*L)wN>TLli(E63LZT%&1iAPYx=b5wR=VbkOQWsY9UqE!?242m$!YIszxLX#>z)vtBk={qlS&<1DyWskvJ;HQY0kuebFr`8 z#jDK5jH;RyV8deycb1eVyl7l0lRr z?Q6||6a)PDom<7?Y~bPZFTQ$m(8#3(#UAB5Ym18o3?$!JotT|(3rpR;&GUxm&9BV( zs{yYql)b>OpSC*vD2t-|ckde$I4A2%r7$d<>DilGb4xRUuMn8?`7fPYRUhz^<4V1s zCa#n@Q)!OlalTv%O99Q!VC~8rG(WA@_8Y@aZgSD6H|~D<;O6aH;sj2Kh1GuMvOq>F zv-H-Rx9+^Mb==@zetdj*+$X2jm*$pdm9khY1+VU`IVuK}DHr+LDuww49Tvkl29Xki zBjw-=cxirSc@BAuHzC4?7ZjQ#7F%He{_~fLKyLH1bY{+RRWQCmg!+zuw=E|Wl5~aB! z68!y`GDSpal41*@OUbB{{9_Uk=R_oT?9R9J>gO;EJGRzd2weHDeXRtMKWmCsFfq5} z;=5ix)Oz`2L{LOp>+{Sc^ZX3}_BpwGzS_w@O_!7h(dCMEfzaQ2|0oe%PPJ8-IE5?% z=Y*LdB9L#+YKzMdfA=)PlSAPGU&TTP6aP=zjAg?>1SL-f*Ea{b?k&i z){N(uW(INE7$luUa~?RT_2s!&Z=Z!Jt((w;=X^ik2-CR& zS|gB*6*-6k(C^2Cah}PHQUPFc39JGXG}*=Z!fXjo8)yw^=tAes87Zb&)@#L-!Z7I< z_o_fxgy5Ldz>}qgva=Wr;@k-qjMQK zidc;tj+?__;>w(Pp7KMClT64Hu_WRH+iOK3mE;IY3r;5@9!Wnic5Sve>*Gwa{_y=f zsWSp71(Kxbt9BP7$1%KGE34G{H*U;rZLU16jyJB%uU}beb+d!xX5wJUIdTjzL|`FI zOqsoGcsTH-*{!?`c^?v2r6dIJFa&2A4QeMxYy(51--lw0%2)p zq1vtuyRBQh<+ZuO)miV#(p<(;X>`V^80Y(3+}^mdv9Y``S1N?MtkT6I&CTdUB=oc* zVpchuQDVm#$LwXQtT3B8%2VwZ%`<^r-_5 z#L-23tATiDOP(ztWxRfKBg4@lE;B^+7hf2P>|;oXe`dn_#R1W1qLAe9hovk zzSYh{mx@tICZBr7#cofRJMBDkc6df>($GrhdFGyLU-zBA&*-vZcqzoV*wHDIg}y}X z5dPRFGg_aI(`v2H)q9p@=kl9{S!5zYUM`~O;+4>3d4H@x=x=JjsmZF8yf`&<;jxmF zXH4Yqi@{(4oqzr0?}o`&{A)}+T&#R@hF&1_^66xV<1-*E+9>@(;xeoAIdPc?+=~jM z|6+r>K$wY0YwbUe)MkDD+|s|5wL4|Xl&P}4R5eBu z0ft_LFCSMQ91hdm*N-}ji}Poxzwh5(?)O>`YZ1Uf5cqzWrZ}$mzTU4{=aYmU*l?DL zjPrcInr85Ok$Za5xw?z3weaj|XQNao;K`$2y%(p-v+xK3(Zn%kMxN(vS&MSaU=R@q zNuk)=S%H{9X%EjKrIqKg0-W=VG6J2$ zFhZkYEdoy*x;tqe1=oCVMDP2g#i!+A-eh01}y zGGEeyL9YveuQg|xCp1%@ota(i4A2^6jslO8wWZ~-i2iV(lm!yN7Dt>Xl+Un)d#c#r$@uI{QBGz(dLHZeRN|Yv2wn8jic|?X8>h3x%~cT3(sWGHNxuHn*G0#h<*pee;!V zR_F{-k8E|&4ejWaSGO0ILQR^OU|Cy%BVhud6g+4Tnw>a6X-xo;hVlt8KtR3`-$!W3 zkSJy_Aj%R)Goie8bM;5>-)6>#pB;R;U+Frdp+Ybj#R~l%9zHs4_7c_wRsadu3UsB% zol#zJ2rhJ_08Ypn4acbV-MyoFf0*S?6M04{qcRJ1+_t0G(!917Ns2tTMjJ51iUx^; zQh-(hf`Bd9L2DDa+|!19m5XE!R*aBw0?`VjEs^$oPb1WxRtn+>5KNi!kA*IOL^H#+{*b30!TrfLF9YB zF`7t$$mTZ7FB)PdQkEbPxh!WdtEkDxMJ`%b&R@x&3b_3XRLR6Ye`y{q5lu82DG=c$ zhC6*njqIf+Bl+&LA%FL{UV3?I7frfq2$RJM#5pHmVb1%n%kaGBFdyd#L?6}86imFnq^@rkw>OJ6$KG+LJ9Y^btoYzyvY$x+0W*`ZVR_Mg zCPFNe+CYdvNCYH=oJpzTI2pIv!`+R=w{}XU)x}Ofs#RK90*xR_^QV<+dwgYexja|$ zNN{x89dskktntS;@65jThU!$Kv#HC&YL>>FjQStkFh6`-98pd_e^W|F#3YuS57Edy z;eg?b7~%17oTVs)@QkP61~V*w(s{rC@V?;kDa$1|zh)onisLv0#^^ z`1J14{Rj7r_HMkf^WKkc)&`wgEo0Cy8%KiUcKrW*_@oh)m=q9&kj=Rz^1Rxxn_C?T z9UN6Uoom;x%{S_$t+i4z#^GUmlo&=xLL`#_R3boxoj??X6c`W&T?}r1bA53sBq2DEW#dGPQlu3Gk}F}u1|ZQeo)1=O%1Tke z3w*X&>NHGI7iJ5Ez(^|>;B{o&!=2dzddBSH~UaQPq_#}>m# zzWDm&!BHpGWoL{NDHv5QlmWQ)pcs0^kUUrlav?_`t%!^_5iusd<{yc3p-G1V{U@&U&ny zwITwj%k_*eVyETOb@bigRDa8Rnl8Of08f?elqpkX`}f~4#sUT$lVcJl2$4Vm6s6cZ z=q{Fze*W&YJ2&P|``NvFwK%oSPA`|)jrHY?<%My#lBPp4n4O#5y?sT8rMnMK`s1M$ zT>#5KG{0>}hbN~$d~^M^S9fP-XPcd5|Fjz=R*NWP3S$LmSR-s81Te9H8N%{SvA7VR zkdrWh1RC%`Bp@i|gvP0>wR@>szjAd!l}mdMTEisI$q0x*raiYC!Z6Vy+T=ad25Xb<~1VyBTSU{|-s8)BN%}lB2dkRhg&N|L| z{pQZK6|IpaNw3#egt>w)AjzP#;@!=eAH1_OUq-)iHm_ZM z>y;}>R{XNs8T1l^;{5FFOu1Y;?myi-s8;$ZX6DU!ajRG^YbYmR7IGq%n6%c$qcem7 z2Us8nLeB&czuz73)y1v#rH!Rlr5+1pX*t`()Cn*FFX$VxN*`jv$qJ4Nah*b*xKB=wsY8w?mg&}qf~ML8aW9B zUadLkjq>H?V0ocjYmBXPnv&3{TRYq98|!maEL(xOM~(CZJAjG2tWlqvtDe0$uF zWb$EoZdyG%JkQfcpS@Jj+F9q^#h@eP&q)%dml}g20DXHf_~I?1Z*=PULyu&PXS7xX z0!}UZ!~ECFb8`Hud`<_$P$$Js_cP`_C7< zk8=?|H*Sg4MT9Yf9NP=~_Odho9T1T=hO|1*1wX&UUz{hpJSPqCzer#dVak*#Q)T=2 zMTBrJ14P`#XeoqKpca z4h*MSUHnS1gp6p|PriKc^ncuV<7Yp-#j#(lj>0h8y*__?cOwX384OQ^6hvvIwQpHG z_A^U|r!|(%#l@LIVB$CvR_cSDg_}Ezz3A2bgL>|G zYklVS8*9ZGxK<)=ygD2H^sQTOUN5#|oU|fST3Pdo1}VhG@Y=n0rQLY+wDSI&JMX@G zD|6vVEpj?rUMc_Ro!hI+I{97xm~1L}0!x6Aj0A$9+8FI0)Lwb>tS;!az*}s!-(EKKY#npx&G+Z;c;iSg>=)E5)qsb8wD)kCNS&skDi>6@Z>G$%s;l2oUD{xs~j!l{3DV6ZPF@8BGMO{LR&X z2mm2PxCl%B);?*vWDV1QyB8*1ezf8|Xp5hr^u_g|l_FmoEn?3fb`gNShvqb%r#(*~ zHqS4{wIPB(mnl=GOqnX%$*wJiFq9>eh@VJF7{Z2>a%>$`V*Oe>{_;^Jm|t64!;M?l zD*de9?tJyI{rYQbcW%wEFTWEFt>^2xc^c<fECCGBaDC9O6XGm-*72hEdK!h!YF=pc&xeXYai5)bPJ~x(G1VWaA#vrfs z@@>+MAx@fclraDSm{gSNY9s59wHM-`(rNbNoIMfBv-HX1_N}YDwmSPi{&cq8ho@00 zU_7oLH8>_!C>Ke48S6oU$a*4L3lprsMNai{jIBx~aNfebz4}=-?XBIVAOB>#+eP4G zwtQX?06-Lh;7DA-m_jj(bIloN*|^de^afa7oLQU=dgCDxEK{Y~jdHy9+U|pUU+f(W zKm4rr#+&Q^)Bo^(djJpu6V+-_=oPPR?*^qE_Ehya`{l1c>9%5KmpdXq%ylVdq~hl& z_G10%-Tm7;SGRV<+jnlZo2MuB!6$c*yu$p>&ivf+e7Db~qM9uu&T#)hyIO67JRv{M zv%}i>uof-Nmnt3HKWLAv&LM`u5@fb8%#gYwq?mDW;L0(Rb(%ygQHneZ6J;#y8e2?O{UZtrYuZyC!_46QM+`26eQ;e+O< zpB|LV%>!pFh>*)wL^y zpUqtFbrEQ{yrfCv@zebNL1kdQ2*rqzlXy?UlqplDOqK1Qi4tNGfE=j|-XKv2)hH~?#!;@6 zwqRx>6bv)_?%oTPvEuLYko*zz_fmidEQ+?Pm`wxt7krJCkHdP6Ppjfm{l;j(q*NTRNrtd)0pI ztOU&3T7LR;@QYu5_WJFuxpJv!GtT7bsIh;zx4wCGc6B%B+>&Z`$9Er$Pip<#iG~AN zN3PNx-F?zNY|${&Dp=}_^(UX6xuy$i8!O646uXn7`o>E5?#yjP067AbxKK;c(;`8f zTTA77JKEomu5K4M*5)gX9w-#+KRK#BKIq=QweZI4uYUT;-v9fnhjEgwuFV@=?DmGA zKYHM4e)EmnrD8E2*@M&0!-Ljv%r|y+({$YH*2Y#&n&U=X?2m1ZaFpp`8)ZytMv0<&vxhpife0Q%iPE4E%!5C3QW|X?UgX&Dd491BgfjC7fmpm8e zX$2u7E%16V-9PM=zU-%Ye%u(v5*jJiJK2{H>%*b?;=oo?8|e^Mj%uUdeE7IO;+2&d z4Pxd>rF~L4*5T~Z+SNucx1jdOeZAjKR4Hc(!!XM-aXQKL(MfmE9aUPxv7as9ZPK+kql{oo+(_LJNZo};wMMIF@gG|T#<9J(}0k}tk`8s&p)*H*)z zh*Z_;)#GS5_uu^haHL{#q?vqW*VpFe7tOtg>1ks~(CS~#a+xxPzP)>xNC3x9EHgXN z#ydkBqqTFAXSsFu!iWTu5CBEcMg^YHT1_SlMXck@I(9682nk4Cs9zyP&%aB1&j)HD zV0O;&v!{#PV}P{0Y?c4jNL&aY|0r=QB9~TrIr|DPPkJW$Q(0Cb|AxAkXi^h(@k%cI zTV}ukBqE;iwbs5dtiX;~UL1egywytkfe(;9!#O8{sjQtcWy)09{>52#9*gM?GNpZ+dHhg+_42a9Y@@cd6+{O0W5)t zykV?Azu$UvIL`CDITVosi!f0y)*pV=42_J3lyK>^mGpl9wL{(>I6w)QNDs&EqfZaK zpg9<)jz8RLo?UA@KX-!Tm>1 zc5c3P{rcv;$H!@I7-6?J{P^ShSFil^=RbQTwf=*LkAC%sue`t)uNWuX>-5Ctd#7pO zk+|IESurg9_?@?IzOs7%N&MRnJFP+f@mGz+wrY(bD6~$-7I*iXk6XVRJBr46hC-#6 z_P(w^IO>E2t(3`g$#Oj$apJt3G!p=}(1XPP@adlhgVih1uxw@uHAohq`c6vsH}BoXT>^1iS%Q2 z)NLNN21c`k;tP*LG%NurN}YilW*#v;JRMYfkISW_z}MP%<2dc6+#2PX1jYo3%O4yJ zP8;|7gDlFGfSCMIE?@6Ak6XiVCREB9&J}d0>*7>v3P`JNFa3Z1zq`4WBq1V0q;wGG ztp4!XUccY(v?H;0ZEbdOsr+QGdT`t-l}d`;uyMM(weI;y5)cYDwVtNAqIdJ=jnv_b z$4A}R8eIS}K`>>?-w~3D^!p3>kNY}>#j%K3k>?WBc)qU{Dawf*3$q9lky6U{&M?L( zt>izdFY8$6XE^7?$uo%5bI5q-cCaD>^My<6oG{CUH!I)h4f^h1tN-*5F-XYxN-1#wkh`GXV#bJ+5%NGF zas>NL#yt^ORBCmrPn|w9QWgN(BM=-xv?=9k@uZ!MMv-OZSP4L+6csG zL{u@y9QQr+Gl&tM8IQR+7zVy?$OEy1&f({o=^a@T=V{W#bocuDnTPh_AHsVN+rzQ* zyyV)(+}h@LoS@z5ylA_J%kvCmjW{QOiBjG_JZiSzx_xK)(=S&J4n~Axn(?DYmEZpA zlOO*0{h$92yK}26hfgcjMt2+&Bgjmd;c7z^e4d}v+gD4kUR%7rz1kk&^l+~?vN|Xo z){$gc6cZ@WMx~Nh6PFH$fQFDEqsfaB?quR@O3DF-NI@BfVnrm-Vp((|rPCoDMXcC* zqVvyy1a4Lnj>v;!NPswid;l#vIel*Aro3@?W!2oW5_DP|O# zscBYI5wR+C8WvuZa@nPZlF#q}A+QhD!VZyx$5LR@L}n1BCb7h+HePO#i!XXcDX?&C3#VOnt%%_X8LfyY z5ap+k(-Gn^7%E`$g*_>HD)1CTILYYfBq9YOMapQU!l0P5d2-Z^q9jWsAh~^G{hha8 z`|45a!IR@rFEij`u)J~YRR-$yewJ#*sub9(SC%)nHV!J|hyOo&=i%SBm3{eh?t1_w zx72Ya`AvG6w!Qb=KYV-deUnU2o8P1-j*CdVckfw{6lh4K+-1u)zEWU>!UupLwDWE` zpL}t>u-!AbOVrb%ZUy0yGr`kQYXTrBkXq(o|X0yF;53k|XXidBNN{JR`J~r@4 zOOVWJRrRPIw(fPivZ*B@#mb+y8@ahapzbQqn1qBgjqXLpO`PlqD=84FRS}ku63s|U zh7?k`BSH})uVk)AU&5D2p~Drj2NM$$6O*#Nw{dk!tCOidRri6{p7V5F2Az>!5^;u;g8>Tk-d$`q#%G`YpX%A{!qpqs{{4Ub=z2Z(^S|`-zx2z6CMPHwz~|{m`Do{f`$--K#&j(B=XSDi-sO^Ya#&lfD&Os1JMOb1R%gc zHwLW=f}E*n7BItl0TD_NEdjGofHEXS2=Lqt#*RV2bvQ4CnMsIHn2~MBB#3kxAmlyc zOo*;y%52&kZQCt81j+;ySJ(Lq&tAIx;-wd#dFHFHzBlHVUpV*r>lZ)$0`LF+n;&jS zXvGYJkidi`az#J_0~7>?K%l8XGF7K&V5Sj(dq^Y_nAii@YHcMdEK5qD0Ynm1ks`tr zDG^M}mdLJjIni>;O|HRf2QD-YuTt0Pg=g?j{^pBsT)y=A*I%3Qg=a3j_S(hY{s|v` z^2NL?NaUNf`AiKq{UWF*K-=8X2~!d#9)SQF;AkrFPIVD)^(nVp zX9EPF(7OILrIeRg|AlNMnrtb_SOGyGwJ|9rnN#8{HCbh)YymS%0BczVX`_syXGijI zMF70TzKi#EnFxR$i@*DD5CPqG+4Vx(nasopjf#{{9bwq>bmYleU04*P)tfaXgF_$)$SJ^q0B9mWm%8fnMX4237U6)QO3W-Au5@-Mk8Hxl5 zK@CZt~oz@ZU61dIUxn|=U>$S z_Wmco{^W;k%_Gc9WF24<$l7gHDq$A{ z!MjdmCAswhvBgFPNdzg#C9lP!Ld~pca>|J^ld;;N1|xK2@Pj~;Oo_?m5nWwn9tXnZ zs)7#}=@{<(We9M$Kk1z#J{nm|KzAi;DS(nTe=Oal!zpjE6@qve2)ttz4@rWNgpU2q zNsJ(=n>vwzLQ=>h1SBctw8XNJh$RA2&VbcgYJ@`I65I;yI`xQc$X()=xC;Z^?Tv~j z6bejCOibi6HzF&6P@q&VwKSPBf*jgK4q}1^P%wZlExOv38j1h~>S&jIZEsGi2{~5Q z+h{Z0p3&%;ERdp(F9s4IsC9qKQcCq2AVD3?RupazdmteM3#o<^hvLtXI-L9Xi^bJ{ z{o$*xeD%VmXR`6e;^6AR{L61|{PBw)=5>bD2nOAG$d;e^;o#bD{_xG8fAP)p&+Px`#TPD|f7X(vEFj$B&M*5E`SbUFc zT$0cgUy8bB&&(YHA&DRc1nnMwy)*MYVa2xpq|hUZprl=6^cpP(LE`q{I0Wr}|B!YE z>4x7%!dWh+uJz#t2tcq~e()g=fJ5TkL#LFots9P|Kh`vxkVD5ypdg0y2XQj$SA zr*-;|dqB|ry|B1^zL-hZpA5!{8D()sVRmtO`SthBQPoA54 zNX+wsdAGQF?dzube!f6;%gy;37LhIGh350C^Y5$KZxfR*a&P7jdjjB7{+^}f%>YkgXv8GKS)*A2- zu9xQETYP)){YPKUQ%=_p`sUuXtLRD!wqgS&vU2CnK*RtC-%EkyF;C7h+?pjv=jr-O zpqW7gnkJhuH%%#Ju~?Mqo6LXLtfuqdb?y6~meO|>U;NiIa^PO@IXLPVM9{h!#>UT~DnL2-s8d&d$xa z03keRCRsa%*w`|=dt<4|xk(wcwWxIuNyCD~#KgqJq--A)fNu!R84a*vS-5D)?Z z2n?wdUv#DGss{oTq^+@B4j*n0V(Qf0B19r33lH}Q0k|jY6Je0;3RkBESb$tzpn(>Y zEYb2oEvSStk;qJ@WCnnsVH*jO6agW*6wv8KkdV#=*;UkvtVCA8Bmfqm zSp!P|wF02Ph$yAxX8Gt`pqy+vuLmq^xm#FBw|qqq!Mhm()-Ffpqd{nK6}{;emaJ(S zDHaMzA|fcMn|B>{`J6>8}&wKG8R{6gi)##1BB9|L^1;{&?9j;k%_a&!~`F} zbAhChfR2@S3W82Zz9b2SLJ<<5GG)Ps+>m)fk|K(S0m+gjMufX-3CL++Zen6$Qnn8s zg;c8!5Vh8>NwZnIihPp8ySSvqq8uD7x-N=`L;<3X4XBf=si246Zjl7APDRcuYyE)k zUe=O?02&i1fKot`LU7Qe5GA7n5h5ER5gKLQ#yI;_(9z^sF0YVWWap6G>6JE~m7&zn`17b5|&YF!KOZy?8w+`?LMjqz3LjS1FK8E|{=HI|S`$ z0B3h)-K`a_hKYtoT_(k9*38f}sckcvu3x`iN|gv0C{UEArfv3Tb47pY%_CL2APBh@ zR-l5UvSd^O;fhU41Hn=B3dGrBVuIwU$cW)CF6zWGPQqxUfg%V{5FQ{C6B<9ThzJQJ zB)PjH0GW-7YCt?7CMG5(rkE`O1*ocUFQv{GW!02Eku+mJC-7jtxG`UZQ=oXWFA=~k zg(E&}7Xgt3xK8YxcO@e3VVeX=ONnjUHcfUfUAL&UL?|Q;zWO{;X(~WM0CJin(g+dY zxXc(JwEiAqY0NgWwAZ$KVC@Zx1epXX5WuEOv>=TBmxpVD0Fdh`BSMl{N|si$aeA^= zcTiGCfe$wjL71ejtK_2HYi6?~MXf@JM#;jxTNH!s&Gwom3;RsD)EXgSS7Dwo$XbmH zfCXvz{Rnb%{@wO~Q6!)sLBh<=?eDEFE_91|DK$a>2M;Z^O+L4`*XDGizxKM$W3xF* z8L*qXNdU3>-egxMp{HVE;t5C6=JtKZ1*3dOPo{|p0x2iUDI|BV;Q@h5B!D_=F&K{_U8UIz&uN%Jx*ntpLy^2WmvA9f)-i;Qqd)l&ZsBA%L4$=z4}2BRz`Umn0gS zrUB^g5%*GID?yU2lxI}A5vh@mScJ+enC z#k<7=5V5#bSL$&v{nKk6;$yAZrH*!V)15Ef!sB zl(S~D49IImC}2_}SGVXW%=UWi+oEZ->_pr_g0PJZ5n>%LrXs*`0GW5xnNtu%7%}Um zce_NP8wWS~oq$Z@AeLo@WxMOPQCUQ{K*&f_01TO!0@5({p+i~gSdv!X84%!rFfs9P z2bpwM-HC+pjr$atnBZt8t-2%!RBH8*03Lz}ou&5JAV6X4$;1b_`iY^+U5~?l>amsI z6O*#tPQi2?67t-*rH08wN+1Qq@s70;9w`x#EgyDo5_GSTNWmjiYf-cjAt6K^FHzbK z?->q7ryx+F-SGg$$gQMC^x-%%N+|>ZiqKX*H;+e2iP0PfuXwqqHn&Z4Q0Ie+8ix}3 zAfvUzuIgl2%ZCvVAvQ=NAXSfp1I=blPBWXKN07d z3y5~Ja+VTeOH&iDVMam)gfqtkal8anPfF#ZPU!`R!#^Z|@kJ(VIzqYUWQiu}B+V;; zq$tH<1D|n1#pxB$2op~>+K#_*Gy+UAh6FzxLZjZ+gy=6y`2+})%WQ5V}k(6 z&8Hx3y%@LJXbZP|?hChLLWf^~NUo=6RqpCB4=+zl%61%&;!qY+g*~&BXa-3D5u3$C zP$Gm!%El8;qHe0LxZBt-0VwA4147lk)OvIHQk#RLtv>iBfDk1>;8=O?PJ|PL(7}8` ziU5%W1R#aLlTuc6bCn`MfdnK%iZF7iY!7Y>%j;C$(s-i(J$39C=ZR~f9-vehs%;Nv z77JI{xqY-vW5fa~wL}OqC_G}(6%RD2$xYsCvzZmIUR~iKLBQ~zq|M-^Fa%WvV*FYx z5&}1|bP*iCjgv;D9^>BLe$E+S*LBtFV!5v}OPe={lvCTb=gysz`?VX3>LHn+o5Fe6 z4Uqt}OW6Vi07*=oIrv06sYoC}Qh3min*$~2_@NWry(=XrV={mz$yxy<6j5p^wT414 zaI-WYgtVQ+?bJn?V_6Gg?P^{XHGXt0(fhS`iY5*#}ENE!aY*TmXgQV&Yl3NGjY;O0%$a? zxxY2Jmm3EmAwq*aB2T!yiH*6?vFHLsY{Ov{42eLpB>*MJeLQx{TfSfMsC<8XtY-+FkKkE|?6g8~h&%Wx+RDOo#f0o01BPi1 zE}CdLWhvZU+On*ZNS^j6Bn1!~p~@bJq|uCAj|iol$nnJ@GYH2KkRm)>gTB3GUG9WH z;;3qM50d+Pd(S=hTtCwN?YG}tyLK%kl7P<6L;QsoU)l1VRFYkgG<(z0e{z zxoNCznl7W1QmY~qAqgD)4Pso8O$diTt_TNss!yd}4w;z14F2st<_k#ylR$q@)BRD8zVg1c<0L z7F}H#*M*b-PFZ#nr<(kh%o17MU70JN+Fe2#AD)ft*t>8sB;6onQX)FEma5*Z=BYfB4}C5rg%hL=tC! zx88c|)t|ijum07)Tr9o;c&5jY^vE7((xhx3PZUtdm@>1ejNKIg5IY7UAIa&RRx-lf z0p0Tzl9uGPkB)=^62u+D6IL%CxmHq2fJnCOtREXIrHk-V3hoY((7?mVj3Wt1yNq)9 zsC6!}LW)q=bpV$|tts1{%@-(LAwrS{Palt%&*!P8rfJ$XH>B>E-y_V!ZB0y<9X=?_=Hhh$6i&99XbRhO-dpYNs zW&lbl9)STtwYta31-Yi2o1Qjj=6>o5xqF1Wmc-$cdz^+foPM-qcBp1aONk)B2$ix_ zs%wWKa+Az2Db}&vrpbGI%RTJ6GM_KpBSN=MkK_rT#Aa3iAqY<#6B8mJ4$77iy6R?Z zT2u)-a)Eew>8U+lsv-zWn3yc<-9gisI_`{gbNW0 zLISvHjV!HZDZt2$vQ@3P*%@mclO`Yu5e~p55)&&3GTdp_JMF*r>Z`x_#V`NLU;Qh8 z_Gdr2c-|5KvS}ML0@xfzosF=qC6f>eoMmY%XqdZ$<3{k#o11DBmDOz*Rm56^NRzTX z^AM0!YmJDUQ%+WD5Fl};OxOfP&uRh*_ex5+S3y!NmKLKfonix-2qzg4y_l(Wl~3nY z_tHs#x0AI5h^GUMoC^qe1iB8w%A>fk9&!Oi1Vl-I0g&L1Rb;j6I&X8%DK|N#Y)#Gp zYPdGZLJ*{J<~IS5D`A^SD#G2vqbnss>(_|TxH_2t(h}?UYqs(LS}YdVuU~f$k`e&J ztr0TL=Y!IN-}_}nMv6`9w$z?d&DZUCRMKj2ue6RXw7|J@$SCp7OSk_DxF?>$`TJZ zOA)bHGMA+L+xtn@Zg=XU_bTl-D;a5twbbex^A8FLfg+#?c`ezz=9I{s%!~nm zAZf`GG!(j0q}HH^wYnd!x5nY>wbd${SkZeXud3HA=S&hnL`Z=z{CV00T7vb0C=qt@rc^5i!csdAEfY} zh}8~l1T6y+j}H+Vz!NUF0fkop1d|!b^_CLp3II>LxRp}M&p-d%8*jYvm;TaU`uWd) z{?bb?wi&?KTtw>`19H;(!lgip%$tRUP9S9sKvEM;S1cxbyh??XW#xI@7ZB4n#6purNB zcV6NzfI`=m2$xi=k9zj-9=Zo_1=M(K_ru*t0ijTcX2ig4a*}nVMXt1(2B6V5cNzt) z;C=Q*=JC<5ktveyU>n^0cmL@2#wtQ6O6_#u)!V&Q!g;hR{YPC_>rhIoBkQZHI1dRk z_6QJg`0%EPC7aJ7Ne~#JkN`;mNg!aWbr=aLBu~aw5pM1^HVOB7aBz@wHnZ8R?HeM4 z5&-Eo+1=Fcqj2UeQYS5TUwx7X877Dzo_M734QzxGltn;BW=o{GSFhm_9K8w-w^0Lc zedEz7rHXPqe+S7<#1qBi*La$DwJa$P^B@HXTFsG)@CZTT;2p@jY*BOME;06v6~;)% zPHq7-TYCms0*`fC)WUym*C2s1~u{+pB0a&Ly)s(De$ZraA9>ZoV%F&2x()vH(C zT@twUa~+TD?(OYeIDa9f1nyL}Bp?E0-{AUx{?GsU{`>#WYsG-TN!gxVaT|*2-p*ht zfe`e_oGZBFT;bujh>e?VfOc2n76M3j54ogp4J$T)cYj-^c#c>qsf*;1a9ey z&8<>(f|8RPd9LvS0*^^if)CL2Cq~)7Z;t>FC-}XhqfDC|4x&KNbRCfTh z1oG&`KmY{qe9&Ql4?br`dMfRvkH(=SoN;u=uMfh2CYc5mPw4% zsQPvE7#LTR>*%8iq>;!??!N)Ld#SDv1PKC1Zx|#D5TWs&9UY^lZug|E;eyg(8g_HI zTOQPr<0N7YpGa{lDIz6%@x>Sa>WaR@@TE)Vj99~?rgGd)mJBlM2Vm!l2B-KB)86(r=oM$F{3F0du{vH>#zUqzxmg4JG{1? z+_?DQpnUV~H(l2$gk4A)-=O+v@bfRc&^AT`1@4Z~K)^TOUjOpT&wu-yU(FA6_GRsq zjm)8nwMvl^%{EqTIj+Vx>RGB%|QdS==fh1r()wX0j_PiGWZojY?q2W(sbU-=fRr}>O=OlSo%E5e5 z-L=A3EKPq(+kbw$H|&Xt2}yupmR2bbsarYJg{a*YL8pBcO-u-2k6Fr1Bgx&Rhb_L> zL#TFW^dR@z69MQ{qS%54KY|!{^vpREaQAUg`N_tNS_NSL_TTt0c5 z)xxRqwdD2Xe-Q$>eWM(!S#AAjJZtOM3u8`}&6=i3DMzSQ?|MqN=nBZ21>TEpRe76% z`6O#^&BTNRNdhfdN-2Qq5djSeArJYJ$D1X9L}FFy0)i+}x{x8M5slaG6G`{9Qle*N{=-+%vo^-3GcpCwIqHHcfv z7J?BvS=pv!Ia!1R3CC0TrK4F822eZezR2A}t&_5Sd=OSI{bkn9nx-kmrNAR+kaDCn zSg~*bf^gTEVK!Sji$>_ijf1YMr!*cT0xtdqb&%o{eD(cGegdaL{141gtDZ% zYwZ98n6zG%B+TcFoR=-Ga$Zz-=vJrQ#KZ*2WQVTNT~|FE6BCaVK_HSPNJ9qD!$U%{ zv>L9>0&192lt3bG)+3UzvmHMx2Tm7I;FZGlA)-zmeMU2Dy2vPASB{W5j6aX2*KUQ9i#>( z*wf3}WVtR02?FI22#+Ymno{x#Gi#cZ8#SH|g{?x80IdPFbvlqpWHP4J8gj~m{jJy1 zt>VC;Kmu2zg>+f@EIOguHrHgdj{ja;8#xvyj3) zq=&j)vRRTP)LPahPWKouC4t+#taJF0h+#7#K%+x+Po}2HUR^Foc+w$p>C&Z_Uw(Pr zs+Yu=jkw*E#Jb0^A~h0b5SHYO7hkya+;e~Ft+(EO_ucn?``h1q@ZpDlTI$>HeR@6E zNkD*rS{32jYO|EC)~Ypt23u3F4uD28lXO$r3bS@*JP#sV35FFkrUY`NbtRH`JeZX2 z5MB?URu3ulSgNQvTQEZ5v9b;~z;RDV;a$D#23d-`M@X2Mc+^>B zB<-*(vO}VgfF4+R6^H5Yv0!8CoR2zW^ z)lx!EBv7YO1}w>F07@x?w;~Pq!aA85fX5Qh2x4Mne$Y0prChqQ=!ycMQ5-*01gJ`g z3TLUwm~%2CjU-Y^9P+cs0Jfe&f_sUG>Jd&UN#@}a0tD=~A!UH>u~>8$&Yx>@vszJ5 z7%?$1AwpPjSWFnvl^P+O1tunK+hZ2BM#TO55pX>fL;yP)<#2d@7}(?tsID4D1hl)I z4xxuI_@oFCP$F|m*^-2S1U$i6n>nS6y~)AFi)~}EiO>#^JH)L#x7_?z{i}fBcVs_iz90x8Hq7aNL_}f`pgo&6EE5Kl|sOfAIy$u_7?D7hZUw zw-@ep>BS2dQ%)7$mGaq&SRvsM9)1})Dt;t@x8HvInP>WN`b8YgQYI&5`|;vlrgtv6 zDZahqo(%#7gqh{s)*7o-8ANc~Ax6a0h?%Xj=-XQBVzG!2$*pLp0BSf;3b_#SA?aOa zM)0_rJ=z;2!X*v94=x12TXyOeMvx{&tsQu-oz1$^Rj<2Fyorek*{Um>z*35P)nFt3 zkeQeekI2o$L@6Sq2P5PN2&6y|)Ja-0WlPBpWv!%9ByfCb=>AKH&``wi*KZFEfVvUA0@2>0mn0D`+L^7p>5xt+SLbcX+-+ONvU2pO$FTZm2-K*EH z-}vN{kNc?^w-pcx2u5N>FOTQ*xpd6QbBftMhT>KscPFq?j(~KgjG+aT+T$tZM7l&u zBJK)VK#%|!nNlJxLJ`t~J9F1J`aUTl=celxrCx`yM&up49TJH3(xjO+^ZBCI0LdM4 zNW{~KploSbVVm;&`t?!@$-z)vfbH;uBRHJ!n`EMJ2Ou6F1}DdNwFGd>7?VqGCu=1R z>Uhb@_BNN&N#o5;OiXO*3A33MUl$WXM+`bCJPal#gd-40Muq%n7eEU41CQC&9^I}G zM&z7mR=X~=WIU`d2;_szHy*QX6(F$(1hnqRn3^V~WSep+5)z&`dUu1qchtFa`=gr? z%m$eXL_oqu&z6y7;`YMD3+FFfc<-n00g&|Z#~*jy!YoO`0EuHU+aYlk6>lA=ekZjj^dXGwQc!SHbRoLftV0!g3%-rGuZ&U!8rsIQ^ss z6o;-E7+`(2DeF}zZ4BO~!wA-sj5pfS;^Nfgr27!{GN+RbjT|7`8yz~J*BCSu%N4eD zoAlYC!)5>g5wQk?sOe9aOMsR4!A;&8V=O|T!)QM!dC7a{gT?0GvXmkkvY{r%9%L)R zX;KrjofsTjg`~vBtUdx_6hKlIckywr+LI^0amC}D^Y`C>|Ni^$FDxwV-~Yg_T|0|H zTlPzll*AE&sEVWAo|&26zkk1vA?rE`>8;CH8M-ZC1O)^{z?F84X1uYm@a=C-nMiGR zGhS~)G=p~E%@6;d>nd)acz z*+>GyZeZ8p{sTQ|woEfLKT~iQ=JFxDSzp=Q5ivkh&J>cjHpaQ@q_*#;Vm&AjG-RRV z%q{9nI5#)%%r*oW;Of=uzy9^FZ``=?ymot_|o_2Zi)+I+5ktbCp#lpn%cI?c&!bzb)%T!BbB?H9tSUW5?X%PaL7ZjDBj& zY`s|^$|_*HkGWMw9hA0or19zu0PcI@!5U*^=9C3JJ`+^sPJw}AbAN({zjdJsh9&YF zEG{kHxN*IzDsA=-#~|V1R(8c6+A4yvvGIm2XF%H4fh0sc@>#?pH>F2?ohVAXtP9{i z7qqi6FiAClVsIIA`|A65=)h*xZ_AFI6sDAx(;K~Y=&)Istze}P!`((B;vYy3Y<_MX z6neW$GPMRVkxV8tBir=Qeza>#0l-*M?AfztetxH7&}(^Y_WO6 z`ugjWue|cgKmOxC`j010obZ}YGlNKWq;&SonK$2j^Pm3lAAj+SUwnT2)=5@2#plm15TLabM%1NOS>~l_v1IrBxY1vD) zv~=e5>5q;bJO0_{-(9{UzLH9l6cLCVHEGTlx~ON&5goMs0yL$@BbT``ZQJ)qew+auZTwx6>E z>GRuLe;`5Nh%xxGl*)Eb9Uy2APK?3HVSA{}OxP@&9wbwpY|LTg&Rsi$YL>jsV;HFii%^On zh#19;X4;QQ4T@p=##{o@M5?kped=`k^3zX0=jWgO`Og)2N95G0-_&&t!3ISK zB#qQ|&!Z~k)iGNDElC(3XN)m~(6C&H4C^FA?hgovqqT7>im_NkRMnj%z0`HeW)So> z4iS4OM5I;Y6bLq|*aU~kEkyhm$H&J2SzKI5DYcw-z--~T&CGB`pVr2)oUN)_`rUy7 z5i<=sK0Y2oD66H`S5`JX0BzP%*Qu;3PloSFI%l(#r9+3Vh6JKK?YB+vhM`|;Y>@$s8QwkeCN@BZ1J z{h4#rH{N)|(~_@Wzeb94-9P%HKRSB!IY1W|7f*iu^{$<}oY@{ac<9X8Gw;6p-dk_I z`Qe8je*XFA{?0%C<3E1(*=P6f->0}a!)+on)6@I*?JtU=s%i|PBZO=xwEWVKp(l0s zj=ZV>WnE>`xMS_{zzxLe1>(p~GGy?HlgfN@e>N>y{?2krd>nMBi)OLT;ov)MM*kLu6? zX!~G|F*vd@*LIyxx>%T9(&+{eKD*aR!&b(?AyDyD8@tXKj{lg~MK%y#6Af3&sV&N&5j04g*J?;)e10Bc?QDJ1&Y^mVGnY^;dUdyrh# zy+)0A}ew%TWF_M zjg$egb?`Rm70G@MHbPJc>Qv`!gTL->TmG=W=Gq7C{UKmn#cVb9Xe|^=RavFl)){C( zHUi{c0bCtxmVv}!oo%tK{0XOZ&5CBGA;|iGj1)R_Sm~*N)vvECQAE3j#DLbrwyerJ zJ&q~ ze*Jn%*~yhtqf@7+KKt~ux8Hudswywpy>|8LrHdCYT)c4jp@&o^9hIOgU9`EvkbG#QFKz=bk@Wr`nB?{0Kt^*CSS&s!&8&R@J$4=Pz8i2(<5^ z!y{SyA2>kTrHhx!vIInEB5dh&O0|QeK%pTuKbv(i+YtkN5u$M?OBSO73dzEIIlNe@ z56j^O5umrusm_+hiix5qmZ~IV8{T9OL*49;xc7?n1JV5rHbwq*yKdRBsyf#VLH!lK z^vMiBHH*u?JLd#9&o0C7OY;j3&?R&;kC z*N%dQ=U;e1RXxr4wb$M_efsQ<>1mgSN-149f8Ld)d?MR3liic%=+UF@So_|)@18z= z+PUsyk3F_`?@lv;3UG6#=0VR<-m~|V8`vb<&QB5}O?pi*FczM8=%K?D5ykQ2$33Xa z)meLJS+StqMmA}KRSq`r!PlX~56#Wbo%sCtx8Izs%Tg8V*h7+NK#EAmY)26QX?0x} zjo%iaf-JnRLjp8wng|p?PPLHQa7(|1oZMDrSykoW zvi2UznQcnJ>26(DW=zS*N<_`tTMT#d%m5^)`rlU+v2^;L+9KNy^cI5pfl-AM{?$hY z(kHms zt*Ua>P2NGbfC0IKt@n>HgrKVT$14R8+?;lSFvc)GR`jZ1nx%D2;{TlTlj@_VdTI--ln~f|vTk?N*s-Z@2t=UgeL;xwLzV%i&`~`%C zAD5sx)lEUatsg7h#2|Xjghn>m?4kD;#n9e;9Xf0~K(t}Ds%Ra&22Z389X1_!Z=W;( z`y0S!FNyS(0;-@xwgN1h2%Ly|Zt(cHu8lssneCO!S4i5ap7`0}KlzhCalT~M ztME)I2}M<_s;cYS6OUb7>d{9I@7_Iks1Y>sVonc`G38j-B}st_aVL39 zBRxZij_`1~ zz&{qn#P|fDWwn%2B4woE3CVtM8PSC*4o0j0X?Pt!O$W;^O&^AAE=&RMwZ z{gaXjZ*WEkZOW zvt0H4qD7z-I?A#Z@y4Ni32;cu@VC;ErUMX#7DeHN#~M|)oHL2eZb2Y`w0ZI=r$#K= zWW$7XLGk6&*gsL z!2`SZ?Dga0FZ}DJ>Cw?C}@}!D{px=J`jhO)$ z${%INY)2XxI3s7vu|Z1JJr%ztLqJhPk7UInDpVkKRi(OW3}~5Bf&hx8r79L|>kAZF z2qCk92)sckz^_@4S8O`n9?F`MsXdynmllXphqtMJR&onAx#w z_pYx`o=hoqJ>mdYuU>US;LBI8{QTMH=I7^&qVUKq8_SXtCr&IaEs<29&8b{p{S5H8 zef`x}UdjC2(dTx~%}q~FJ#pknTND#}XQ~*^pBQn4`;OU?1RI6qtlFGp1@k=;j|>4Z z#_?Y2anTs+vUXJduVhYUpeRj5!OV*F(&ECzR4@RKmNp_uLRO;uLq(o(tI`?7%~6sd|vr#e)rc4nKi0RoJ+f+w(CSXeY5jbvAF zyA9CkBy;eSIzq_FmrNr!@9?2RmxYXbbzyRJZvku}Ekpn&q{`SX@{TVY?gWtu8U;;lZ+{Hz&)oQpJXzRpLNO$)oE66d^wI^wTq*BR)O- z#E~a7z+=KOMp6I*psWHqW;=3#HzRYlqCg0WJC$tZa1X&|Y0z#Feff|WL&Ikg6lAK( zy0w2uDbon9l(Paz`deZuoZ0$iZ6=FLmC%CrN$L8uLT^ocpV*k1pjp~&-nnsXA5FGx z^nh($*WR7woL95oZR!d`i1-ACH#2K0oNuWF0{8VW5o=t3u#T94HD+rX)F*Uuu7{>( zF_I^@Lk9}hwJ4_$A|#}JkpPOKrw(prRb8c=x$@#?cLQ|Tl)zot8P~IiSl$3A9ElqO z0&FDv9w ze)H|Ot<{KGPATu&z5Dn7;18Vodf4`Z_uv2bfB$!Xg9jU3OWAW`&z(Pa@xld9S0?EL zS|`9E1OU$8QdL0&VrwEXq+shYT1o?e6bjbbO}3glXV0CbK!E3;dwzC)-c2U$9{bbB2OJTvTNtO)7!@%e|%wKp($_fhZ!(ztc5Nc zc|-wnIpJ0nz?}>*=xqff9HO>Cp-l@Rw5i7ev)n7kPswsdHeC*4F-Q+4EADMr=F(CP zgd+AbcvEJolL#RcQf_aeh3nm18xv%~Puqbu&e1(Z0aaYTexs=)r8u@bxuv%dLNQhV zsGF${@>VtLuz5@@A(YiHvX#HCQyOm+l5Q4SA%hLbXk#$IFw3p1u@laq>TPObeAlksv$He1 zcJA^N<$e42@7%r1y#dP_+i&c_SZl9=p%Qs zt?GJeBG6{*I(hWM%V_`jkN^1DXOC7@dFISnj{uv+id zq_r#lG~(tHLjgc;VQGA42fl^DoTr+7&|#A@Y_U9i;S7`pG?YN(Djj^ixl| zvX@70eP)3a5#wavK`PZhZp0^(5B7=yPE)LUzZol>fkqgAj9 zGBm&6KvzXhNvZ23vV_T=y)B7FSxPwxW#mv4#g3U-4eI?}UAIm@kPdy@r_cyTwUX2; zCn+c+4Kn9k*R|*1d+@5c`fEUCE2BdPs-U^8rAf%oDU5rYCLTk98sRWh*X$ZdzNJY7cPhCs_~AvP$M$4HNZCnu*49Xj~fV~=~u>#m(U z=jY~~)=p1Nj1{3MR!lqW+Bx&Pzx%r`ANBr+E_?ibUDs7rUcP+kXOBL5^ytwC_U{&v z7hihOt2#eC_TiN)m%liE!h^t%Jo3oz{O<2MTUHgv)KmeHiHZ4rd*`zqESJ(!SuQNx zxOmZPdas}S_M2lL9y@jFl_V zPe1k4j@cb6r!6{As2Bhh9ZPyX17WYNjOwR*fb2ZkwOpZ|2*y1HyWKP zZ#kv9s#0IaOhwRd8v@Nj5>T3W7XWBevK2+xIA7->uYuf`dHa=0O?6XHr6dEEO z=uK7p&^XMjt}_LIZneIJ%qdl6SxirR*tSs)+p_4;VNhWkV{Gp0dSD(nJQLobLkBYp zL4_?ufXzEMXU7VBK^KZTN=IPUimdd5v z6t<{xIZ&o5r0VJlF91W;hdtKd(+x!tRpAamF1c)$a!%sFmJmJZm@Pz%mg>|y%|U@7 zgR)z7#x(&&mXnD?h>oIxLMc|)t#wOg4`L=FVNeZwtC8UW56{*reK+K`Fw4f?YV-G@ zA?W19gmbs{PiT#KMIh3FfW)9qYXQ_L=WIX_W(v0$gEv~;WZ^fr$yi-y*@EcMffPa{ zG>?Q<8{90D0t`}-DGfSw=#X;`K?NgmSD+1VZ3$&$w0=KcR67=h6z!h^uunrN=)YJ* zdFGjC1Sn$RCAHVDUyHH$_>+&_;AduLS_EEqd-m)ZmnJHTv1fn&^FRC3KlS&|%uGV1 z#bB?iP(*R;*f9aXvdQ)WT9C|UmJLd?OmY=PO(|WvbjcI8od8TtP0!BG`f+Y6r9D*O zx?g4lHcBM`AkvUqkBW;Bz!OhA;d$t!38313fox_SvmJ5S4_Zo<*jN#RH_J(cfRZu* zWqk~?(p3=}ZzyM*zE)K;mOVyW<=ocOQlOB~ve{v=gj*AT!r+_aY`$laUMQgDLRAw8 z7Gl%R3x*I{T1(@?(v2o|LFuD|03mrOO zP}~H>9ttxy)9a%*5bn?c_ktxNR8lQZg&Bv&p!L? z(MNxF_|TpLgHpZT8n$92XC^^qVqBkk`f1N=y>#i~>C-vJ!fCBn%!U|)20(mcc6QD! z9-PxY{q!@lvy&o%TPXIU$j2WacL&#N*RGP=Nr43tWAyn2uXENIvZW?Bv+@!CI*Ixe z_gHebOm|&+;)x@B_wH>chP-syGmYH}$WIM`ACQD4=~E9*Ze2&};K4&4vDc&7++iq+jubDiedLixo!2_hlJt_jc$>fCVw#M9pc}qFB(M4z=2TYUzL>MA^ zVO}T1L46TH<3ZX+Xf5SLqX;Mp!3L#OSi=#xzG63ST7|acHl<`V5p&My4S)#hX85EC z?Kh=RxMA=86Ut4ozh?{!vC)FU*^AR}U+0v{s_c9G=x<9M@zaxA3fCI6@=iVv8@nfQ z?c*=T;MK<=gt|_t&a}Z(g+Jz^x&G&IQyL{b~_paQrvZ|y3N23}h(IbA7lM|jXd-lwkV;_Ftf>d5-J2O50qqBjRuPpq}|NPI#AAjt-!eCNmdjQ^= z1fOC|1QLQc%%W=OKTHJVKtx$Z1ajlrwJ%N_pP8Qa*tkz9OknKs#~%wpeLVTa7hgE< z?N6q1W%(@o#xO!*bIPKsIIx_58i20Ob`${Ns(y?#k|AQuQ50j1cx6;b5ol(VCVwpX z-C9;D83#QWplzJi$w)D#ssgc`a;kGm)~H_!mq{grExE<5la94KtDmpKm7{+)O3Mpm zH8C+gJ~19bEUUUK`y(h2?04VmD)^Bw7=i+X-%xjO!e|JIznyE%*+>8@=7UOBDzj#Keqt zb5@EYEhY<~uG3dvef59;_rKFqQ_ny5%tRr^4H8*Ja!MZUOLetx@7{w49-N=s5mZEG zWeBdZ{K&%(oj-SWX=#x(`7tak+FNhE{qDQ(dS3Gi8Jg_^I9A=RW;aEDz?Ery5LH-C z=g*xx_Q40V{Oq&O&d=}k*xX}3dsIar{PN2$o!n-#LEbqwt_&vy5EIGSTFvi{*^Z*n zIY|?h7>h}tU+S?hZF?#}ph|%d7BwnDC>P7R&RGOS434Irc1%@_JvNea%E>HQ&So}D zWid#~{7Heq<)3?MOf|VQISHqcO*S;jvLw-x>y};LQ%rjBr}kryJ@(gs{nsAjJ#^?$P#J*zTHNu8;<@LZ+qZAOFD@Uo z>a)n3kwl0rmGb_3@4x!$D{hu+mZifQPy{4pC4joBT(7I)qHM_Ryyr+h{K&)EG60{g z@Zwu*$Str=fNk(%hdAmUw(VNgsbFl_R}rI%na~sp5dfr7BnHKJB$!2{B7(YRQ!EMq zBnoI$h84#pH4+Jzs)ev&dWh(*}Hr9?hrx@E2IAuhc#<= z$KAJYpTFft&*X{YCp-_?t3y*t0Hnh$C-eZQ>iXQ-vk*9r9(~Tqt@}ql{P4pfLh|I- zC$G93ml=SyH4*4Klbj5CS4FSuwF;3VP`+RYMF=^_DE< z-jZ{&Y{DKOl$EuadiTpN&@V!v8Ot)IoQt}4A)?R=M34JVt=&*ciKH0?+Qy_sz?EKE zx65xa_+2pP92^SOp07_)#D=OR|8$=Ptn12503|tmmv&I7^SX)g3DO#0V(ZtU{!nQx zn$?!D|6$6RWG`%6)!B@sbm%bZAfb0tP^k_hIfmMB^1b6VEcaR4O68!#ZH z?SExeojQFgTXsDE{PR0^?wp>UdGwJ-V=O`lpMUtVuvKEV5QZSV z;OkiM4Z!;OW9XMw5BLRv=Ke$=nat|CTq>)avr$MDhT#Nw2)TNik8*dR%emEHPG(lM zs`xQ{f!Q)C}I&~bpBt}m5?I%LETksjqV4_K>%9L-il(ZcMz&lI&>Iq&;cN5 zh#?Bul&OrUcSVOTY)e%@^SROUej1AToFBxs2h}L%7m7iXJ z{q?!o*_U5>abi+MgzHf-;}5Yj-ojWsCAoa{`m3+J`r)w;JzNY`K)CJzXnTN78&ul@ zP-t^h#2QEgtPHj~RJE?_vuDpXB3{MMpLu3(etv3ds)^h(=Y0J5ajyyNYwmDZ&JDm1 zCqZ}2c0>RP+EmP{{(z1X266j7f2+I}zARHV5%jZgZHUNX4Xu^COmVS-+zP}jndF>B zmSSvZQ{yCC6h%teo5LKWh{;XUR0r4hZ^8QA7StESMkTOcb*z?iU8SY6=ju51z%g&}jM9}0KRAXb~u79?$xR9AeFoI))4jl&R7F5-u zaWzjlRaG)dhYlkKfIZm#n2K(*z?v*M(|d2yi}gh%(F_=snWVAh1`uq;hjdj}o*4YA z*M2oK^_Yjy#*ad^qWx}P+tt=rJTYp?z4&9~mXeCd*g00;%!jaFgf5oCRJssr@nJ@bJAD6#z@QFcYd~5hAg)=!hjfXJrF_#g2Hl6bs~BYQC+E$zqUWTC$6pN zuhIYF^j4O2N_E{7yE^TR(Kp7Lu&w3X_a#b+!jHgtBV1wKjaJHrrbcgFJz%uKWm0jQ5fY7MJpBd*==v zMlC5wRYQmZG>I~q0*8zcx(WPljs~q15I1tl0e|`t42MW{&N;@Q4YJu?`5XCT5d(lX zzm#YQ5Fsf6yQ(_yx-bR#!`M>94(l!|~WdOwhatmxN7xLvtRR%MUMjR&*TmYA+o9pm=V zMxa?$RX+LR66IB0r<{bsPO-1UhyW`~1%m2$F&0CpQ>s%Xu8od7n)ah!g`~7F6Gu*R{P>AiUU}v8>C-vq zZEOz)a7eVq+i-uYvSV&?$Lu6@^(DHrWIN_&tN+j5dGIHWElt0wk0fBy-Mlw$xH)`- z_uIeuxjE+?H|H=Qo$h)-0$DcTWoJCI#^}GGk&Tw0Q8=hhcURYo@Bkoacnl5^CTyDw0RH;d zfBvuk^}qh`hu<&ebC71%7ulW!!j^#wwSFLgzyLzp+Q~#Jj14hJs@%@=(`HQy7%c5> z6U>Vei6>aYnXw0( z|2B{m)PkHBQn)*yPSdV~y#axnnqBUxsU)DyDo;5f*@&+ExoLbpUw8&!mwqAg#@%ER z8`PdlbyF8*QBBGwHy#faexAUS*W_51592W4{oCQWkv=$;=N>?%~Qi@j8J(fvF zqQyOk++|c+N|7@M)t-;K&bALa_4LYNEphS@n9wCt?arnycJ+Gg*9eNySEpc10VW@Ar})pu&Ue0jdz$cYy#3CH|L_0(-=%BRvP@J`&Yn^- z5+wF;gY^~BBMg*n!Y5D^#G|>}CdlKndL5N5*^N_63+$9Qom9gks*9Q=B#H8s5!u|m z0eB=f?}P|QTcSyVAPjS-NxIPfGyEWf_^qVv@(L(Gm^9SE#K>Me8LMi#a`W=YjVnZM zmmR5Jo(Qe2Su?wJyDal!Q9}V@WaMZd+76tB8S6#kIYYt`%G!|;l2V{0XD_NMr4->g z6VPLw9u^cHO;punI+=!hle0Pi@M>e!oop2$+LF{+SV{7>@=8)jAZQfN?i9s#d6NS6 z5}?hh=JWY7Z8+yq>tz5mmn{`C9b|G^t?yb%$=CYo~c zjb|ie_YXe!;I-G@FA7U3kphH0eE88vZ@%${heJ4o#bP1ANDvgbrDj2*tyNrFDA#o@ zX;ikyM?^Hasq4D!{lhb1J2VOw1PKv=?Abj>xRj2{z?KjOP!PDYF*G4cxBzV{!Zt+x zf`{ZtU~FJpcIC}%?WqMC$^B;9r$R_MXDQnIi6WxG$jDL2+M-C7Qs+4z60U~57B#0} zI5+axLMPmj>qtscD57`VfcQ9zc?d|5)YP?2CPi6-ou5=ly22!6ZA>&FFht#x7L?D69f-D|eqWHe>h((d@@KmYmCQi~ikE7?Q6SS)k7aWIp~WHy^!KB~}?z`Cwq zef5>)ks6il@gXogI%Qi`rJ3!xcnAq01%iZ`5!?}88?Ik#2xzc8kGR`8Q|`V#bPp_rL#bi4^Tafc9jaQY+%{bB!&RD_ct`q==9Z;iIxWDkK2{1vX8- z-^S>xsxq_f!yRDu2vGuQq}_yui#j};nm7UU=qJkw5h~K8EGH0}y2;rg19?JSCB;2| zKLvzP*L5n^xMz%vT`_%ZqH{|gqaa+A9_HK@2!1z_-?UQXD$}=rMNAN(#V`M$;R{f zuWSt+rY_oiLc=XNP+?{S4TKrXq97TH@W}2QC<754Cm?P)Y<+1ov*d#vvV{PV>ILnV zwxycv9umx?L5CrM1jaB)C>C`c-pSe^d09-IAb9wxMzEi^+B;vCT|Cm5)u?PYvesqc ziu>N=x~f%8s)aJn=kIhnW!schR4XQn zMHAtqfjX1KB&0F#edLjjj@n(})c;bZe)Ay1tSF0QWWA_!b_p;7BzBs^G}36vic}=)s%k?a2@1)>3$rq%h>-MPU2Xe3)pjXD6i5P)HWg8lI-Lm;I_C`n zMPMmzrAU^lvXE4|MMU--k_02ct!6ZmAf9Vdi$sIO77!3pGAjxrg>cX7SgizdquZeK zrX$0QY2GN05%&F^z;>PNYmA}YuF!-ozflPRT%m63Ed#H9c{&K^kh(o=gn`&eA2H_x zB!BSoE0+=mqXkE0dt?X(NJ2PVk#oIhyl5+>&=Okd+AbG`WL2`77S&`*EAa9CrfEW= zpfMTjXhnwlmF;6F5a7WP90A-bZbcv^RFxIUvPWGv?gFfR9D+dY9u2im++J(6a{^xw zkaI4}vaNBVL?EC+GxJAnk~lI#0unD(OdFX%Dp6%(GAg`jlO#zB9Ei+a_LopEXou2c z8-hEp9U(+X1ZV)EetT*#AQ6JLT^D!)6BJuozJ2o3MHRJcc;y3&ZGn8apa}T@9Uz1b zohe&@ARz;#%huYh-4LMl&M;h>(XPhxCQNm87&9!`! zq^&^o@N8a>D|Qbd`lR9rX^>{JfA&o$Dr>nlzU-B@+}`=pm2@Om|5oIvY>)J!?UwR@ zOBUW_0S#KDkOF|#+vLDv0v0LFW>ZpiU57sL*pG~ehE?(CWFoAhI-R>Cx!n$XL`_py z<*L@UDl4GwT(+)j0D5GE)<$g2*fd?FM`m_4+DsrpXn5~JngB-@M?q3`mOh zPxRzziP1nWrf%)ubq@MdPbFiqM{< zkP&+JCg-wR-9$CH+4#5-*M^Z1@@%@rl64ifO>TPE4T8GTl-jLC5epC`Bp`?YNEs;M z<|ByU3J4o`Sm1{McP773N1@&yK$Ki4$1~8tl2(|N?W9a6WovgSLI|jrg8~3WCnhZ{ znFSGj^!ah4K?K)5JR(^u&5f%kp>x5(4iOqC+YZ8#KoNp+lLbi-HGVZ;y6Z&?HCy&7$j3Fs;V$r-6Dy!t15FB2c-y`Cq<(X zFbN8y3?K*@V1R>u3WYBu`?T?plQc^Lwhcwu|0oSL>{+7eKrY|dJ6MSVAR{JaRhH`j zC!i*K8>e+i!-CT80a8E!u2x2-)TaIhgvSkssr+X?RuN)k0x2=IBA3Jp(TpTFp9w=V zsQ{bkqqHcO<<=Zln~)QLfypc(R|!Rtpm0ZaRBfZXa%-hevS8DV~Iy!C`t~#z|Pt!gp|Y0Xl<}p?je9% z$4MgjNasE^{F&6IWH*|K6c#iycPCkx0T_Oio&YltgZMu-@R(Z>jEszo%2xYFF9F)| z8=+|c)b5;AOL02^v=X--TXtm}JL#V;0tq9lc7i-Nz6$SqJj6Yv2nfW@Mrd_H6A@Ka zEmszG4HBO*BO?R<6M|4H5=z4Arg4vfOgh{kmlXL4J${7lA+^W9>*Y|mCW#E2A8 zrerdUkfc3#OT^VeXwT;p`(Ol(4n)WxmnZ~8?whiZ2;qbvhjT;3x|UOMsuD*$Ve6JOUJiEsSP{b_yW1<+OCqIlD{IiF-*vuvZ-Qz4nZ_GU=DeV6|+gwr+(a#-`6!z@3T|FCllV$B@4u|ALx%)Uq&Tgfz zg}#15alfOthE*t!Nzsw$kcLNU>m}&{WbL-T9Vt%Oos5DY0pa2E`F%pwWICBl0k&z$ zP9~5FjnUc@gwT4aNS0>PX-a9{`5{B#;lX_34Dm*p;cHJvUP?ksgckL}QYy=mW|A;6 z^4ZZ}ekM>dsVIwSE3xXP_6QLm1+cG=mA39A2pB+!SW5^8C{P5n*}@V)cMr(qL^DnC z;Au9f9u_k7xOGf zN?hORgVu*_O+1Pj=f2k{Ni&vJnNli?LXrUY>{7G{l0p$XN0x3uL*oqzGwW)klQE@I z5wxl2ynL*qxv>f883wUj4o7)D#!@2=0+ibb;y?g$L))0vmb4l^E7mezf}PMwM&3|aOP(n- z7};m#lb4D#nVEZ#BKhfMupp^D8{I=tuW7Go3It)KyR)U+eMUrR(=!ECRkiDKkE<#< zWFNbbQc8WYe@O0Lq%=6g2ZPc>Yg*|_1mo*UdRPbk}@eFi~+XqSQrNn!*Agcrt1q0il#eC4?(Ui`vS z-}>U67oRNeO?b<)P@Eeb3{T$1^OM;#UzmO2L;UTPC;#@!oBw(1gSS6gEIyt?GE0&O zBP|VMx7T+L0TLqIQmU%5D2lG*YOFYmv=WF=D2AOyhYSmu>D`txGZHyxGwVW{MwCS{ zDT?rD!8dg+F`VIN39klr08&OB3v!qSu7)njjY9a4M|Zr zL)`3ykaC3FX*9F&*c+z1%IJc&zb3Y2uy)8`9?%^F+)m89dC^(|WVdj417H>r?Y{=_ za_0*(o6Tkb7mK-j;851eNg;;=ouCG{CDHXk@`$e3AtbgRKO_L%k0f)>L!~J3%1lSy zg?4Ca5lO(vh+s>v9|b;`1|>nV`*e!Hl_0Gb=R?XM$C&N0!1YoVt+<`f=Kv>@*;3t- zT>jWJE{vQsDW$44j%wYr^CpJRhgO3;g_>JQP1Cr0wVE`Vjmmc9Glx)LWSded%R-9o zECqP*_3K~&AZ+t13Q15&s8R^yTNDeH!;l+$IXZ!0o$m zGgKv>C2mdd;_chtd8Yc?n;-r8wfF!2>f7&roaqqTsU{Q8 z8AMYz9^MHx0gwo&dmIpN6JS{3q4xddk=?`02^J|;Wo64{O5WKkh0AjYypRVhkhoqD z){I1+iCER-8;%80{n7ZyLwQsbXZ=5*~HaL}-}G_Rs+h-t?N=4!P%OXHPLQ zavi2K1p;k+Z%L|i4)@UEm!WtT`whT`DU>ERI2pH6^W@ZC_`~2m60C7zh$^7r0G4HN)qt@{3PC@xt>@{Q3E3{`AT_ue|o=J0E=1cmv3-aHSxz z$&D7SBe!ndwqztd=cZ}0XGpqG-^xQ}ZND)!R<2vkVGQc-QC^T*toE;aQEKbrNzans;b`J z-iF4;_4W0KBO|l3vpz`M4URCZ9UL5d=k0e!Mn>-6yC=wYS98#vB8n}kRGM3NcX!p* z)uT{=v%Ar;(Ye`KlEROMsQzxJZ~uXfUOlbLBp38lJdvoWsiBkzK)ANHwlF{c=+Ps< zXDX36efsp!(4PAb?*H`TPt(&=^cY2tcamxQb60!a9U2XWeF2ZET#}-ijvqh4oc{Ie z*Kgjup^a&5Xyk(BWhNGuDJ#bc8EbBCUR_;%^k_Mo%~E%y^d#6KVn}>&VDO`lKFa2@ zzxjW^xw3k>$U@7!!xPaa8I~!(;#GC^_4RdiM7Ydm*Jo$vR##VK+Jf{-ui6cp7mI7> zIj^Yp%i72+h}q8E8XNLq-8cq+QjvP^`N(kFMB!xEe+ib>6W_c#wt}6aT41s(nMT#6im^k-{Nf+IcC4mQO~LH94>E7 z#`|mIV@;K#vnvxbvrCz5Ca<^LNZ#@CnoIM6R;`lYC$E0cudEFlYjH|M9w2FLq%;@E ziQpQFwKhmmp$^#|$LQHr@5G0;0z>+6c22&(!sgq~VzBM^H@%!KXZg_Z@ zYzH}!r_jm@`$3$8gF_%;PA0)jCc|l>;pt(fNul2{h$Nx`;| zD3OXtcrz^DfO1im_DHbscZVjJYXsG0M8ITSyZ*yTVZs*NI1+s#>SLI@#Zv9QEyQ=Z0rj z7jw>9!O4+*qm433=f>l4s*Ja6@4kH> ze(+&WcNe_ixpU{Hrl!Q}xPoE;)Icl0-SXeMim=B|S{I&3B0MmN)N@_f+7ri5eDcXB%`J7aGfUU5Uc>QfWo0F32j;d&XC_n0 zL_9$%=s&Kmtdgd@th6rjTbWKX24D_lHIrFiTNA_J7jB3wIEzwhX=(Z5i!Y8GJ(^6Q z66~z4ZB$oRb#*om4Gl582i@udh|Xc8?<%)wJQeanT+1H1ElJA0pw|5qBG5Mgjh@vI z@o5i48+PIBT@XIyulQ-bnUgkU-qh4oU!RJ_YpxZ_R{~9I z>z!S8O4E{FT}lQiSf};Mm4Z{R3)z;@jcdQo&gPcbv=k~LF`Qm-)Ph^Fu@q6|IJf_# zm{!RY6g^uc6k=|JpVkIXKUZ~^MJ5I5pttz6MftdeLv-R!S<3RtVJ(Y@SlC)*%NN=d z1XHRf#Gl@7Z{#^9WRzAuG}tsI7mKONs`OApdVdSb!?pEkC+RtHo2tdFuH1j;Cj`af zTz11%E?Ppn;v;RBkn5x?oH*KR6;4|^*3;B6vamU}lzTWcKRLIY%`vp{G3|&9 zwWfE14E4(>pk0um&1YcXE#p736S9!ob%dwY3~RS!0ICFj8aaDA6~gB8f^xMRQlgeb z5gJIuwgz9DH_gMadELCnnx3#x( z+QWwrFJ8DXJTeTvYHn&ga`~xd*HzS zmX=mp5zq@bbo=(Lt<8K@b=9$B$9jAFs;jH9^#f3m^7-@UM}~)a6JH?a_iM^_J4&gV z>N=p>snf?j&$)49_}sal*RxqvsWCGq4-k?>aD=;3C0_^L#g^vgfq_9FsMZ=4>xuF4 z(a}--%R;UmI&=_hIXg3R?fP{zbsHKQcwYEkh}Ij|uM-2JvAJnraImAJozV_FEwCT# zw@IAHt-R|fs*Uu+%4un;I(_QY{M_7y^XI|PxG-d>$x#`0);HC6TH%gV5W3~}@872$ z$!c3`E7_~Bt7T(oaHt|2Smd}5rcX>V^w-I|Xb zIUE}wvx@6|BtsKgT3X`KsEC!7M=fpbXHK6va_BIP>%uvoq-CO(WNAF$=k>L9^3SXU z1bAOxZ!8vPz5`|Wd-v|a9y{8!x3|$0LF^>R1O`AIy@2WI>07sMPEAi2Z1?hd?QF~9 zI+muVd&fPp)+!o}TX5^>>^jO{+ii2o@+)^F6Q~y#1B{7=4u@O^Wb1l%Y6@g)hrX_; zk-c#5k?Kjq2=iLooGNZScZ4nDXl*R-t?9%!JY^)96)?ds3I&lxsbhJ5hm|b>>mAA!o&*urv_KXSK&_(% ztaH)s#jlgD0sApWN=*`uE3#!`|mphC?X zG~ly}!G%!R>*(kJD1v`Lo`3wqA5pKR64APEY-+@OltIhb=qRefZLMu@ojtp_xODgK zo%QuKX_D1dDQcVTSPVGo6PVw9_bs@$tE=nNPe1MN>l1C{p}~Rnw$`uy?^oF6Ha9hQ zc63k)*RNf@e)TGRBW;!<7NyffB0e}U@cE~ows&+WD-W(9 zjZK}~*nrhMuzv_-_<#P_|5{pJmKWs(l(D>Mc9ICcP<6F+M~@wihn<^u$G`get7})U zif#f#E;Q-w+qcD;PB9P{t$+W%eeb{jK~GOlI+e71X?|j2f|7v@uc@v+b>bxcx0oQf zbLVzuC0|`x_1?Shg5v)A*T0TEd{`H)efQn>PMtah)v2`un@aDOPNjU(Rfd*YGn6EM zfZLAt=C|KEyR^6fcqRudq7$5m$4{L+`R+UK)YR6(yL+yaS<9on{M~ooahx&)+E!FJ z1O5FC_4RPh=YBrd+SUe62HAo|=}*p{Im2^Nq|;NAeB@BW{^aA2=NA@cXQnxN=FFKR zM~E%1{h*?-`>5H$NBS@(Jcq7Q~kL7q^#(az^FX_*|*-}TAnu^ z^WY{Pt*nr-Z@&3vWMsIxsfj$-*40hV%tT{R<^{;Y%E~5F8()3()%f^$*@M(86Z3fH zmhFO8OuazYTE(hRJQmG5hS?_Nh~oI0;<}*B2}+p0XNvfw1lg*B9bvd=VksquZGzT* zfbkD-Y$KQR2OLJ26W3)WmHvxI9Z&pTnjI$p=5>PG67M@t!m+G(`l6ttk4Aj0H@>Os zxN1|LQx7069M-I^y4pQH|+d8SvI$96u)$Zjb_ zM??ywv=E-$be}dWr0_&$R=r{RS}=S}iBwm`YN}(4OY2v!Ub%Vm`f6rHIf|wft%!hE zt%icjPggluQL{4~^4O-HX@KKn6W{#pZ;WQazFTx_uud90rTuxT$r!))r@1Al?+EWH*#ScMOsY3l~GldmG8d&HlC$`F@OK&8}j$w zd+%`};B9hzj5Fi07ZpCw8r5_yzdp4ub5WcL^%1 zudiI&bS_`HbnE5~Ozb(no?UrgapB>}aBFK@bvgpc zfA`Mqx!D=$aYvQ4kzc=Ni!1Pl{+)Se zacytqxq{imSYI3ChMX*Y|JFvKz=4q2Ex36AIG+#PKKXlToYx-*U2a^efB&S%-FSrm&@}jIp1lJwMVP@avpu>1cM@3Z7p(E| zB8agRp+8;(#ylge4RS#l>TTNOX6FN{l3u8QvlqsEKEU)TN+ZhB9D9M`fp?-L3I{wb z@h9!Pxq&tb0&E8Vf%1T4#iCz9UgmifbazaLN?R}4qXIx+@wFn0KJ0hAykqJA-S4Cx zXU6-CP0QUa2J%kNx;v(MU$~O6G;*{1etpX zq7@|ukem`MT6(`=HlKcP3;Y8ZM$;9T1Ph1FZCzb0N*1jxEhtq1Y(QDo15-YcsK9Yi zEJovXx$BlcAllG6kH{PZ=U`)m{VJzzZEl=D_w%>kextRitF38iYHDk1t*rF5YV%;S z4FhcZ>8Bs?LOjsl2eSttbd}^vnHQ#$EU&VgOr|PaXCq&LsfN!KdKtjR+6R+}_WxxH zk^w%C(9zyT;i0$u+u!^K(iPWS|NX!J_x84CRB%azf|m|N77|neFpfnly1P0X8>;5# zSO4 zhNi|wK65oyl`D^y$3`Dgp{Pk$RD>Vgp9Y=sbXDn8XJ-ezI(3AmaX9R*Wi~UJ6`F^g zngeY6YbvI;rg~^_u)eN(Y;58WzyIC!8#hpj{!jn*ZwC(^92n@w#Z*N&Z0sD$_}JJV z|MW)`l~0~L`QH2Q(>R&YprSZav_Ca+1 zFcOIvtAL}80I;V4Ra{YUjgB}v9j`drQG24RuBk3rQ|Uw;ClqK4BPO_;q$r4K`#lPF z(QJ94s8i)$1rQzv`>XWNrGQwwJ;_AhAL`UJuL2iUIY75Qa%&d^X6%2JU*RccY! ze%&I_f>w2l`Oj)cs+B?0o!k8X6FeX)85vGpJednAN3#588!laUA0>ql9i&C z2@*pqlsK0(NWfJxk=zJvfX6DLq2At}mX>C%H7#K+v(8srDIdWCYcb8Mt*f=3v~yaU zkjzrM4%*0Q(sB+&AMZaY&(F_cG!B&qyxH8$8KVKWT*c|E@lAz!FzJxOmP{|c5QlEo zfr6lJ^rbT!^okyNAlo>~qTA;biLMXV%$l51He`ttK;|HPEJ%Ih)6o&i2ksPW*qZ9j zhK72cf&$aV34Kf^B1kqy? z%iU2gXEvd$s}n*FW%a4)snM~A;Czw_hgEw=+eh&aDI%PNQFl}ps2=K_Jr&UiB>>HX zZ!o2#XL9*xluZ4S|A~q5;gJW6iwlmUfzX441I$g3tYe#p));0FW21|WF%ulr-rv*m?t$ikN+;}!sxhK%=OL>A;F#^&7HYe>uGaieqyTg1g_Q;XdEE1vv?^7w zX2?boQHR+&r>ap^$Llw>TF+0ftmg9OMJR13SJeDW(1l8G7{r|LBx(rZS8e1x{SvEC zz%=<$vJq62(E-{q>ak%QB^(GGl5b}e4oQp?C1Eh03q+EI(rs~?EiP3dhsBPLdld4P zO|uHnI=~syPK?ZiyiWf~##WyZ6i0OiL2lm&aJ$1Bmz36Pm)8JpMKN0&JvBM?!;e4o zboadf{`+Unp2=n*{IWQL!rc))+xANYIVuUrFUU+O1zH37@PliCOgaBQ2WG|((6wt< z!FckfU}m|vurN0_3z}+gZv{9()4^dXrLAGM0-J%mKu7#crBj~OO-QFx)*-VGjM?Mk zK!rHkb|auluA-QGGY>W=cZZ1oqW_fiv-5#vqA2?7{WHCl_IS8NU5>Lo< zES$)vW@}4&DG7(BNDz%^ymqmTnRI`DH)$H10!0KIednFGiM~q|B8*+J2L}BTgD2z( ziZ-Iu%s<=YP&Q-{HXhiFaW&4L(e!q>qo;tQlmoSFGvH8hxTFFVQ%dZFPREv*8)0hO z(b0lkEJL8_LXRnb^XCOBBAL5T@uy>a^>S^T@Rn_GB( z)jQ>=KX07KG;1zLZD|u#77@iWTrYdeLama%gG z;c!zqA&=hQ%NM+@0(G5tyhspovMkRdyhpq?;Se8>_-0y05t$v-92eg2C5myuv{nOIV5HL8W!tcyU&1Rtx!?kAtj&n>w-GM zN*T)J4Fb1iDdHP?i9EKJ=zn6;`PSQK0k_M`%YXm-H@PfeV-qV=+!52;H8nMZKpZO) z{)e|1?Nh)H+W@CjItiG;$q>PN&`&0S5LT4bKE88>ZG#EKEc4;W$lW`4>+0$t6}kO` z`w!qtmd%D|* z5`?iRkB>VjU~ne0f${L_$_nt7TR=5`{P8EVvoky(O26O&rmMt>adth|+}y-fJRK&f z$BrKZDNuOwIJN`Tjujv@*cDI3kgLs2wbCU}+SOL5K8%6y-o1;TYD%&=CktXEt8^_O zb9{Z#eaVGN`V%YgrMPqgrtaChm-ERd2$_aK6@YjYE3zD<+vu#N+Q~tk|3+`8KatrJN1XkELnbO(J zZ)tId6bw*b8e^+jxppqV+7N2bPF6x2E6<}y^V(3g)b!)=m^SvjEz2rkA?(l{`TAM5 z7N4GtGc3y3wWWYaON@A3J6NY!3Q{V__UQsv|Dge^dL3M7FoMyRcJV9~Z=A@R1#Zh? zjOTi^S?Ud{6lU_f@4u_5tpNoQj1(jTN*Nx003-uq?HSqw1DQ-kre};;3IK;Z&jtCw z1k&=8so2c)JU(nECno3<*&w=4Pj~lcpMMUviG;au8uzyN@CEw*^z+YX$%1Upo;`Ww z$Wg+*^7iA22Bz+B-~5fE&CM-%JnHD^qy^!&d1Pel^5rX-bS4r>#>_sfwTcxY0<#4L zc|hnA*Viw^e-3E#habLASEkOKIlX^RAHk4Z=_EAMn_KcG9_e=^9?IP_BMB`f%UAeKpm|x8HsnLYphezE!pjbMigp{{4do zeecT0jvc3{;XK3K%tBVrKECUDtnDaBlRdyKNE^x>HDUIHCh!!E)^uBY%ZDF+bolTQ zrWmlZm7rfNfaJyHC5EGW_wJ$k@YsCm(j|_OE~|pIe#T|-=;#>8yuL0?P$s-RjRDIu z8Tf0wRpD|OU)i62{@MHMAAn$)-o?2wvk$X#b3gw0BjJ32>jw_(A?^`9At93~(reeV zWQiM5GQEAh@4ff_i4!MLUM2c2C3gP81zKQjZ5`M0aN6plh_fskBw1iB|GS6yC?ng~ z9?8tLI!nIGHC~x$g@c*_Y`p7YI>=ir-F}IC#y3{V=F`fryKqQ?2x_a^V|E0uG)2P4 z^}ix+I8@l8o_~=|by<|JCR8xIM4^a7=eU{Qq&qIZGA~!k6O+o=w&yVfVPV_HwOR+J za!>S3ZY(hiQqr+-q`9`TF5!e7gq2Z&Mr3W^EkND`vl0U4WxvZ(v`Q0$5A6>( z5vz##*$KrWF?#mZ&8>w;tNDVM1nbi4cBOY1ol|vTu&5V8yKu{jHa>s;biU#Zi*+X) zaseDH$`>sP&xJgx8GiqN_Rg}+Z6k`pmb5ar%Meb=%>BiM?dLKt7R_u240Q~PCdu&{ zq`n%}RhhY-Hi>ZHUd=N;b1uj-vimyKnXROM+DwtApBlwBDO4m?fLm37Th~7U+|CM! z2n>aSpyr_da9zE1^X7Z+y;uFBDiU-{r>)lR?k-lM_uP9A4z#*PG-Hidi0i}i9B^@+ zyeuBi1fnwDBo1&ko7MF`Go3sx0MSPte)#IOt59C; z-YUNH?t40!ldfE4?l3g0!-4vY3jUY9R77{XR%Z$7X}T#w?H`|m_RFx7dkgrD6~6WSpHNdOB2;vjgZbHOroFl zh7PqGi^htU<+j=G^oXR*hk?PJruDPgd?XqnVsn>;Ow;O(=3qEP6s)a04AciOga2W4 zQN+v|`5#!Ok=rEOoWMLYMrg?=d&FKZYu0#hHfDZ)cU$5gS!Uo?QDht>P@Nr=&k3UX z-EfhTeHjA_)(cD5W7d1|@Nb=_vw}F8MRtl{0#c)fGlOFOIsQtsGTV}L!mCQqe8K^Y z9~ojZQnGLSXi6<<>Ul2?SOh$C%2LoX?^C?uNkgP7A-1l=$O>>EPrYm%PZi@UYvofN zOgpTebZ^xUkJ??R+V*hnWFYH9YEy4EJN#bqsmgr7dv(Eo&&u8BH`eC~SEWKoIwLYf zTgLQ25&ugL=4>ezBt~Ahb+bdv}{0m zP&lz4=JGds!!cTMy5>~>cWs>t(%I>ai4?HWWT=YeL2a#pIcq2|hHnr2kV+=8AXQ|Ui$GsO-!>Z!5RnxsPdIl2Y%nP0ZXyvU z$Pl)vpdFOi>vaJkwr?t@R7E{t5O4$VitEdf{N^H@9e^n!UOXdSc}lUd{_gzb!H8L59?QH`{xstO{2A8 zBMl0TMq@xw<~Xj%2#{{j-NC@UwY5b;S6uD^U;srh9+_j|nK8PCUheCC9tMXqIz^F- zrSfU3{qZNC&|!wm=+FO&Wf^II@ZpD~Q$ncdSMP&(Ek+mE8488b;QTz@R_HK2p$3|^Jvi71sMQv&Cm-z0TNC1$Dmo(u*`rf?n8f#E+}?=H@0nNBGo3 z_r!Yh7tQD|+568ww~eITAHJ%(8vx_vk!BQ~?RD6_H=ley=lyg3+4HKA?7s_;GnWtviwq*<0= zoRmI>Fg@+hB<>Fu0(4pc((~6=`f{f{(YaAif1S*Bdghs)Af_2VW7k%*J z^{FT8oiRg|&J413NtPsO6e)`FO*&K`&7b;c`YxJ0LG(mJ7W1`iY0eQ;Kn38mDxzcp z@Zg}lwRKdZ^11cJi}Q7oQX{i+Qu^m}vpA^?#n(lG!T zM3&MQg^RYruvKQ&sGujwT0MK&qI2S3$IoALp?mzqqs)-8{(R)yT>4|qin>bARO$-= z0cVUZR30zdsT2#woF4#Lfe=EaSSp!x+B>Hz02#v~N`&J`Q^hV#5AA9G;nPr3({*J$ zACnHFYMVqS0@lY^sjtZ%na%V&^Y!9ZRh_j&^pzpRrgfTrv8(!)J3D){2arx7<`CoV z-X2ZkGEL09qfx9r;NjsRprag`=}GnFk9)Uy!iXULspBV%^s(Hqp{$-4+%FZQenh>;rhVd+@KDvb;5Hkq+O*)Y5kH>iJp-CdR!Dh zVs)-g^#lY_7@e%8Dhw}pu)F`y@86)JzxU#`-@f_c>U_$K09ZyC;)8pg|9hiKa^LTk zu83t3B{ga4QsST}SzuV5O;f5DGOEf^q*0%?hDW#eN4K|j-+2DQYtLU?u7e3|p}<~_ zKizon-p8Le@*{fd<+*kiBqD&$E^rn)zj2+uJiE(r^&dyXo_3ZEd{Mg!=vi+e!kDFEysmjwAMdKke)i(Z{8C#D zcK2}O#_k_(KD@ElF9StLBf*u$=GxN2%3Qlk_Vb;+t(|=e{_=A3rL~nZ4iCDOt1=z- za5x%uM|S`2;l|Nmf8c-q%9X_y5=N)Qhc~w0|MbpYub7)}H|7_OvtO=%VzJn6&DD>J zvW!Ha(;>j$4f@LVrs*BFDgg?5`Ucythg1kU^>k4HPjWWbNjVeD)0d?)##tj`%hI3T zp^+*XW;%WtK6wx`LE~7{M$u!Yl9oVv)@|U24V@ISS`}oLQfo3&M^sDFMx*IL2c3Q) zAYwKgLJ%UQFU+&53VpWY(BW^$n5<7>b;9FN0)h1VtKQAAFZ5v0C&ITt>8q1A>Gxqi z&~F*AUccF0sD7s&G6E^iJCEqM8OZ?p`hz=Dwm*1?KxCLeA%r|10Z0-z39VifC1Ml+ zs+jQugDPME9v6l3*PROpN_d*gAQcs8QRL29V~n%53;`fKWoFD!08GYN2Ov~>uZamnaD638Tt5=rh=9ap>iw_TRb7OSl z{${n}NbC1HW%=R#K^|Ut?vh3M+5N3g?``(`!^>-HKYiiC)%DfXJ>KV}nc9YhB`rbv z7ra3~XN^>V{wi{Vj`>o8BB)}btrhxJ5u`w0OcO!J;x^6F#JOtX)TDH3r=0*02qMvY zuOgzV=Y_nUZuxH}A_GBT0HeBE&n~py*znLZK2Lc_#A}C6qB*t2!hy)^n>cnRFm{k-L z1EUv-fWcE?rZ!Z|qVQ>wWi^*1We9lc%$Px7W|ufN>_a#?SF`zM$`(W_s!$ljlx4MD zzcAl$s9=Ij;|duln6TWSU;gCM^3vM7pKVp?F5bU4`uy-VyKJ}L-|6N~^QD%1;qrx- zuV1>lG`HG7GegSQE}+q;hE%8~!+FZrvur(c*XHokb@$no=I=hfdwcWXXq0RXVsZ2E z)7=4SKIn9twd_}KzFuwZ#RXce>0?J$FtE@-+eOKH5h*i85THczH_`J-K$5001Y(vL zbqoMw7$Q&o!m7{^1D9;9O_HQ6OG8XNk+qf#Lp916JuhjMo*2dEZ2GD5C%`8-=jvJA z{%6{{&R*116$ApFluwGGPUA{ue|~NCauanx01u2ZWXA3kJ#UlDvNgoU22~jg`%}m3 z3WY+?-S-ePpcP*sK2AfeJ)!y{&MsaRrTTc z;D#7f@f4Y%An${7(OJjLF+R=VRWoL&Dgk3-o1KK?7Lo5Zc-}zk7EA5=TqC9FeHER` z-|@5qT7(xaa(iKYZeitL-no5y`%t(nAUUVYE&jWgR(|&CwaZIMn@~T!mkZF~NrXjB zEKnSASV;{U&MZ6BuX3yP+#fz$di&<~U@+Xxu~Q5Q`FgYQ+Dp&B`TWXS8!HKba=d@W zmtsP+Gpw}IdSZfFJw^y1s6I_!Ryq*^-_=A(<0obUpl61e5m7Cxfgn+a!@*=e5gTTZ zG3Gmp+pjp!sHpcI1ZS)o0lWX|lJHh46ywL&21EGP4WbI7$%Ia(ClG!2U%{zb z{vtWy`Jh@leq~jSYF*=^!8}kK&mRG)f(Q{PRE!8f);U%(#uAZ2pwpe46}I1~(qH(k z(-V};MAoW;=&w8Wm@$c`&LRL|#`o$ueZ~O$m-2p}F;liPNKdcF22d@_QjbC6TrH~s zTok#8;D^Rc+!A7xyeO<8TT4Ou6*nf6K8vC#Vh|`l-LbZ_+_vLrqUw^=xl**itTX88 zF@q`?&RJtkQ4}Et0L-NL7BI08NTgLuSCAirJh$p_p4$rTkJ8VDLJ{okp zg&ILL99QP*Z(Uz`ei=1@o*a#sfp#71D~s8!t;1o-Y(Wg@)06>vI!XEsG=o0*U1yz3 zoQMub!&42Ws@B-ken|q1HC3_3SypELu9xmdV@gexIdmLLD#=`yrjy`y1jsQlsz#{j zm^12&04Bmyg~s3n{fkP8NQfY#3Q~qxi3u181ZM#P1R4zGP>NBaMmB8RSo4#rh(s09 zif{Aj6f&)kp2qz7%8Jup-bi~UayAoBh=lK-8GnVr&NF7pc7{TUPRCx0NKqCdQET-q zt2GSA7|OE5_r?qbKwp|AS|M6WZpt)q#+tKvb?Ce`B7~Ud-uoyDfatH-Wl7Jta|tOB z6+T4k)Hxf1m$5yBXci~==`b?{OboWFD18V~zK9(~-{MY6p^Vr_TqAQdz5k{!Tb)z^ z$&4W&mKYVqD3qf}#uzrX+Z*1vd4IlUpI>ih2Bc6Lmk30Fj1UDm5eI@}4A3v|>HXdZ zpKb3SbX5aXm7p*x67r$bx*Bfwqv0BNM#JvY0NYjQIas3tzf`Rvf3yBR#M_Kl1V|6#U{rDF;E=U(qydK0u=H1 zznp3>B^E%Y+Iu2vd9sP=fyKBeDJxh(6*!q2i3Jovk=Q_3g`o_LR}@kLfybyqL82poEWrZbWQC3!hXjD4ckq|`?$UuS9KLW-YnE)oTP=*?i zHHM&UI7$pLf(S|qD06@+=v(!q(ARn&=+x1b$GavbB4+&UFhjqyY@RVwwlhc-c*NSL zIteI61m}`gt1%jxJTFAPc@iL<{p!Axw-t!_{P?Bu%qK=ipePuQiWsF{OVcbhhNfXG ziHTHE249hvT9zcj0O;JcQWbwLN+5-P6J@Jktn#sg%;`ibMcM;Hbdo zcXl6aY~|uLvjHShZWZzmw+}pv_`i_xj8W-B84VL54#eya z@Xnp?A3u9=cfZ#QY^|m!f|?NRy{+9(A2u&9Jhxf{h(Koij4xw=xi)HqpNp z^wkMP2}&SqoplW2(*}DqVE_qfBnbrkeAwv?^1e&$B>bX-v?;DHFSFr^nk(mO`=es>uE+!?MsaOKqBO)mpWBJ*|^c zD8-l3Ey}H<&R(ZG^gdb_KmrvaAuxo4m?$ZuZm3pEvdq>RX|0~7WNK_I5sBu(=f3O& z><)^m6iOLF-G+9SHWSxMlZG`ev5B)Vtbl+O+%IEqMn^sz=Kf$<6j2R*+KFg}rm*(2^g;g14l*0SLaMTNNGar>8)vt8bK22{! zAfhLw{3A?I6*Y#fwPXw%BO*a!43Sk-K@}kV!S&dGJ&GR_q~A7n+6>i42QL%RQwgW? zY$)4Vo>cXr#7F^6)2vai6H!r=F$UGI^{Y{Q#aVa8Q*%^06}Os~UR28vk|bec2p~XJ zLx`m>!}y|B6@7F7n?6;4zQ^SKq9>Y<{>C9voI;|Q41i2*=8P$sV~lj>pekm}_)f`w z0Anp%D00(BWux6u(4lm>>Le8rD2_GyH;yH z9~Ql%cu+{W)$5P$>>qZ2{=(XowWWHB{xIfcjATpFp~y&*z!(L(xt+g#WB2{L+nfEE zkIjV_ml_ur8oS-FdpI1Hy_*}`Z>}$_UTnddugC^rfW)AYIcwlWb!ywAFJMN@MklcLO*TiH)vyz=_h#e@(Dkzg!Fflmoft}qMo$ti(wVb= zMtEthwb-boiETD9L4jPbeYAXgfAI0<)@G;Ub#_R3&)QSh<|PdTr^-ikoj% z$}~wCDIh@f=$63{$fetx8~*v#%9^tQUA!#p>MQ&uGcl%b@WsLWaO^78zpc58L6*-T&n1p=LHrv!Q< z97JGi_u)}zG%9T3=*a?*zJ-SrF_Sg6Dj>uPk@1tV;)ldvK-SJ~9%txN*B$!o*lcI0 zst+DH07{dzR?CPe&n3pFN|^D3fB;gEh~%X&eK3V*!#M3aR*|TpW1xS5xbQPbRY8a% z!Fx#)J9e8Bk=ey&#-A;2slpg*j0qw57zATz`t~>elBp_4Ygw|oxUkS}oloXMm_Vli zt0?!_ILNp5x`hwT3_rWF`ub~^yNB|p4?ey1U?-3KaG<>#JKLT7^%uhWLi4kY{%(Je z6PJX&k$?Z;Ag`6fQTN?jyEitv-6BM`HH*s^=70H<=hheN@7&lM4IdV{ZtWax>>NBd zmo%Cg7*7(n1Y{PA^X<$XU_7S1!9{hh(}sk0)TUY_OK`<%Q+tIs5M2hxfX95j2V+zyK|ZS6bA z6?F#FO)(-M2~uFQlhicSW4T0CQ_R1z-kRs>j>gcmRCu6djM}(duPxRueDshWRJ|i* zR0Y5~Sw04%q3z6-#r97xuU?<8&!?siIKTpWMqw`0Fn^)dbS~RD+}P_3{b`hEJT}C3 z!y;)Q5@ed&-2?T6?7mwG!a>!n}2a>{`Cv(rG#4ygL6!FJ*;6DxJjrHN9W|vORYr2%w(qV z%ZM<(R}&v2h;(Af#f%>WM5L-;5n=nwfwdny^Q(=Q1UfhUl$iw+`I}L;GwAGGxr*pK%%vnI}gV%$V_=Q78dKY^@OyAH&pC`CU5d8Db%sp>{2;rvT(hQEzGd=nQdhcht=z5iQm2 z%ggmwS74U7fCT0y747xraTGg8J0?87g7AJcum7IX+%zhI>jnD#`7G@<; zyB)|NYp-3XuK*LheHypgOf_}}qzT-GjLi~4xVPUef+#(H z{P{%9rsv+#5hw#BL@ylGiigB9#r`?+XPkE>DSK?WUA6@<2eg+ZlQM3S@X`; zMgb)<2IOLFDxRBbzjbN;f<-Ns5C+C&0s}hTCP3c%e3aFiMT~%`%6G7M637}hY^*Uv zh!Rv(L?+f+FhAmC1S0(4A^HIcCmlb&6_2KmdTeSe6(YtMVnT&LRlh<2f7Mx!K!EC_ z()TYHlA8#snDIkawlnC936;~qm}OZ;sW=w5t!Aqz%i(YoLYR7=D-r4U_6480T#)MI zHAR7fK`N^Uknz?5?e}Us=u87%9VDHAb)ei_sgE zLvmsEgDblsi)CR`)r3(P4FP7iREfDKVK{`)pz`HQi~r{J%WquG<{SvPx@0aaHum>j zzraRkv_14oLNz8l0Rz++7Z+RYdV1I^qUuyF2%ukoizyF5Ov4+Q^2*gGJnOa=m@Ib^dPGJ?th}M{vO?+v&S;*R-26y+0Si*QVZJXAz%tEay zM5e)$m7ODVVlOoS1EYweJTIWa#3WWrCx9rV07r6(POYpC6ODUv%o@NvaCw3Ey31Rg zvhc%`(A=t7jH{V_bM3+#tMw(o#S&m;D}+&i0WybU6ijGBFB937B>IK^V6Zm`B@7;6 zYUVu~WP$*JMvxMOEGRV~3t3ETWhj1irM`eM3W>^&1_B}okOB^1MH}#hD+$U=D-ZgI zgTYW?4JcErSF-%#;<2nnMFev$QJ4z>7#fi-Bmc zfAVqX=%|dIh?qIXpsL1LCIS#it+ipC+RYe)F)HH6W6fyHBgsmbh>gidxwQ^NfKGPp z5?7Yx-pFB&$Ro11s;-05~Ztyb&B7hhakTMseZxqZxNG-6}wwb~0W zyf8O6f9uxGhYug7X}Yqq^5TmxwcG9L8~gd^pI2LmApnGD9Rh(K9d`RSr)+1OjVz)d zQq>qjp638+SuIIij54|85s?C-N|^C|A1^=^AX3n|;QxPlr`VZW5i=816;Wac!CGUj z^BSN;N=mb#=QF;o6cuKw3K7B>5h0@Q{4F&q&S1Wlv=VCoK6h47B|0l?V~mIWVyBmT z@M5cRVX3v;Kn;Lm(c$MWH0#Zu@Vj?DxV<|l>_}K(VgRAc*c+6>wiFr#5tr0e+fIM7 zzVfera^;OHjTNVW3aFjYg~j&mdb3-MdS1F-y`qLELeLb{NYSdN&Y%eTZR3jY6d=YB z&xWc}z*@wR+c3CvVeJ<$U3uw(OHX$U-0k6?-@p0EgPo&7qe&H_#7YD{6NXAb1XeYw z3|2rjvIh8}cz^}aiAFU9%bZxSbsP#naUKDUg!T@*9Zz0JMCNMlVk>P}O9BH}U^&IL z#rlH_i~ZOc1~CjLaVe!2R_1=Pwy@-2ForOScyNgQqyEUpy3xx^?e#Vj0UGKQ7qjHb zVryez?x=S-iZPHYd{N{BMuZYbg;>e>h}}-n9pzpkjH#trBTaV)u~+y=EKFb`QUX8% zYrv+6*zQYr=to`?DHm#PzEx{xV1glJz#MR~Szm56hDG63Nnur5*1UM}!mDfT3kC)N zTLle8cLt-)!=BH5YSZ<_)=D$0839Q~b_uw?+}!Uk4({)kUhyRqh(Ipr^h+U92CG++ zf%0yl121fy^C3~WP|vO}%*~(J&KQajQ@6xMr#Fg{z|JMfdd;;AVBi>a!de|mjoMz} z^9ZYyfR^gD1&12Okn!NdSl+n5ITVNzskXDUQE#wIcgr{wfwe(IpByxZ0GcwHW@5v{ z&N0!%4M9|67~5!7Kv1X>DHP*Xl)i^x4>Ljxt!DG}H{PgJ;_mLQFMTzzy_VHFII7j_ zF-BFTWBB;NJXWnb8V;MyhC)x1k~rtGEX(seir{1g0QW!$zw%}2Dv~50jjST8)yl#% z!&(a}TG4HvGz~ul(tce|qn|cVB$zrQiJeH^v$O7gjGs zky|%!HX4mL-+c2IzxX8)jYh+rot^oGgY1{7mL zj6Q@UOS4+dhd%TPc6Q8|@fBezl$o5fNs=(1gzkf?{29i7f;LifacNimY|(6Hjk-ex#SCnTUcHDIujeEG!Mz>}KuKo9(qir6@@itj=H75~ zf9ON-YUhl9b+z&D-?;YH)mqyC7y%YY5{qU%1)0FEm-i3*qapwfPbMZX3oW|1vUF$b zs2>!Vl&?pp!w`)z7!yjtRv&-=(ak^Hx!)oLnG89K)nc5S3b*q@E1|B!DNa3!}&kYDU@_v$I8SWQdrcbn0cOIp|KW z_~#Ek-{=;RjH)g-vsW*#ytuZy)=UzB5gNv>&f(gnrCWQQkn*@C&;(s*+V$1tTiZh&j0m2zBp4+R;s7d{ zM!;qt?|-&&e`njPs^vynYb5m}n8U$H#1dGE0b23z{0ob<=4lcs;PT4SMt4-^1)!lQGndre zGG{#KzMl&_Pje%<@b^hgj*!#t6#y>9P+|L_03zqe0>%4}OH!&hH@?ZLwb zckkTE8jYX)x8M<0E(urOcQMR#|17MGR`)9-%wyIQ^W(n~L0x^&5VfA{X)+qZ5F zhrDf%j)&|#S0hKF0O5CY`p*8dzUU>uJ-cs%P)62V~>p=83Y26LTAeMZyaKRiij7l z3V>?0daafL>b;*PS5yFJF3x{(X6WNHedzhQ-@o8Ki#}V;Pefn(XzIu2H9Jl%QkGs+ z08*lJB11u6nWAvU>^DtBYvTE>GcF-c-i zoo5&@tcj&85?l7FOdvW}@lx@yTiiL!2UJTN?S8+197lLCGHk2B(vAQTX4DAFbMAhyg-CCuowFVkHSp)Q#`)ED&znZ9A0#r4|C^~Vp*RuofnT^nR-egID z=&x}3o$=_{T2(M}tyZ(nHk-|SG%CwdRhO2RfBCCl8DlDCnws@JbyuRMz1dFP!gSFirtfBSFqyci9KjYi|j zl`Gb{o1cIFv%mYhpZ@fx!@>&HUVR&xPRgB2SXUSaWoESk*1f(imT^*N;C` z91M)Hm@!kfKVbCnanBU0L}UPnNLiLdl%`3ek$KOfQ7kozV0v1fiQB(g;8c1<77;l z*kBv~{NIixo=fWsjkY=SQ=A6)IlFlh!Z6ZcSa6iOMV*Ugl4QnB*6S8J$o=h(y!W7c z3lKfCWqc z;io^b7O8;&l;H?>hxn(PTkqf9KlDi?!wL{blM3Sc#cM5Fs1n8~A)d%VC8jiSu2y3p zbxGoqlRD^0^9n%B)goY?Z_gQjM1}=ZCQ358kt79Y`3Mmi+g8<@Rm`ooF0C{(GcmWO zK*OPyrBG4Tdd*#1Uuo5GN-yBn{{Cj~5Kfa6Ax25vs3>-hj`lj8W;SP(43G$t2vUk( z$XLKA48WKSd{r$O3-Jp0lOWbgo0wu$YBaNquA^XHx3R~ z+sjQ00usP7j8URVJ8J68Lc`9dfiYeJ2*AVR6uyIC5@zcZi6}vi`bVR%DntZ0pbjh< zv(#*DmwgT8V2FTJ=uEFSJ3_KTtr zA4G;C;=PapUsz;$GBugShRyiZRF|r%sKyvn4|?90FTD7IbG8!9 zje0GsrD>9Y&JK6eN@iCU=4#aL#IqXT5CV_C>mdBVvBp*dj6eG5!~KJUnX>&GLh&f1 zXXdLSga{OXGQ(O*5~Io?XpE$o@jV(up=YNFe+6d06oNi-ORY-YM{APP+$k+2o$5RwI$M>J~|`W z-rw2WJ=#6!KIrS#hzDMS!jS;MN|8xXhf^Rma0cp2^;{}8kB0m2-gxgbdvPIKUumqZ zE?it)tj%SrFiZx@)7ZIRsox1u2U>1mzMfcCFT%_sTJeN4Ax$@^6Lr7@tq_G0&R~G~ zjFrAfh)RUbk^bSsoA2N2-tR<@8UTktiita@L=}cieU%g^g$x~o( z(e!XT>#Vh)vqGFAIj^=94k9tA@gS99UIrhK8i1iN1PfTh#ux#aakJBzHn5D9h1MG{ zURO|(WK1>$2|g~FkS0zQP&G7Gm;}SbM5GKHm(y>BQ8kllLp_;5h(sWe@>m#?s)CdN zhMw+yJeLD$Ao9e3VbYSsEA-LZ3n2nj8R{$_hri|^YKagO9%wZi*DhaLh+2|KMN*B4 zEVZ00Hk%{@0)rB!w&R2uPbPv&G9{@e2hcQLGC`w-)>Av*NHc?Dh$0j~V(G;z7cZ(Q z*hElb+4wvwi=~C8VF4*13D5x7AS^LTOl#Rl^q|Cw1Ctt{0V2k0OZAn7`h|9T^YHL! z(C-&T-}^{L$Y88r84LoSMw6zgT>&Dhq7@>ig7$GfT9cKC8Bis~8A$PcLBvUtwA!u8 zOV${JXOS4)h_%Zmm{p)GJ+C zUtL}OSO4l?O&YvE=vT8U-)1&O;xU!1{pBxzv3_a&lTSYW;Dh&@t(HBBKt>`$gh@DU z0+>i77G>dsuPV8-yoww89y=z3_zhG@F6BaT#{Ow_#gzFBw8e> z;?&MV=Y;W$89FX7#w1AsRbP%r*n+?FxtgXy&QfQV7n)0Rj`>vCPPR+u{Bt0HiDV*7 zR4UZ&=JflUyPx01e!sWZIqD6ILGA})bIv9fCMKIp-0l>2ck{KhsRp+m$o6(Gl%PmC zxd3NHx3_jWFH*1IU?d_GtIU;m>L#QMyRz}8`776ebNsW|2^Gz(xx7y9YLAUhL zf0OHtR!qzS#?li;%vn086BHN}e&?{e-x-wJ2<%kFKrF#%G>{_COO2;FLTOrzHEazG zJ7Y*ZL#H()=;@MEL>b7^ZuGczAWnoP~`b&G#a9rf}jc_Fe6Eh4Z|Vm z@m8E8Odvp8gx42quP)X^P?aP(Il@+vC^QO4!3sm{V5~5MR7t<-a2B2Vo{&xr<6nu0 zgjhYM+vYPaw<=JnC26x!cg_)=Y|jub&%1^8QUDZ4#>EmWP*YfmPzAw&vIM8PiRfJA ziKd2o)vxHM9NW+sVPEI@y$Lj3I}Q%o~I5=Q~Mr zeW{_*Kx%9pOajyy7w61#_2t7sd#69x?DY@4??psnI^P?u)AuiKb#HI)|NQU&&BhpJJd1cA^l{PVxq>K3lF?}NFaPq-l@K+?mSr(X z_G7I})2z{G8s>>oIxF*Bl9b+;Wm&1ytWvJoF$AjSRMRTrl4_BpUaME+qg?d(v9-1J z-uv(Mx?KR(l3UwbG{#z6#b^H6Pk&Zfk1MJC&f9NyyWMKbmHJgxA40XAEX^v}R+gTb zjkTxYoxS%}_^VF0`{^g2RI60uY%7V|=^T}18Dl*4Mh+oVD!S9{{NeY%|K{m{RL4(e z{{6*x3Og`{MI?k^*gUH?+nKl(Kx(ymnx;fXwFD$as48a6_%T5sBK{(=vyLMZ8sm(K z)|SCD=`@NsDUDy?(NR@1W=yn-wd|bnA&g}$m`MTsGvqQ<9cw$GW_r$9fK-(TfD!~? z5W^I}h*7GGic-yXC+~G1xZu4?U?_}mqQ)2{iGeAqL4djOVBdytc{jkg8yg4rHxFXq z)L4*?iz%Zo)RvJ4L*DKWK7V*nOHE&Eo~$yf!Ye;sa1f^dgcXj^OigCMq~AR1JB?xH zuWqQIK~z|+GB@hU`r6vN4?ZJ^0t%kk77Zywf-j@hN1QM!CQ)laXMIQkeN25O&jc_{ zUY_H0AZ1V`CS#3>Mit)sGk1g_K8aGJP+~U5(CLREN>oVHS;LG|c}5JTNt`eQVgW?} zEQUN&gj$@UU+-9=-{w3vu{)}Mu^Q1At-*k`hMAZQ0Zik9V-h~ug#+x_eN85r1&}8X zB_#rV-CgZiPFI|KCQ&h@b!LS!PUeQmk^nlER<}9^sSiShMo-e5;ul zC|N=S$N&YF94^l%`}JhC-MZf&y#HV$vIz=N0CY~i>ez;+suQ_Ne9lNjj~hQmCzVzF zP*8=c#2Dc{o<%0|@3A>}ol9bj&bdaj@#|mz=IH3K)9F-!paz4XHKx*smAt%g@xuE0 z`i&bmm^n$Dvu?6cYwY2{L8Vlmd;YoK{`R-beBr{yd)2!AewBbHNz*HruT=4hmll_@ zEZf=L+1lK!dJw81xYf&}!=p4!CIhgob1IV6YOlQV%CCR(o5RC{+Od=K8^gh%5~~LX z2UR$zfB*0Py{cAb-YTe-NTe}L?8aDYnYps}E-Wmpt*<}--1EI&Z)IUZ1bE zk}qseV+3kMG4MDVO5vplacdB7?({yrw_BEFB+dl@xyYc2Ml#medg_|>jE(INhTXwv zsrmD1(n0qI#GhNzYqJYLQLP4?eto--q(v z`pR6h;aKZgve?d2gPJ{S&8CU#>~g!_s=2*xQ7SB!q;YjenF#2JfgtB5${j&H~g^B|D743ZfV1hsbQH4%zz6Q{fP+=573@}iRE6^`x z(kY_Oj6`l$3Z{GLUwnS8*{oa)>QhM!QK&+><}jDoc73ip9NazZ^h1o}?GZ7~-i0$0 zl%}th9{FUC-gc`B{SZ^vQ^m6@YrjVHKHPZt+N;M8=aOnT?r}i4z1?@;eYcwP^I!a; zGJaN$%ERGsoMI=h;tvl717l2{=SN3Jl>ydT`{o;O0;mEz{qFa_tK_JNKs8B|7oLA% zZhr3O=Qlt3#tYAmR0nF4?cKUZLQx2D67>PjYhZIF~+=h9Ev$s zX4tp?^rxyCS(a7e*_XabX8Yj>A5_(ovAfW4G^`w+`@J4B*Xs3Ztty?a6+h-CZLSuh~L8GBb;I^#l(biFZ zFpvmRf>-T~ijQtRym9k>t(Fm_U2`v9TmRWhS1&E5=iP*(fvF|Tu@vu>98|y4tvulj z3Jg8AclR32WTD+)0#qz`(cxz=HgbRU|Ge{gUnuyJRSgqCEOCfI1&`d;IcKf09{6Tf z3&o>L(m5k2kzuyhIU){amZtSuee$MtR01(#5@SqMLkMM2C@iLoqHOnuyNFNtY@O+>h5CI52_|nIisx=cC=mm#`>Qo|y{JTq3?&*R*?#o@@AJEB&DQ$z%B7`+#mv=68jb;i zs!=pSOBQR*#=&sd3*JX@Hhpp+{__^XzY%5(hQt5+fBw&({`kkq7qBSHZl{|U`TzaD z|L5(u-*(RCc|JL;)x^DfcPr~_zt3*1)h3 zi9(b`K?(xwBxXbE;wUc%qY(g|iQ8E$I6$GyCUMR%#yNMqssZ@V%|fASon4rpTUltp z0CLueREH6_5B2xA4?nnfn6R`nTS>cDR+>FDZtjf^dL7ZYoS0v~u=3MaFDx}tOOO~C zfg|(@v2Q-)pg;*zvV}=t6Cje$p&{dNxfb7=hcWQL2^I@l3gewq zaW}Q`jaQz#x;SSnC(sD9?(plE=DPj$w{P4p%0O(LODri1B$N{LG;=ru1f*%|5*Gq} zFo|dG4CCDgJBzzV*XEncwdCr`(uHi&B*DToKv!4i?shtH(D#6a5(qP% zA!3Z(F{(NlqM2n`@V*zqI4Ch!bF6i$avGq>TAL(^NUSE?f}duXN=POYR+WI38HF3AR={cYimnX6HLF~%kx|TL(v%|sdEWRFS!?m4)eSi^f*sz^(=$diJF9dhLkB|5nrZ=b9MLD ziuQmK%w4Hbt~-)@u0IZ>?2|>-NT8WSeF+YfL`yK`1EW1Ob?V znnlg14^f4Tih%+w<9hSdG*1fBNd?5JT7^+E3Y;-d8v?5^7!oNIiRE@;R_0oUaExkD z@<1ubi=ZGSM8&WSAp#Z?oI!hTNoZ@occ2DYlS?1UGDH!DDpneka1CBAGPoz?3lm`0aAl{y`I#vVeWfH zm^75oWED}3stSlO36UTwXqv>8BParH+FMX#G>yqry@-%1fm8tq0faKfQY7fPk6H$J zc75arq6Butq+?|(VxSf%*;E{Ky3v@(hMA!e3K1;;PcR@GrPCi&=CDP}hlAqvs~1;O zaRvaTzyf2j)lLV)QBf+9D*on}`P)2+)38!R@ZL8XjiM~4zBzU>kltE1nNyDQ02yPB z)o(3}5iv$05)lFsF`z_Hk)560%F}p%|6nrn{7_QjG2RC@R8LaxeU#$|5ueSQO=BiX z*I5gYVGS`FHX%edY$~jsEI5(nlX~hb6aP%v{;kVe)vs7O0-`DkgaFm3TBZ@!Woed) z#-hmeOs+)w%2U7$nufW+GlV{B)qJLs5~(VPRF$Wb)=-*QXNVvGh$do!piHNRvXi&C zTC$cUqr4FCX)KkK4P(YL3q5X{0Sl@p2@~aI5MdBO=}fF3{>~j9#FVA!iCO_vCqe?G z#sSwCu=exSQ3)nwW0R~kXesh5$=+t8(-{7aM8v_U8{jx;OHkx?XQ@9=^Ulf4u<%A|Qx5rtR&)ty}j8{oHD?(yqU} z+IndbErTS%^Ub7p=@R0F)S6bt)2Ki!P=WB+?-bIoT3QyiASj?uxPs}V5P=CaE*gbH zWKm0UR+ScvKGpYh{=dC_-`du1U%k`rMRIw_ZFt>U4u8<7`$lp@?C$)#(*z zrvyMJ67ac;jc$M8{Tug{6N)i$hHc#gL~sfQX#C6&0IV?;$|5o$Cjg@&0tzII1Y$y& zC27;hV#2Rqz3|fIrG#*dioo;&whp_!!B7ANiV!#Vj*g>FuQWiYXaFxZffrtFaO1&I zx3G!G$Rm#$tu;xSWoZNK%2Hw(Sqvc4QJoW6OHhNaa*$=fOscFLj^{W=8 zsH6(gM`SHg1Og=^;F&dtMqcT{>~Es23TW0{Nr zn$1Q(mizsl2oXT4tQZc9+-sz1PXW+oytuGMIDx7KrZJN__yB;wP!LW77eS#>^xpmZRe_!cM>~7FP7`YrvjM^T01&gGMrOkO6z4oMCcAis{g7wMc7_7eUK~Y?T9)C&>&GSadYzf_qLe5)`vv@n zIiD5dnYBm~z))wcvu2u5@HAtkf)wXIz5;3pY2reNWr%0Tgz1m06KDnjOq4oj#?EOm zst7&BGp0o8_Ii5-oxGg#pPx_ z0RsusW)C;+?d=|PqenZD7gp!5F4S5;q5{OWvu#I!dt@x?>A%!Apj3nyRACH+pg4nm z<#}jg<~yr~6`;vKPj<-llKA(Z++S+9-@4Ya9@ZfP7JLm^=t-QOlD)s@&V9^ShR*rBc5!9m>aKfq2uc`<{Y5Z~uL5;#F zSV0ORn*7I?7{3VylsTx`>yLK&`Guvqx&;tPsZQp(rAEE=+~uy{?G5{-uml6O3}a|v zZZ1iaPquf5Wj-u|LRBH4QQ^xV49L`1yh*s+$jVC>IuV{BhOBjD-L1Xdy`#ez?O|R% zJUCdcrOVBRAvmDLxZZa4=B3MPEBk{%uRBm71sSFJtd?c9WGQ{PpHyi|y#mC58VRfQ z?3GIw40PwHcQhIeVt`00s4dQ4Z75R@_mEGJC`!i;AhBAz@yZvF&%l8{z` zLcTUH*ZC;g->^4?Fk_}{XPo~PQ4}f5Qh@}TI9IFHKs1hTA*vt{Xl9O@@$5oy&Qyho zNWcDy!AxnI7C!hWs`wEz8)r*w*k*|mO7DG$5vot)rbEa0;P(y=9_}RV=H-S30Y(*P zldLj0p)Z7g)5C4DS#4)>db$TSP@O;a4wT z`1$jfF4aK5i7uv5~S$0r$5C^tIzjM9GMF#~x?QuuS|XO9 zNE8TzvfMxH#l?oPCv|Ty)*D!^%|(I$kw6LhfG_I@d%IEjATRD899>#$&D9cPEI=Wf@k(lb-gXIq_EINeq!{Koi`L05C9Y*) zF=T6ph@O8C?4f;GKBzUa?X`rLJ<7Om~l1>qIC&dTY6sxiK5B`(5c3w zKW{H0vbooN=SJC{UwFRWumChpM+Gp59!*QCaHLHxudk+Wy>w~NA3WU4clv`r+}LqR zlD%@lw0Lub-C-yqImTK${mHep%X4f50<5<2%C(j4y~FOP8z{NA-@mnWFxOt}dE9u= zd;60+n}>a8&3r9;cAVLWQ z&~Mv|5EW5jr}o3MNoQ4<0@;yme7bpb`PRev#`RVTumUkryHdw%*Vpdsbn-zE=Cs!A z7M;&Gc3!={Sfd0GCZ*u(F3rYTGyn`FfYAuZq^XsR5CP1}ASgr#fse~6257kBvI5As zumTc50-_;SsiFK+F3=jIwK4vaC1l#nu{Di#xXs*)q4 zX(GjpI7Mk(FfIfp7$~EH3?_4x0n(}8GJyyJ{X!%YfguKB zp#t*6BjYv(<>$NIR;|96xfDVQ3st2yzy?~wq&3rNeNtc{a~Tt>f>ek+Q2{;}=A9hP z)=A~shyhzg?J!rwmZ%+>3c8ufA2h#7T7jWEX`{-X287?cqO z&?2=^CJHKualV$WH|xuQQz&8HzySSbW&k@b8U=uY1!e$j4Rp8LR|U^pb!L1ch}qz; z#f+Jxe(MGTF*@I^5m5cLWA3zK;r(#7FUnm%Yff~d`hc~Xc?$L$b z;C?Z*w+}UZ_#iM{cz*r8d)?bxhj|&A)?Z)Bt}ND4gE3lTdf_7OtuO3$%b5F(9zNOb z*B9ru_D6sE@Ye0^!#u=>kvFa^z4hGkavcdzW2`ZhjS^@od+3X)qy)D5_?J&MKE8jn zKV%hls*G=ikmQyML!=c`i5CG62vdXvW=Y%Iqt4s6H1uU}uTt1y8`9q9Qh^xEEq z-iNpH$X;k~6o3EOz2)VFSFc={uNeneb7FROQ}rkk7z8pP29pA6GvjJB@{XC2Gx7Kx zc!fLF+GPQ@u=doA8DJsfi4Foj-t7PJ=Ej50u;Apx!x^f?Iczo4e{?f1{m-7iyxQOl zV8SsWvMRFy(W$Uxg(K8!Nu!=pKe8YJQjk+7A&3DRkOJdfFsrIeoWKE6V!-IMlTH}w zCT*$=A$W!-8_B5BWHxE!sXm{6PEZUak(DASfQFQ872yweHv75%`L)$+t;8|mv|n6t zra=HLCvlycS*=m4*-^<#3KHgAardB8yLmrcS--SsETO`XKx9&71zE!>p&_ zp3QEEAMbV~YH{h}#maUIumB1o*fZ(fCL{(PsMnK5qjorqP|+ADYAdT*fJMP%01#T! zNL&oSu!k1lA+V9-!`lx=UNdV0lte&;88c?gm?_&I3^e&ufiHs!tl>0qjarSMK2Q`W zrpzM2&UnU9M5$0iq|h&Hv--`3wU7cd`#JphAeT5|E=wPxM1>OJQ51h3B_%sK-n+Fq z*R1{ajjM}|?2)vhkM2_mFb%@x27dO!a)0PQzH$GsA8sF%gZJ*tZ4{kh*&CHsWo@DL z=JS`3CU)ebF4;~bqh>tf8_XeH4{n25kRGYRN^4i+cuV1@%Wf_iu z06>H|5Rwsg^%?^C=_aENw&s0v z-AQfe)Go@_D~bpe_z({x-P_(Pf-)?G0po=p?sv1qU7kv$aG3^i(5EUOf(ilBxoe^5bF#5nN)^G>uVXh#>?e{3R$kjYu|n-M2rz z)vD1iUtUWy8wjLucq{}>6JVb7xkd7?ty#wVxA*tEoxMGu9z9H*MU3;Qxw<-kX?Z-X zP~iz4s?a*q#TBe9)Hk{!P2_>0c;Dg=}8cE6^ zPT7n!ySdYM@7}sF-()5M3t38PQ= z0+B?GK|_>&zh4GF%FCl5Hx4?(Xo(V4R<7qNcX#^zk-wLrU9&X{OUTlUjjf8ZEhBOt zhK29t<-P5lDmvFu?j>z1R5db@LZ|&=6kvwx5apc*dk+pr&s|tsn{O>PY(2HL)D}TU zL3O-lcQ8DxG*})U^zwd~SesP{Vr(Z5AKuy6+jZnjAc{a|(C_zpsstiB4Z#W$!bEBA z7xClG-MxO9hzb$dJoLOZ$V(zY1Oibx0&X1+E->W<1l- zkHapnP*ubzq5w$GIM&i*f;G@|0*Wz2`XlS*nL&Ux##&2+5Pg*JRDx70z66n|Mx|cl z&v*K@PaiZI^%t%!WfsH`0eyL=(^(d=ZSBizsMpp~S7r3^o4ry&Uf5`0;_k?Qd~dtk zZ!culbMO?3GK@;U)k_WzkAfm7hd_p1oy|fcd;ap=&)>NAlZ&PaNPrr!0l-ruAfIY} zkOCw?34FGh|HFrOs{p?RB@t#Cyq7AJzG<5%2xOG)QJ%klYtyI)BoG3Ta*_vCa4kvt zL5Rr1btQT4&ThNj_mg}tpkqm!rXHr&Xb!r4Wh*8vVKzGX{^6aEEH1AvWsYizbHoh9 zAR%fL@u4Dmkz6q}Rt*st!~%}cvefQw@3Y}yW?>MrBsInoGao-&Wiu+nU{nl!7?eJM z)z+iZG3OJ>5)s%5o17w`Mb97hH(IG_r99tmI>)0jjC`yRkupjVq<>tLlG->2rf1NI zWV7Et8umUzBx?f*8lkZRr$~rS4Ys7fiOrn2QZ7(y#PzzN^MebuBG{TO8 z_EbZfD)DJd=SWn71DVo?ZeAXgWwNvD3>h^HL+TW!TMP#wj?@$|%7%#nJOvK7Gih92@*y;2#OLOfOL0M15?J=Iv zQ`s8~gUd=b6Pq?G(GMk;&x{!}X3BP^XH|)4Jgo#$RFIMqnIIayMk&f5^;%M|S>~iD zO7B(lWMv}!NSLKaR(fV-EdYTOW$BzvU1FG{#K*}EqKOQKM$zHW=XnuTMDb(JQbEFY=Bu zfga634S}JEw!y25n16j~A)|l(Y-3}uAGmI;85QM+_m4i^I$TWMoaIOoNd@eq_HykP zC?#<=3DE(%)HH8BxAgO$T)n=G8lVw;8ouf=pHx6L0ugxFm4Cjm@$o_I$^XaRU3kZh zr0*X8K5t3g?eOft4*g*1GBf=3muZ<{m@yo?B~`u8bsXDicYaQsq0MacIXa50qoarn zPgm)!WmT8}no1HvJc{p9OV`?>V0F7$vMq22HW-XjftZH6kU-7Fu``jk_G}iC3|hl_ zlmrtI!a`QBWmlcIoqzo3&V}Bn9*hSzsjY5g1|PHd1b|16DBrnp#e|hCl(r-5N%GhrVP-Vv$KxA|cC` z8{PrX)2tWX)7#TU#P-|`yhny%C!V?T-=hg_LMPiF4Rgh&ANQflC z$e<)9q{Yrv`$k%OmAw(1vP5(d(z)~J2Tyxe3poV_B^sie`m)A1@yEADuueSJMrxv! zB^D^sl0@0uYBrnAI%ZEuZ)5>77zvPUrrHdpc)9FW%LNGPG^H_MTN2-Dvzcr&wH(K3 z-HppdFNyn%IAqcoTFa?UwbrLF)y@Q^l)mq5Pjzaong|KNeASGMv#M0ejj{dV-A}LV z^v~}6;KgT37$5}Rj+X||A^-~f;5%ghotqQ4Z~p1(?rns#-cC=WZ{8Z0wq=tLCQuOj ztgPv_=(_K{{M^6(`+xPl=Qt0TkdcnhSZuJtZjDkt-s6A&=FQ)~_xaUzXF9+LY-Dtn zXe$ZG%$AN)Y&E(`z46vLo0KIdU|dy>;u8u1!m>jW1jYs#CAQQQh3@9S`E!d^zcL0e z$Ls_^l8~BhG1NLtQ*)r+_%3A0Heg$Emm(53+-xL?FoqL_zmSCqA-Oo*?1r)w#j?_^ zwHN~Fcyj?5u)tuc!K8oy3AFJagbWWnqB78k7GZ460n5gcMFJ3sQ>p|)i2I4$0&K~4 zf^C45Qb!vLHm)5H@Pl6p6QHKR;vQ}6#Tim?e1_(b5}AUGj`wHFh%Q1>I;K>Mg>8(m z6AOr@`=`$d21^2vCCL`joM~sX*_>>@B@~je?bg~bBzU=8?N2b7u};l460_M~0;O3` zHHHysv0QBA*#L9Mt$C_#Jx=2^)oiGM;bgX${pAWtq0+^2u}J4MOs%FuU(>A@22x=J zG;{OL-tS(2`^u94@Q=Uu!o>~@aU3y|LxO-X077taNteSicVD@D>DgzW1+ML_ukWqv zRNXAHkbyg?@4LP)=gzJAMY(o+xU-&Cnx0+h5(K0{VuL})XYT~7l$(v8zy9gZ-uU$D zoth2_knrO+XWLG@wbu6?1mFYMz(!UQfb#eYpB|!`Es(U4xX4W5u}%w?Apyn!$WEU; zWaD9+#&NXG5h+4SfG}XAyKGw=(8e4&5Fn5dNTiU2AcUZlB1Nsurj$+rIQQ9r5XWo_ z2oMjlkmSwuXdc0wfvoLaK0X<=b+mv0#HP)dn-E|b1O}bfdBnDkv|3wB+n}S3g~j7B zCx|D>Y&M&d?M$G}0Z>4aOs2C|!vtlyST6b=7#OChf-tk$x0wKg?WuY_R-#0|kQ7K* z7T7k2O|@-3jQdlwIrenK+LJ+`Mw3|A&A7ufPAj=Vd;-#ns!pw{Gta<0K!3+f`RC zT|9sB;`z&$`mX1i3@CAL-}W@4Y~Q@w*J|T+96l8;L$xb}WK`2Eh4p<@$%K ztM9$}`!_B=cj+HKe@^O9IrVS;Moh`C-n#z3e*eK+pAWaa0}U)8iN_=fCs1p}?Ck8c z*0ME@*v8zW(4+STwYI+R&YeHETr93%y9Po6(svapTOIdmM08yr5ga=VtIbD{ILS7+ zizrq#@>w6GDmD1%rWUP-xrbq0TMnf z&`RQKCF$$iuOvl@R@-A9jldE|%xpHBeVdc*On}Wa)%ALBKiMu29hi#7zUu&tZLsZ@X_MY)*qA*njEEAY3t$+A+L{3{Un`kINmTWzZt(Km z8@oSw?ZXR~pZU>C=Y+t3jD?RkS4`$cOEW}0#E&eh@PuYd5`hc|AG#R|z% z@=-h5JzNE$mn>jEvrP4l<7n;UCl-6+lQqA3@6Jb`-@1I|%FEw5 z_uOh)Au#j^FpLmKlMIFeBo~8Mp1HVl={s*-z4O{9pYIR+v)H-31cW&xFoa0)-i`L- zKfe3N_pe&6rD>WtY{OI=X zPhWd~wJOWi4_~>|0fI0o4j3q)iGCWsv+(zx+xeU4F0Q%-1SI#^$Z$MK8;9S)3zt`~ zK7aAr?VD}5vmVyY^AtfC8Gr!62%oO`>7U;J`5Pa7vg0%RN0TGRbx zCQh_G zU?eur^ebZlAjF~5f0z@J4(DpNz!n>f5hs~LLSLe)f&tLiSd3vdo6X5~CO`=BxTG%a z^^h>_th8En5v$$Zq1KkhnE;q=n|~AdX^yozy+^E1iAv@xHj|{o`61)&>n$0y@i(!M z1;`xFv5~~=FIp(>DqX)^w%YdAYum@klWkkcR-Azjr)@%;n+PzHQ;T)!KA&#>^sV>L zUtC=Lr{BBOi6le-7*ueddTEFM{Qc+t@s;}B?>_VDmGc}5?oq&s2P6;}p1-31^bcQt z^^%^wbneAxpIvprqv?}wOh;%5z|G`$AMO77y=x!t#U1sNmc3g?9z{Q~!5{#0k{(># zNVYZ+DqVLp;108uD5CHB5;(Kgv9&58gaIKySRi9rND+~?K~m`# z%jHtiI1a-w8jK7wmH;7R%iJe~Y=nW0WNVJOL>P~?W!jsU4Fm$v@mfg+vq1=iA8f4^ zAaR1g$Eg|E*yQBO6Xl-lQ#12Mo9$Q48@JMuPOSII{@)VF3Q;0JPCM^MciUi}INQxd zE4J1P3`ly+P8UyNxSh>rbF!TY(`v36psiSTUAME-uh*VxOWS}S(ORpGWh0(4V)mr5 z*dY0UK$4%jCiIy(LEwm3EEYEFVW_o66i$Q^@zkcDeP~U+YM!1s|O1L+@ z_Tlh55kB)QXvg`Nltd`R_v|2Yq zjL~L*#8_ZzYf%a|NK)x~KvAvx8OFzmdTMurkhkV5Lt+s)*+j6<=y0Tgo)~!M&i7~U;x0jyYC_CG32oyv_*OjjC$9>ZX;?GZp zNsF30?|gRWXMcG6%3|jqy!0$RGDvB_x{Kvu6WW|jF;`2Rz_WEbPd*`zoqoUh^2#Euh3}C)ALAI=QS}gjm?`vyz z`pYL~s8baPAKLcoyT8>YV&~HE?#l7b(n{BhPLxYSt;?0GJGJiSARuYdxND4`rX$)dEo6Y8AJKJK*c${jj z(}fG?`+l*r6YPzXTT8GFV)ob!W;UDs`PnAuO1E0BcGr8gRvXxkwcl{KwXu}YC{1OT z`q7=6KYs1Q#bWvI{>k5e=`w%`FiN(FhZ90*8+ zEy5HG#?1nSF!-xVGp%KGmy`%UUa>k6yP{ z1Wg>UEeUAZ45h?YoZ@lVNR)`>a*4_~4bxOn2yMOI4`zdzJ(TFW;seZPv)P<%XQvRB ziRN6dC#hR3mZkK&yTdR{EmJnL*=+VLWo$7hTqKG61OmR1hTIcPm9kR( zo>w2<9)A4hXBRG9zVxFP&#h2MV2o`GkRCsA8O#ZS1c~M1_QcO#yYb^cy!G*&S}6j- z(d_sj{S{scXHL_!TrNx3wM@6>uOKgznZjUXFjAUf9Mf*K+E@b89uI~Gu+2S^1>(z- zkc8<==4-*(v#?kf=isfpb>=?(PWny!W z#J9+7Hk*lv==;8OWgNy@Yi4d%*WOOH-<-rMNZ26D-Oe5Q*KhdBpWb_U!4H4%;$i_v z$Vxi^3=)q;e^&uwL9jRQt9P#d@87+#Up4AZT#Vv{CsV927=*(h8-G<{%yzA{@B6NF zEoy5`V7?R`IzCTlW^LJWobLY9TBanRw#GC9wj~)P%vZ#ORJyY0dtjQTOzW=j_MkZu zfSEm%Z9q~f#o&#PB+P6!o6X5~cKQH9CdaAXzO$F6@Al(u4{aRBmd2nt*q+KDX0zE7 zN7AA{s8^IZjDs^Dr_oFRn6HPhu|NhXDM?H%U;p^#|6P0#7q0xxE2~9N+$AK)2xJt9 z$0snrlw@ISynA*1%QrrF_p@7*7AhSyA;1{KnaWk^Y!}m8orkAsS}Yc&>pHt;!(aN1 z&BHkY0}_x2P9A|;OG&n)2tEAtymhl4a;gHbIrtc*l-gRRLEBr*q3rP)6a|#F!5ixl zU$?Enmc(o}o0IMAiDw&tn5MjW^UnG6ZL#Q<%OaKa(6U*?Lw=RaES@yxK7q_V7PHyY zKu}6qEEiqr?v@c6M@u|Gyz$*B0~nR7)8fbP-C4&w|91U1uRgmxU#Mad!r?@Kc_<+X zu#m(t+eT~38a}#OfAgnz|M>p3TVtdOIFVJr#()rDal~xc{yX17Je0ROB$ra|l_~Zx zwB!6~^O$T)0*tqU&c$pto0IMAiDov~*j}&K(z5Tn<)V`q$J!bOurS{e zbF-}h5a{U}1!p!>L@6br)K>SWzw4tp{TYXgGT2+Axd^n@;?t@9=Kbq;?o2N|bK%lb zQ!omVA_3vEwPP9~vB_xd{`TOHKEL+S=T~oznzU@u87!G^NUL=k$FA$j{#0u^ZN3r) z5Xgv4ip}V3Y!2DRmgLRwDTFrO$KGZir+)+yx|`J(eLAP9*499P!59qsLccfWSgVMl z2%DB^02vTxF`Lb1bF!VOw%prW!=mr|Rar)qVW_pH(F1iv&1^QCJz)f)=t>t6!+NZ> zA{{5&M}=D-&c9v)Y`~jqCQ5a9FK_SN{?jM7JLM=Q0fyuZBq6{8=|oabNK~7v-RNqR zR6!UEgg|@)q^Z{Kpp;bC_cfba@`VY&+~<=g;*AB{0t;-gB!d*@fP7-7=%JO3gb_$8 zQTl!nQHE(6>a^Kn3(1*_>==z87u>o8?^#C@|KV00IjE zUpM7+q!47*;ATq|37n@r?5$rS7_coyBZtvRbyxkQVgvM^-@bjbd3d&;Bpx687{W-h z4M+lPzLZi*IW0Na?*9J!@6m_-Cv0rA!wJMQ#o|aSrNjeUBp6%f79n&!kjPulK-WlS zhX{iuVGKwzpU#>wxnNte-yR}X`0$*n&)ZCF^+v1 zITB0A2m}Umg8^|R%55!Lo(#OImzPpXCEN0SWUx`Do-4PYhlkyMKik`)8T1z8gJVa0*^5=&6gy&Fh+_{Q*5SrnkJ{uyK6^c3$NF+3v7N&-4F#M zGe_D6Bw2i?Qc9_0TMPrh&BPevyyvz*JnZ-TU5obk^wj&Hd;;;aN+}U>O12S)0Md zE|j^D#RbBdqc6)bA~dxo8yLNVxy6VziBF|lUpB=Qr@zB_O0{Fy7%%~P-P%fk&28e7 z*+@URlmLXlpM9Bq{aO%#ATvS62J;r(dAGbl=+)3C@h2&zRI)9?ZQY-+W%Bft_CLGb zcADC5w|RPcSvDr|0{_<`{>I|pM-qS#V9YOI#Cxn?Jcig{ZZS3(1mZqQS-1YR6s@&~ z{eu)wkB_4d8;~S_EUFohK=Sve0^6L)LY%F}!`jk1#{xpvBx*@uixdk2Fp*3~=+?GP zVJ@`(X$T>&;c)>Zw z>+5O|7_H7n;v*y>jQI`%`TDaA2u;y8?R2u1z8w!^3~WQ%K$uvcl>y9}_@!ydHabD( zLnM9rN$T~q1>uk)1U48M4AyrhTl3Ro#EAueez+~ARI)AR%%=Cue1x&zKWw*Ko5SId z?zROW-XmU^yDED3vKkP-j}fJOelg8p;no7tdtZ9bG~$EIFJjL0=CIRTMNr{)8%Ld3 z#sa^<f4%UME8FTS&srF^Icu$LP938rNqwB8 z6N7FSAI%$#R5!)<2%2OYV=(%W3h+G$Oev+3ZTUBmGsofZB(&Y`p2MvqKRzx?N1~LQ z#N8Gkmm=O~i3m;8gne9&qj!T4e=xIR@I9`bL3}A9b1chp4Yv`|mt~A$$jr}a)2G+5 z*6Yk~KnOxjA|j?XskMaV+33y$3>fpml%L|Xrf7;~jI<3FrIb?1w%m!4A~U}|J@t_f z4-cDp&K$NgZBt4qpK4W|TqTHvL`>7%nwGP)CGoFfeuDWT{MK^U%+VJF)43=b5#w3! z-A)@Ug$)A#NKO|*NV;(*GA9<2z}N3tQbbHs3#qk9(#SEE(|M&6urQ+Bm2pG>*dKjA zi&9FdWLw^2V-OC`>yYhJxZUq}^E`k1_OvYBHl>vEX>6}zhYbd!#3`8PfLxZ5!;)Al z2Jw3tyw0?~9^0g|_l_82%x8cu_1jDkSYTe*j*!idTE|)-4FZrc)3f;$Y!Q+&^Vm;p zb783o1&$IN-_S)5QAb;?EUP>vIY)gP4f#dNQk`%2?`~4mskIUk;1n7R%f>KKH zdNHXKgcLF)Cm}S=ZJMI@k)z{8fGA&%ZnYaUAOVaVEEs3zWm)oNQ)|nUeq0pNdP%Vp z3G+kb>j~f!od8HgYtz)G7SUq#eq8#uOF|ULW-tP zw6ln?^LSV?b6FMxk|>}1{+Op3DaS|;(7gu2mr_b4+hSG(*an~4Bc~%`o=%lpn=z90 z3x)U*rIcbo5>7z`+fB3<(=>%>O~RbmPKh)TA^J7E$+rgtRk-~f1b(}-yVeO)wnrbs z9+~sAX_B$v8w}V066QU!1jt}xu#6CrKtZUr7OgdfLMNHoPu1EZND9E% z68D_(T73N|5LO6CJAL~fuYO6aHSmX5gG(uuY>VIcj>Uzf00YZ%{PykJe!t&twh`_5 zxc6b-XSR7MrK+`|$aFOAcJpr6rWP&00))7TFbMti@YmAEvJ4>*R|ZiL#DKu)ODsYb z7{Ef;MFMzN9oz3$eM>3jd*m1q+HB_Cb~86cAX^ypYJrF(Z#SDUo-)nq zIYpf4OYXD1@Y^TsG5Q$&S&Nuji_mVjvkhE&6&vR>XQsKymJ|XA>5^cjXe};ZNaPrO zSqHTw7+V&typ_y@sU#8a$H7eM6`$;*`_v4 zlSDMxHiD&=(G4 z{*H$QCc@A&6PCjvmYAmLvWi$D6l$90SqO1u0|*d8NXE9)CY?EY=F+>J;Dj^?(9Jei zqI@n%@WNp677IX>Qc5M;au1R~z!>A7fBxC+c9(`y>C7=~V?$-zQpyjSH=AjiMh?D% zB6hpoap}D;5(ULL&i<;Xw}(T@n$B5D*bMiCcNeKsx*AW2Daj3lLyH z^19iU>K69-Yzrh!Z31lj-5Iu}l;5S2t)RTme1Lw_$;jLw1Q%YG1L??2BC zrLN18|DU}xYHl0J&geOiJnM0i-8t%Hh{+M+n4z6ZGd;~+`04p(C;0!1u(YZ_g*S21PF0!*e;{ROkpsJ(czd1`2PSD z!2Bl!#eg6LTF6+^7A6zryKqOR>EU7iZ6~$MQ%`OUS0o-=dY1~-Zn~gRhSgi?;N$gE{rGN3IFOYumyo2 z7$XRnCLsu_X2}GoLFn{w#0V0EN=1T}_L09eoOEHteUtj6(6{%hy8u@ZrNW{r%I^(_eo1c2GbF zavKCl(1xVzDlILSw9nuKV$7TMteXok+fYbAf&tDa<=iKFy;6CFPYH>FAWR_Fu<#!7 zjeOo@Mp{fVZ9bNg?|n5CO5yX3lO+NYYbdSn#+ctix?9^toDkvepyRskLJ};;m>LrZ zpg>@plQo6$h@(7>W&&{LK5i+;23=?&1R$UQ_|ZHEgN7Pok>${8XtgNU@)smpAG zxN;ty>^?*Y@JxX#;ECs8;x#D>Q4nYnPaDp?nULpjY7!x8Vglequ)KQY5v5Y$aNHAtR z`HwG7`H&?dw6=kgc5$Bx4^1P1WI%z)ojZ4aPGxJnA47tCS>y@?kj%Vg`+(i;onP@H z$uWkp84=k*3ec7eg3E@1Qsc{)->Of)e)_}5Pmhlurs?tN>GR=m0J#}u8ze-Mh|#Gq zZF{yNx_TMpiI$o!ZXJY$r&@!r^5z70mt@E!Edv^n7z+LTjprF{WdW4}xn`4j>p3 z(nA%GS#{Dr#BWuWEA~=m=LeyahMu_M{HY4Ad8V@LvdD?D4ZXOr5fF+380B(EAV5%-==`Z{ z#gG$ZzyPps-9aHjJD1ZZ?_WtT?rHWzU4(>~Hi~Q)4P>1GK`U2>1t3Wj>9>>Jmrig} zirKa&k%|^k-buLlMd|sZ3jt|y0wBGRBWucs9!66#N&$#FckbL8k4L6kSQR$hblXL+ zBHG@rJrcXyn}UFBA&eb_vK=plX%S*cmoJ8XFVvPXefaR<_XkQTpFe*-9FE)si%TC6 zmL--&9c%+$5Gm{Zh+8w+i+cInP-Qwn~uSD%&BtHnwoI$&NDgB(xB>$(=iQW$QL& z3&cB-wE4D+*hlQ(x}B@$&hy*m+P*Uz8-T)%$$^y$}MKNX*! zo{nS0?b5ZvXvTMOAwkZFcI;}WvhBG%zwWtHU@*;_4UlVyPK#dZThB;xX=;KHUAx@b zkG_ktok`lYCPiLN$O!?e0J-TH>q^-9bm-}LcsPFL!!+py7b7@oG&(#To_;$#;gpM; z+_rd$pexGunWZa%FhOoLyP$ikv9gsE6x!&4s?X13TXqC>r?L&YbWjyJN98g)bXsjw z1qlpJr?EyE5^Yj5JJ%zG;N?}gtFaaY#>V7J`?F^Ow*96E1&Te$9%4xZ2B)t9L!uyF zW)fhNvK0V%y|Pu;3KRhc(T;Y60aGM5%bh!S9zXPFHekqmQ6)bwhR8;w=vuWUo+;Z} zR%ItzPq-y5;{xT&)>HH_3-$3Hn;--(urx{*TdoA@Y{#u7pI3B z;QZP+vlI-85_N;3K;1W#y8W4{s>&WT2--Awx;6$$&?3|*z2wiGy%Pw6E-Ksokw#S~ zRngTI^U>*!)F{EC7kyjTHKWLkSI%?+g(L*Jka*IKrFXp;ISo-+!KLJzrfIO_9OkCt z^F|1gFzB?6N;>&?=#k@e=s2p9fe8>V<5;ivd){XUAUMqi?FH8pWq+WUejB|fL+ctIqU2q@t6efWBVYZn%Bdvwzp8_=tU&mc|^oTf>grZ2_pJkUuUHr^P~iHC>D%}yS(I-dy! z1i`?;vhBw&U%&kEkE1b2h}-4PojZ?zcleJV`UEKfFa`N3$%0@}(UEbukay>Np==O4 zn9R$5es2F7uv6LYnp$97O>*ws(<@ga&&ad5YfZ9Xv+=uRd|eFqk3hZfjO0dyY`_GH znRdz+x2>O->xqJB9A*5kcA1{02~bzXz=#o}Mii|_%$SkY*efIf+Ui-g9#CQ{(GJ6V zMa{qYwYLsm5n#-Cm75BvUg(U9?3PCR!*c`y-x5Ae(A_{aYqPfzHuTmIdxJ9qB*kN^7b zKXxfH2<8NW{A2{#fFL$e5IfLWN>DwMZ51fSpbPffsmFWxKGs-&5HN#w%Hf#n?HW@-sHz} zN#P7em9*a-)QuvhEz@;jwCjlkkr2>l#hXD9!K7Ynzc`gHOb*i^U_%@X&;W^-sDOZa zc4;Rwv6CPS+Tm1l#~`FDf2*rFD`u`M+qbv{T_Ac!L>f!Vq5$J+nho}%vUL$w-$&U7 z)p|E6LwspndQqc|$`-FmyT~dnLrVb@Jm3VzJE1|>*K8LdL7En~--aMOM3zi7Q4qv! zap%q*|LI@;ZFBbYJrD?kpIF)Me5UeXC!ScQn9-GC3TS>UKe;rRary5AC@M%NN(Dqt zi1^+}4glj@Vg(s8coqd-qilP2_V}uF!0wvuYl&}>t7-F(E4j05m;tl#TQh^)5Uv_0 z0%9B0na3*6Q_IcYIqIt-tquBF*bIs)n5G8-o@;;kJwX8G%3^g{!#(Lq3eJX0T$8u} z#V*<nEq_RgKVvi;{x%2wWy zAbwi*+b4Nf0t(yHA-RM%K3tLj-h^{X(tEnkPI?as#+4ZFWSpMqpDSsLA58TQ_PI^s z6!n=F1yDuPG>NbA>2`hWwF6nqsCqL~7m(Lc(Wc#t$ZL@AO2j_=9wDLL;`k~B^e)BIAH-rMzz9f-cBBa`U?Bvam z+o!$XTb*v!|Fn0=t9+O6x;u7%TCzREfe z6fyOq

diff --git a/content/zh/author/volcano/avatar.png b/content/zh/author/volcano/avatar.png deleted file mode 100644 index 7c4c8a734321bc8ebd0e5f1da39d525c85288fac..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 5677 zcma)AXH-+!w+|%{dQqtX0?L3CK}Km22pvS)D8-00X+a2Oq)G1}B`5+q5D|%jBS?!B z=|vL+=}5U0=?H{QKoEEreE+xJhxg&Fb=O&E|IY4b?|s+3iDoAHY|H}85D0|L&_L%e z2!uu!+%Llz!1bDV(-s7BZre~t^Hw0u%J7hnc;m*W4R#t+qf3XAWF$HG$Cuzl-!)jI zs62gKkEjQXl_qEr%8x3z^-jutmyt5v1yKuKC|%lmxO=DY$N521#Y2+D(FQL3*UY4b zZT%y&ms+$wv(B|qJq_zz)cF;Q7}}j!`f|S%e`wy}_wy_5Ki|QiOA-bM!EYXn(7aH2 z4n%ammyz(OicLXkzHUjk*x5o6(MS&i)7q#U#p&}oACoFzQ1^Ix$K^$TsLI_oeP!n? zS1#s(^A|V;lYn!xSGYu`Zr#P}N}BN+AQ-;6S#heWg~Z%^5XzH>x~ob1-0dr9x!+L` z^+tbEwC4VTw-1pVnGNb(f-pHHPf?h)3LALo4YZ3F6LJ`8-iDVGED~beD-S2(e5>^NO#G|?U5U`QG43L z`9F?NxCPTtsvd9eYKDkIe@JpL#jdE4x%#>uJ zu6WcR(!^AD-;@d6ufvYf5c05N=V6|?1=jnW%k+Nv3iN3V-4vqe(W{Fp*B%~I+In9> z)elrP#}im-W?RRq`WXV*3cBk}%u<|T2qBv);5=yq)!y5C;3bjDb2a41FM{})tH6oBNtJ9NpiPou5Md?|jjU;TCbtk#S1uHKfeB;#LOr7TB;e#NgG*O9 zBwHNDdMn0xQ+YyX6i{*Qpa&{Z<)IE0bv;tMuNvlJyB<~vlt5nH(`B|l;uE% zJinAZ^ZjpgLg4jpKu$4Q5IgXBB&dS;aNo674lcV-b&jDOOo-aPcQEU}B)=5jeiSkA z9#3iTAub}bz%nsB>!i}^UwM;G#Nh?(Q}6rz@B(`$+l4Ntq|)?G>Jrfw*sw-4XZ?+7vXiE2fDG$Sa;R@dqwd^Rv@qfjF`ZScYb}B(maGfe{klZJn!tUg**t3Yof|loe^

qgnVPnJ?cHFF zRGz~{Db2eECu|+zt{EVk(J340M4S8Y3sH|6JO#|ihDFpz1<>pimoQ2-*{z$ivZu>` z`h4Ajs-}>RCWFAp3(?Z?kN7>9oY4C-1I=fhBKRr%w16y-h?C!(xqp-Nn^Hkog(Rgkp?9y7Zq%OhQ1am z!D0xIa>t)yje)-ZFEb!ZfTk+4T_oqryg<%ETtP+h&F*>)2; zl4iIx6zhrVgNtQ2KIxY8W6?6+o!;WLR48K`6@pv;UOz6F`mF>DI_V{z2DLr11vbO) z>jX!1Qdh%`0(w?cyVs9epR3ql@+qL=+GDnH&06VTfg=9a48;jyL`8kDHQ3UBD5D!2 z63t$Qp=fU-wD^y+wMXfkDwV=2j6oGqGkZs^{N9D#UgEA4nu*eSpKZ^j#_`5bR?9n^ zKo>PnfAS5O8riMvY%rk5+zD)zf=p?=L}`FB%)kdX(e?%JD=FXWX65db+adK2u+pB; zD+mK@0Nw04AcL|s!eA|Aa_4`zx<55Pv~#Qttv&3OQxB`8(_aoy0t>&HoBU-M+b@hT zb+fAM7JcD3#lJ+~O*bl%XVOCLchd;Oc+I%gN2)ew(@;3lM_B;VH;3P=gd*~H4gd)M z5{jRe3{kDFM>!z@(g&adx5$e?@)hy;t#6c_Tut;;xk!P83$2# z{C>a|QI4lDQ2q#ssf3c*8Q$qq)YU61@cDE0)uYkQ25i46=OtVfjluvsXo$hhH8_$? zg$}P~HZE2|Mw;zE6gHDc{*`|d;jC8R2zmS8)Q!U*im14KCUC&9_3NF;I1jJphPLt1 z_}(vt_jrbdoRpPx)65-*@7~U^{87eGxa!_auzfA&Kn*xdv5Z*RK4yZFQ;)5TN3dh; zX~V0|#8odkr@}RG>M0ft+9IMna++9iqaJsCUo4L6 zcsUMEsM|<@h8qA_Qzsm&Sw|@x2KdSfAviIcEvvZ~;KBUWl%|!Be=t zd;?t7;vOpR#}WI%2@^o^z?pRd6n&PXhXrQHi)01@{(~#5*3&gOrhJe(g3A+5Z4eE@ z?)oWv1jyUCPV#y+HnUd|lGYT-QOIEyF8U6FvG zNbWnj>mEvZEC9>c?Z(>18P|ix@S^wMA&wCgkF}jTt%H1HBw%#_VH>K201F+dz+_QR zST_}GDoxZV+6f1^6CT52DgCDttL?bxB|(HF;R(v;$)x zb^9mLIf8XV3@#PO7f;Jk*E-l6!9GNJxCTd4*?3UPq$mCQZeUFsb0mOK^t@FH+1+*6E}~1PG8^U{LH4f< z0Fy72KlfQ<1TQeEBUyEN{T|)K zPoFJN{q)wv=ii(7rCSB6gKsY#(7p?9Z13p$y1mN=2Q)DKDtJQwtaI|M_Ka6;*Gi4w zatjmlo`6ayiko61Plx-ZR;Z1w<8TJh#oY*H6MaZEi~|F+*DWn4>C7 za%XNZ>uk%zh;OUpc;RE#um?WvudlMqQ+Q&-I-?lH8I90q+{h^%h{EH<@}f^oIXxb? z(6Neg=Up?B2-z|3ugVltMI|)R5_`ut9&Uyli2;3ZbvQF)*p<|j=G9#aXpUy;TOvCJZn-5&++ua>a5h8J&|C3cD1vp1Xw*ul z9A>_DhA{Viq0XQtc=pq;#Uh272~ke&;=z|Ci!U^)uzK~Y{<0w-$Vy)`ZtZKyZO>G2 z`G-iK9#Y^t7~CTCs8M@9u$uTt6NArqMNa)QdOj>0>dg2YA=~N`F=bh>6~4P>)bU!E za2IVq>%GgH_mL?7%RMM)ur?G^L$pCNPy1pV_69dMYZMOpd_ib^KtFh=&M0Z;scGKQ z;oDQ}W%YVt%j?)FtB#I;CvxRvvoeiXcoEt@HN?G)<-g#0FDFrhU%QAT9*XYkxhd)Tox`DfCX0ns!I}^{bymVew z1q{6tZ9xgKr3rW=mgsj2m!Xxoa5`MFg|PMDCcs$yVIkd~?Ty9OInE|gWxFCPE6018 zAy$fm=%=Co{6YKG4P%SUmRXC z#nCko<+;{cTD+3P1YXk^M*h%f%RQUeGQ&>1PoG5Z!8#y+@&N@xG6wp?Tpz1uR45k%jlcjI+#HY%T$F!g#eI7`d5jLe`Q>faP>xMa60t@+czI8)Bn z*YO4|5?k>G_nSm%rNcP9LR3NyDFlltM6N_}Hjz(8rhBqv=%ORpMNG()d~^ZhIek1+ zlYbZ6CxEo;Fbu5C_G%@it_&nwnBicapP$G44JBsW$xZ)>zcKFEZDb|y$=-vDFLEYM zJ#M}X?{(#PyHc7MPn>qBPlo?u9{_#`5)b$tCD$)bDU}zy8^{^iz3E~^rDHoZxc_o(PYZafQxf-(- z?}B^>yKRx&eAJ?}Xo+Lx3p?i<6;1qOsi(})@lNUe-F%oB`7f_*_ic^~I`$0_M5)go znAMp5L{VRo$~pEc&6G@6y9bP4m~Jx#agYlgS&T*2!?rV!GM~q!mbzMoCa$C3UTvG( zRq1n=Z=by$FY2J^+M&0qSM#Q1un)F`j)HXZJE|9nWMO&8ky)m*Llg~g{t(E6*4Bn&udujQ|x?9FsJDTMVOn*(I3HSxRA~p5R=1&!R^yG-F1m&)4MIgoV9#8c%yX z&{0#x=3(vaNsobx7Hbj&i^jD0O-8oy%Z4l!Bp4}#5##es?wfgJJfe7q9^*r`W~$Vz z(RY$Bi1j3M^-asbRTbOu6)jsygirRbwzk5h9rpiD;OpVpQY(WV&Ua2aQHE^$RKw%h zXPIlPEgbF}h_&*_D^GiGwvFvR;mJ%O|6LbN6niRQJZ^|+Oe9vCWWSahl52h0WR9C9 zRtsgv7&BwlSrOHL60V9plX4b=o%_|+)4h9PUf#H26);S?_c4i*{o#k?^t0J>c*0;( z&GuKq2Y<3mT_lm=sY9)#w=5<$`T$E-yF{0e(`!S3#`6%$k|-mXzUk!2(a~A=m3`Md zm#0a^<3!$N~4jL!MBpFDz4;4Rn zq4=C{EVu11H_na7$%j?xVJsNkd6f6Albc!aW+%{U{dx5P>;gJJNXIdV4pW9ZGirlu z%Ml1s7`=P`Pi)RPQa&Npvkr|(DbD?n-jBXvf8pudG8shYK+re0%QIK6;`5`4?#o74 z%2&PlnUPud(P4)(Px|RNS{Xbp{{Vi__Wf&Gxx#;lAGm8aCGoC3B~J1E&h1N`oS_Oc z8h_4rRw*yrX``ExHxa_G;K-QLh~bAEFTUbShT=L5jjr`bZHCY-!_&A3*nw74^vX^D zYx33qF62Zkde`CK(IFDss@GIxQ>3`R!gra2rH_S%?d)sgh(s)c|5eW=_Mp!6zFi*8 zH$2N(Pk3zkd*=9uJ}cUvCtvWltup+DL;bP~YxO2{FP4N1!eAebzs#r5$hMYbGgxZ; zWmjfwk2D`JE)+!64#}lTn0wz8%h|IpI^r0OIKC)&_G!}5!w`dm3!FWl3rT6|jYbna z711Gs@l1PRoj;b>JHi(@n*^1x-r~1b^D)@5>W$)rkgyGP)8)@8f|IQCFE@?RO4zZz j>)&Z+_et))_OCC6(RSvdv;mq)m#-5=Kcxmh6PFD^j8iN_wN$QefYc z`Qp@-t6(Y9I37OpE@jtoY?kvj^=j)joP6;qLDG@eshmX~gY>u@@70<<;%b#SZ~#tnNNO%;7`$n;Jzf~YbCGTr)XlR_ZxaeCE3a0SB?mtwyjKw5@_uDlF5ZCNIK?&-Yp zw{fVs$Q4SLZCD_PR`B@tz|T|*Ka_fLj!U$R*C9dt3O-VMQTdH}slULM6zEvCG2rhH zlqDVotoSifuCu#m`VbMYuqRli zs~x~~Cxx1$;zK@NUu+C{YHWdy{;_W@sppnwS_HscL?%-_dvopC<>|9z+3DLMN17wA zsh^MX6hb(Z;BS3np7yL^hB2aFQid8HZ@O3i{?+V0bF+Dz;)`g#C&cIY7t{R(q8aAn z+6}aJe+OCeFMm~(&elzhd8t_|e$psnbeVvTz?bW3whjhccZ1*D`>+F;o%vB9U!x{| z-Nl#O#a#znDca)bK_7K6udEBQo=QLJ2!FH=eQ)-!y8_L_Jdju^^fKq`@~oy;xI8q5 z8h&#&xlh3o<#D4p502MRK@%R?#_PxQ~V_WDC5Av--kRs82XhLic7qBkUg&T03 zunre|)@D1C{`WWQKm!BO*ggP%5@ZDKnYwphhuHmz2Y*ocmBHUrDA|R-X{~xYNAgIc z0GJZGc`ya%OkGwRqTiMh^B+M(FpPF$L(yv1pFn?jCld(H4n$Mzne!FOFCB(DEap<; zyTf^Z3eDFM0Yv9n+LnFBmH}xRjaHX&^Re9)R?TcF%f#MPA}`^LueB(XyH2f(U3s90 z{y+iiCJ7vLdb%-vCAP0r69&Qv-IH|Lj!@3-gJJd^4-SaJr`#y5L@T6G{Xb282~F=S zhXYoRI^5U}BR%XGymi>>!`l@xw-%Zl4Ok?R^5@hxUyAE@%%pj4#<~bktr};&$jm8=!hq%q#z|2|lvC~7$hp6mJNegG4;hvQ$-$w~ar>1tT_?@~b) zrdafZn}?OmPDkO_!C;+j5wXrYj%{+$r>Zs7X@sP!8bFx}^-6IXy>~ty%oS7-<5eEt zECg;rx(NTAJ?tw2QTVA3vLtj-WETkG`Q&4aXxq>I!Iww|%cu3K1Stb5`K9!B&xf=N zB;4kkI&Fo-uFmItVx1F1W0GunP|I21QgHM$!2=8Eubp+J=WG;PX*Lw#d|QxTgWqna zp`=VSPan>vN1Sm+r6w^B+O|IQOfr)?SDv48yhb6aLBclQSHk8d?_k3aQ|KJU-G-9_ zyvY?g#)vFK!c--6pn67zRyqM{|BMwSSu_sG7kctqD@r66aLJ75wq)*GCg@y+;%FOf z=Z$t{XXDm0fh(<AySA2+u5NO(-jmyFnOt|A>mio_jPw!XX^wr>i+_u`oS*oaflzV@v$C)`+D7xDq%UY7*wdwa&pN2y@s=U2pjiC|J%mr9MRVB(7CO$Umm_TLIik*kU=mpBHwyP$x5lrw8x7Dn*AHI zPEwkaE6uR^EDh#Wni@$bDNQ7zkus{s?WL2T?`eaDg9~} zEl@7B2{d%FrMe`s@*c*-;?b%GHW7k}EDiX0)U=%$1VgH1$*%;emmmeJdp7tUP$Xs{m^pEfHwS8;^12{q!omd~F#I>Zc95L6Mj z(U(=%)TqMSCw1#oK$XV|D%J4@zny+ZGx2)eylaxIz4ztPH*6-OhH^!W1b+-X4#$P+ z%IyL}T5)SZvnOi$Ra?%3scS_?%^<^_JiM|3;`evD(Sw0rF&Ds{mj^$+aCWS{lDp8# z%&rQz(#;>1&5+^5*1aPF6%x;Fvs~6PGMzupsUf0nQF5(<3M?*l&UQCF4NxRSYt0PL z7xg2fVWL4%8As9r(%%^tKW6 z*(~h-)0nm>;sf)4BMP}4Xi9sLshYHRR$GErV(!pUV(Ga4`%%=I{gy2A6#!J6yE;9_ zs-pM9+&XKzujakVv)W~8^|HRZ1WN@pzN_pl@LRm*8}?k*#bQR8EAlx$z99PRcI~Oe zrZCVBy3TLch+8tkf}lJ|HK9WMI`xm4m#gcM<936#3Zp6Zg(Kx1H-z{Ms0Thh^myLX z5@c}yS6YLg!i|6_C#yD9QhOLvxX4x10skr_DSoMNLjxfYf2N|`Ggv+RMah;p7>DJ? zi8;n0F)|6$U09a6avZ6&aVhcBxi9;a1RV`J7i<+Q)V36{Y?eLB_+X_hAdY zEDp4ns+PY*kv^UnTzLWnB)bRbrkM)J>QGVcqO{E73ViXFPKgwEG3}|vn%4!Hc>%Ok zS|65rPhW1}z)6KpeCrl93L0A2>%tAVTfifsX=Elkj0Dktd>41 z!|BaPFc^O|k~Z!0;A-$u3w}gjC9lfhd7l!B{iTeQSqB}CSa0boCz;UhGzyOT4=S!p z!V@dE3XHEIX*14L*Q6EpKQx2Y%43D1F#-OJT+jV+!+mgwQRFnK~|`aZeF znk%_({L0={@Q z39u_y87>?pKlNlZ^7HmfND&KNbHT#KzyuWkl?ih@k0PsH%o{g>5elJngZUs!p`von z!3{lv;`1enteIZkn(S-{hHd|G0I&DJjcn^hRAlvdjUO&b(G=*M7;6FswI9icR=xpU za=SyeHDr5?ahHK6vV!uY)zP;oy5usQnN0%h?>5(mB#tijj{k=53P#PzoVwjeS@Dxe zuWj(3^Oe~$G-`F`N9&GSPMs4Uah@Zj*G?Ji*56p&2A8Hn+n(54d*OUs6GtPYr1bbc z*psJ$f-DV!Zv9RLb6^LuuY?9I>5W86ALpRvITYEujGc^ETgh>?AwnCvd9GtVoW_(R zWMK=X7W6hqKz|ej742Bb_Cy4Sjv=EZ3(t*Z@WJXgLNK;#uz4_#QS42HJX~_sCf^ze zG$qvbxbt5yPyArIEuAl{nYM^d@6y6fe^hfoiV;s28D`d@V!ljT^m0Tr%e=N^_2VY| z?DVEtKW$| zc2b3Zk2d~|1-wHFI`k3BzVf4_wkN>;nH-h}GYU+}T|ntlZ{Jxkfj^;Y52J1e7FdGV z7AuiURs9jUnsGZj!WWP%=>ozCBX#gVtp58$H*fEok!806RAQ<)C?V>60xv&qF^Vy4 zUJH*3Vnp!qI|tTJ=_>dH-Rd8;9^}N^;cd#O&Lf8!SU;9lyt@wtFd~rbR*fOU`N*Fu zkq|`Cgn3<;pnrVa;+)h}ky9)k5tygZ+$R>jlI1ejwjp^R=X?^ea69lx`#&c{(2Lm_ zgt)ZD+Ifh-_pP$Z{vEk`+4*#ur7qRfg&MPHKlFW{<@8@0*Gh)z>f(MZYn;NU383}) z7O>QRi$HFjQayJlFvf&SaB%qB{vB|=EO#+sO@}(+8q>1qHy_TwlUnC{MM>@NkfUv= z`~@(q{hu#ec+m>qSpKJB7jqUN`G;bpbd&0udIzq4F(9H7Y+V-BweD?f!75Lv+`2(~ z(Y*(}05;)lnmBc677=Iq0f!ui*`+g5MUZ_?qe8;}@SiS&GeZ5u(WtQw(g^ zY|Ml)zUUwP=1liij8H!EqmxGz)iHr8&)sp2q^ zR&f2sMcXQOd);C}bWj};e2lnwwa}y8!$9TQP_n_=~6;lIsOAL))uuzS-W+#p- z<4-7-_~NDtjSDH29Jg9so%RD6alw_{bBRr?c5muj=$Pmf7dxT9hBFG1_<3HJ^-Cyh z%$P6DUSiFuiE~cSCr+=h0!$}{Gul=(zDnI)Vo?YdMJv%A+*W-72D;fMf19( zkkr@PiZuDl2j?b+F(8kqz+JCn*CH8*+e4T$k diff --git a/content/zh/blog/1.4 release.md b/content/zh/blog/1.4 release.md deleted file mode 100644 index f40fcee2..00000000 --- a/content/zh/blog/1.4 release.md +++ /dev/null @@ -1,55 +0,0 @@ -+++ -title = "Volcano v1.4.0-Beta发布，支持NUMA-Aware等多个重要特性" -description = "Volcano v1.4.0-Beta 新特性介绍" -subtitle = "" - -date = 2021-08-31 -lastmod = 2021-09-13 -datemonth = "Sep" -dateyear = "2021" -dateday = 13 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Thor-wl"] - -tags = ["Tutorials"] -summary = "Volcano v1.4.0-Beta 新特性介绍" - -# Add menu entry to sidebar. -linktitle = "Volcano v1.4.0-Beta发布，支持NUMA-Aware等多个重要特性" -[menu.posts] -parent = "tutorials" -weight = 12 -+++ - ->本文2021年9月6日首发于容器魔方微信公众号，原文链接[Volcano v1.4.0-Beta发布，支持NUMA-Aware等多个重要特性](https://mp.weixin.qq.com/s/S5JAQI0uLoTEx0lvYDXM4Q) - -北京时间9月4日，CNCF首个批量计算项目Volcano发布v1.4.0-Beta版本。该版本发布了“支持GPU节点多维度资源比例分片”、“支持NUMA-Aware”、“支持多调度器混合部署”等多个重要特性，并在稳定性方面具有显著提升。 - -## 支持GPU节点多维度资源比例分片 - -“支持GPU节点多维度资源比例分片”是该版本具有重大意义的特性之一，主要用于解决GPU节点因CPU等其他维度资源过度使用引起GPU作业饥饿但GPU资源空闲浪费的问题。__该特性由Volcano社区合作伙伴中科类脑贡献__。在传统调度器中，GPU等稀缺资源在进行分配时与CPU等资源离散考虑，即CPU型作业可直接分配到GPU节点而不会考虑GPU作业的CPU、内存需求，不会为其预留资源。在该特性中，允许用户设置一个主导型资源（通常设置为GPU），并可为它配置配套资源维度的预留比例（如GPU:CPU:Memory=1:4:32）。调度器在工作时将会时刻保持GPU节点上GPU、CPU、Memory的空闲资源比例不低于该设定值，因此任何时刻符合该比例需求的GPU作业均可调度到该节点，而不会引起GPU浪费。这一方法较业界其他解决方案，如GPU节点分配独立调度器、CPU型作业强制不允许调度到GPU节点等，更有利于提高节点资源利用率，使用也更加灵活。 - -特性设计和使用方式请参考：https://github.com/volcano-sh/volcano/blob/master/docs/design/proportional.md - - - -## 支持CPU NUMA-Aware - -“支持CPU NUMA-Aware”是该版本的另一个重大特性。对于AI、大数据等计算密集型作业来说，启用NUMA特性将显著提升运算效率。该特性实现了CPU NUMA感知调度。用户可通过配置NUMA策略决定是否为工作负载启用NUMA。调度器将根据诉求选择满足NUMA要求的节点。 - -特性设计和使用方式请参考：https://github.com/volcano-sh/volcano/blob/master/docs/design/numa-aware.md - - -## 支持多调度器混合部署 - -“支持多调度器混合部署”特性允许用户在Kubernetes集群中同时部署不同类型的调度器，并合理进行调度分工。最常见的工作方式为default-scheduler和Volcano混合部署。用户可以将Kubernetes原生资源类型，如deployment、statefulset等，分配给default-scheduler进行调度；将Volcano Job、TF Job、Spark Job等高性能计算负载分配给Volcano进行调度。该方案不仅能充分发挥各调度器的优势，也能很好的降低单调度器的并发压力。 - -特性设计和使用方式请参考：https://github.com/volcano-sh/volcano/blob/master/docs/design/multi-scheduler.md - - -除上述特性，v1.4.0-Beta还新增了压力测试自动化框架，并修复了因资源比较函数鲁棒性引入的多个bug等。社区正在进行v1.5的路标特性征集，已征集到的特性包括集群资源监控、支持层级队列、Spark集成能力增强、支持Task依赖等，欢迎有兴趣的开发者issue反馈。 - - diff --git a/content/zh/blog/ING_case.md b/content/zh/blog/ING_case.md deleted file mode 100644 index 37c887f7..00000000 --- a/content/zh/blog/ING_case.md +++ /dev/null @@ -1,125 +0,0 @@ -+++ -title = "ING基于volcano的大数据分析平台应用实践" -description = "volcano在ING大数据分析平台的落地实践过程" -subtitle = "" - -date = 2022-12-28 -lastmod = 2022-12-28 -datemonth = "Dec" -dateyear = "2022" -dateday = 28 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "ING基于volcano的大数据分析平台的应用实践过程" - -# Add menu entry to sidebar. -linktitle = "ING基于volcano的大数据分析平台应用实践" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ - ->2022年10月26日，ING集团在北美KubeCon发表了《Efficient Scheduling Of High Performance Batch Computing For Analytics Workloads With Volcano - Krzysztof Adamski & Tinco Boekestijn, ING》主题演讲，重点介绍了云原生批量计算项目Volcano如何在数据管理平台中为大数据分析作业提供高性能调度工作。详情参见：[KubeCon + CloudNativeCon North America](https://events.linuxfoundation.org/archive/2022/kubecon-cloudnativecon-north-america/program/schedule/) - -## ING背景介绍 - -ING集团（荷兰语：Internationale Nederlanden Groep），亦名荷兰国际集团，是一个国际金融服务私营企业，成立于1991年，由荷兰最大的保险公司Nationale-Nederlanden，与荷兰的第三大银行NMB PostBank Group合并而成。 - -ING集团的服务遍及全球40多个国家，核心业务是银行、保险及资产管理等。ING集团的全球职员大约56,000人，顾客5320万人，包括自然人、家庭，企业、政府及其他等，例如基金组织。 - - -## 业务背景介绍 - -在银行行业有许多法规和限制，如：监管要求在全球范围内各不相同、数据孤岛-全局和本地限制、数据安全、合规创新等，想要快速引入新技术不是一件容易的事情，为此，ING布局符合自身产业的DAP平台（Data Analytics Platform），为全球50%的ING员工提供安全的、自助的端到端分析能力，帮助员工在数据平台之上构建并解决业务问题。 - -{{

}} - -2013年开始我们有了数据平台的概念，2018年通过引入云原生技术打造新一代基础设施平台，从那时起，平台需求有了稳定的增长，采用率也在持续提升，目前数据索引平台上的项目已超过400个。 -我们所构建的平台目标是在高度安全的自助服务平台中完成所有分析需求，并且具备以下特点： - -- 开源工具模型 -- 强大的计算能力 -- 严格的安全和合规措施 -- 所有的分析集中在同一个平台 -- 满足全球和本地需求 - - -## 挑战与方案 -{{
}} - -目前我们在由传统的Hadoop平台向kubernetes过渡，但是对于作业管理和多框架支持方面还存在一些挑战，如下： - -- Job的管理 - - Pod级调度，无法感知上层应用 - - 缺乏细粒度的生命周期管理 - - 缺乏任务依赖关系，作业依赖关系 -- 调度 - - 缺少基于作业的调度，如：排序、优先级、抢占、公平调度、资源预定等 - - 缺少足够的高级调度算法，如：CPU拓扑、任务拓扑、IO-Awareness，回填等 - - 缺少对作业、队列、命名空间之间资源共享机制的支持 -- 多框架支持 - - 对Tensorflow、Pytorch等框架的支持不足 - - 对每个框架部署（资源规划、共享）等管理比较复杂 - - 利用kubernetes来管理应用服务（无状态应用、甚至是有状态应用）是非常方便的，但是对于批量计算任务的调度管理不如yarn友好，同样yarn也存在一些限制，比如对新框架的支持不够完善，比如TensorFlow、Pytorch等，为此，我们也在寻找新的解决方案。 - -__Kubernetes + Hadoop__ -{{
}} -在我们之前的集群管理上，会把Hadoop和Kubernetes的调度分开，基本上所有的spark作业都会运行在Hadoop集群中，其他的一些任务和算法会运行在Kubernetes集群，我们的目标是希望所有的任务全部运行在Kubernetes集群，这样管理起来会更简单。 - -{{
}} -Kubernetes和YARN共同工作时，由于Kubernetes和Hadoop资源是静态划分的，在正常办公时间，Hadoop应用和Kubernetes各自使用自身分配资源，即便spark任务压力大也无法借用更多资源。夜晚时间，集群中仅有批处理任务，Kubernetes资源全部空闲，却无法分配给Hadoop进行有效利用，对于调度平台来讲，这不是一种最佳的资源分配方式。 - - -__Kubernetes with Volcano__ -{{
}} -使用kubernetes管理整个集群，通过Volcano进行spark任务调度，此时不需要再对资源做静态划分，集群资源可根据Pod、Batch、Interactive任务的优先级、资源压力等进行动态调整，集群整体资源利用率得到极大提升。 - -比如在正常办公时间内，常规服务应用资源空闲的情况下，Batch和Interactive应用资源需求增多时，可以暂时借用常规服务的资源；在假期和夜晚休息时，Batch业务可以使用集群所有资源进行数据计算，集群资源利用率得到极大提升。 - -{{
}} -volcano是专为kubernetes而生的批处理调度引擎，其提供了以下能力： - -- 加权优先级的作业队列 -- 如果集群具有备用容量，可提交超过队列资源限制的任务 -- 当更多的pod被调度时，具备抢占能力 -- 丰富可配置的工作负载调度策略 -- 兼容YARN的调度能力 - -Volcano的引入，补齐了kubernetes平台对批处理作业的调度管理能力，并且自Apache Spark 3.3版本以来，Volcano被作为Spark on Kubernetes的默认batch调度器，安装使用更方便。 - -## 业务常用特性 -__冗余与局部亲和__ -{{
}} -Volcano保留kubernetes中pod级别的亲和性反亲和性策略配置，并增加了task级别的亲和性和反亲和性策略 - -{{
}} -DRF调度算法的全称是Dominant Resource Fairness，是基于容器组Dominant Resource的调度算法。volcano-scheduler观察每个Job请求的主导资源，并将其作为对集群资源使用的一种度量，根据Job的主导资源，计算Job的share值，在调度的过程中，具有较低share值的Job将具有更高的调度优先级。 - -比如集群资源总量为CPU：18C，Memory：72GB，两个用户分别是User1和User2，每个User分配1个队列，在提交作业时会根据主导资源计算job的调度优先级。 - -- User1: CPU share值为 6/18=0.33，Memory share值为 24 / 72 = 0.33，最终share值为0.33 -- User2：CPU share值为 12/18=0.67，Memory share值为 24 / 72 = 0.33，最终share值为0.67 - -DRF策略在任务调度时，优先分配share值较低的Job，即User1所申请的资源。 - -集群内队列资源可以通过配置权重值进行划分，但是当本队列提交任务超出队列分配的资源，并且其他队列存在资源空闲时，可以进行队列间资源共享。即User2在使用完本队列CPU资源后，可以使用User1队列内的空闲CPU资源。当User1队列提交新任务需要CPU资源时，将会触发抢占动作，回收User1被其他队列借用的资源。 - -__避免资源匮乏__ -{{
}} -在使用过程中，需要避免批量计算任务与自有服务出现资源抢占与冲突的问题。比如：我们集群中有两个可用节点，集群中需要部署一个统一的服务层对外提供服务，比如Presto，或者类似Alluxio的缓存服务。但是在批量计算调度时，集群的资源空间有可能全部被占用，我们将无法完成自有服务的部署或升级，为此我们增加了空间可用系数相关配置，为集群预留一些备用空间，用于自有服务的部署使用。 - -__DRF 仪表盘__ -{{
}} -我们根据volcano的监控数据做了一个drf调度的仪表盘，在不同层次显示更细粒度的调度信息。在业务集群中，我们有一个队列存放交互式用户的任务，另有队列存放平台运行的所有重大项目的计算任务，我们可以为重大项目队列提供一定的资源倾斜，但是此时对交互式用户的任务将不会太友好。 - -目前我们正在考虑增加集群高峰时段展示的功能，为用户提供更多的集群使用状态和压力等信息，在自助服务平台用户视角来看，用户按照集群的繁忙程度选择自己任务的开始时间，这样可以避免后台复杂的配置就可以获得高性能的运算体验。 -{{
}} - -## 总结 -volcano对批处理任务调度做了很好的抽象，使我们在kubernetes平台能够获得更高的调度性能，后面我们也会将开发的功能逐步回合社区，比如：DRF Dashboard、在每个节点添加空闲空间、自动队列管理、更多的Prometheus监控指标、Grafana仪表盘更新、kube-state-metrics更新和集群角色限制等。 - diff --git a/content/zh/blog/Quick-Start-Volcano.md b/content/zh/blog/Quick-Start-Volcano.md deleted file mode 100644 index 186e1ebd..00000000 --- a/content/zh/blog/Quick-Start-Volcano.md +++ /dev/null @@ -1,76 +0,0 @@ -+++ -title = "Volcano快速上手指南" -description = "在Kubernetes集群中快速上手Volcano" -subtitle ="" - -date = 2019-03-28 -lastmod = 2020-09-07 -datemonth = "Sep" -dateyear = "2020" -dateday = 07 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Volcano"] - -tags = ["Tutorials"] -summary = "Volcano快速上手指南" - -# Add menu entry to sidebar. -linktitle = "Volcano快速上手指南" -[menu.posts] - parent = "tutorials" - weight = 1 -+++ -# 快速上手 -上手Volcano最容易的方式是使用Helm。 -### 准备 -首先，将代码库克隆到本地。 -```html -# mkdir -p $GOPATH/src/volcano.sh/ -# cd $GOPATH/src/volcano.sh/ -# git clone https://github.com/volcano-sh/volcano.git -``` -### 1. Volcano镜像 -可用的官方镜像请参考： [DockerHub](https://hub.docker.com/u/volcanosh), 您也可以在本地自己制作镜像： -```html -cd $GOPATH/src/volcano.sh/volcano -make images -## Verify your images -# docker images -REPOSITORY TAG IMAGE ID CREATED SIZE -volcanosh/vk-admission latest a83338506638 8 seconds ago 41.4MB -volcanosh/vk-scheduler latest faa3c2a25ac3 9 seconds ago 49.6MB -volcanosh/vk-controllers latest 7b11606ebfb8 10 seconds ago 44.2MB -``` -**请注意**: 您需要确保镜像被正确的加载到您的Kubernetes集群。举个例子，如果您使用的是[kind cluster](https://github.com/kubernetes-sigs/kind) ， -为每个镜像执行命令```kind load docker-image : ```。 -### 2. Helm charts - -最后，安装helm chart -```shell -helm install installer/chart --namespace --name -For eg : -helm install installer/chart --namespace volcano-trial --name volcano-trial -``` - -运行以下命令验证安装是否成功: -```shell -#1. 验证Pods是否正常运行 - -# kubectl get pods --namespace -NAME READY STATUS RESTARTS AGE --admission-84fd9b9dd8-9trxn 1/1 Running 0 43s --controllers-75dcc8ff89-42v6r 1/1 Running 0 43s --scheduler-b94cdb867-89pm2 1/1 Running 0 43s ---admission-init-qbtmb 0/1 Completed 0 43s - -#2. 验证Services -# kubectl get services --namespace -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE --admission-service ClusterIP 10.105.78.53 443/TCP 91s -``` -#### 您也可以观察视频学习如何部署Volcano - - diff --git a/content/zh/blog/Volcano-1.10.0-release.md b/content/zh/blog/Volcano-1.10.0-release.md deleted file mode 100644 index a56c49aa..00000000 --- a/content/zh/blog/Volcano-1.10.0-release.md +++ /dev/null @@ -1,235 +0,0 @@ -+++ -title = "Volcano v1.10.0正式发布" -description = "新增特性：新增队列优先级设置策略、支持细粒度的GPU资源共享与回收、支持Pod Scheduling Readiness调度、支持Sidecar container调度、增强vcctl命令行工具功能、Volcano支持Kubernetes v1.30、增强Volcano安全性、优化Volcano性能、提升GPU监控功能、优化helm chart包安装升级流程等" -subtitle = "" - -date = 2024-09-29 -lastmod = 2024-09-29 -datemonth = "Sep" -dateyear = "2024" -dateday = 29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "新增特性：新增队列优先级设置策略、支持细粒度的GPU资源共享与回收、支持Pod Scheduling Readiness调度、支持Sidecar container调度、增强vcctl命令行工具功能、Volcano支持Kubernetes v1.30、增强Volcano安全性、优化Volcano性能、提升GPU监控功能、优化helm chart包安装升级流程等" - -# Add menu entry to sidebar. -linktitle = "Volcano v1.10.0正式发布" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ - - - -北京时间2024年9月19日，Volcano社区v1.10.0版本正式发布，此次版本增加了以下新特性： - -- **新增队列优先级设置策略** - -- **支持细粒度的GPU资源共享与回收** - -- **支持Pod Scheduling Readiness调度** - -- **支持Sidecar container调度** - -- **增强vcctl命令行工具功能** - -- **Volcano支持Kubernetes v1.30** - -- **增强Volcano安全性** - -- **优化Volcano性能** - -- **提升GPU监控功能** - -- **优化helm chart包安装升级流程** - -{{
}} -Volcano是业界首个云原生批量计算项目，于2019年6月在上海 KubeCon 正式开源，并在2020年4月成为 CNCF 官方项目。2022年4月，Volcano 正式晋级为CNCF 孵化项目。Volcano 社区开源以来，受到众多开发者、合作伙伴和用户的认可和支持。截至目前，累计有800+全球开发者参与社区贡献。 - -## Volcano v1.10.0 关键特性介绍 - -### 新增队列优先级设置策略 - -在传统的大数据处理场景下，用户可以直接设置队列优先级来控制作业的调度顺序，为了更好的帮助用户从Hadoop/Yarn迁移到云原生平台，Volcano也支持了在队列层面直接设置优先级，降低大数据用户的迁移成本，提升用户体验和资源利用效率。 - -队列是Volcano中的一种基本资源，不同队列有着优先级区分，在默认情况下，队列的优先级是由队列的`share`值决定的，share值是由队列中已分配的资源量除以队列的总容量计算得到的，不需要用户手动配置，`share`值越小，则代表队列中已分配的资源比例越小，即队列越不饱和，需要优先分配资源，因此队列的`share`越小，队列的优先级越高，在分配资源时会优先分配给`share`较小的队列，以保证资源分配的公平性。 - -但是在生产环境尤其是大数据处理场景下，用户更希望可以直接设置队列的优先级，从而能更直观的知道不同队列的优先级顺序，由于`share`值是实时计算得到的，因此会根据队列分配资源的饱和程度而实时变化，为了更加直观的表示队列优先级同时支持用户自行配置，Volcano在`share`值的基础上为队列新增了`priority`字段，支持用户配置队列优先级，`priority`越高则表示队列优先级越高，会优先分配资源给高优先级的队列，并且在回收队列资源时会优先回收低优先级队列内的作业。 - -队列优先级定义： - -```go -type QueueSpec struct { -... - // Priority define the priority of queue. Higher values are prioritized for scheduling and considered later during reclamation. - // +optional - Priority int32 `json:"priority,omitempty" protobuf:"bytes,10,opt,name=priority"` -} -``` - -同时为了兼容share值的使用方式，Volcano在计算队列优先级时也会考虑share值，默认情况下用户不设置队列优先级或者队列的优先级相等时，Volcano会再比较队列的share值，此时share越小队列优先级越高。用户可以根据实际场景选择设置不同的优先级策略，即priority和share两种方式。 - -关于队列优先级设计文档，请参考：[Queue Priority](https://github.com/volcano-sh/volcano/blob/master/docs/design/queue-priority.md). - -### 支持细粒度的GPU资源共享与回收 - -Volcano在v1.9版本发布了弹性队列容量capacity调度功能，用户可以直接为队列设置每一维度资源的容量，同时支持基于`deserved`的队列弹性容量调度，实现了更加细粒度的队列资源共享和回收机制。 - -弹性队列容量`capacity`调度的设计文档请参考：[Capacity scheduling Design](https://github.com/volcano-sh/volcano/blob/master/docs/design/capacity-scheduling.md) - -使用指导请参考：[Capacity Plugin User Guide](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_capacity_plugin.md). - -为队列配置每一维度deserved使用样例： - -```yaml -apiVersion: scheduling.volcano.sh/v1beta1 -kind: Queue -metadata: - name: demo-queue -spec: - reclaimable: true - deserved: # set the deserved field. - cpu: 64 - memeory: 128Gi - nvidia.com/a100: 40 - nvidia.com/v100: 80 -``` - -在v1.10版本中，Volcano在弹性队列容量`capacity`的基础上，支持了上报不同型号的GPU资源，NVIDIA默认的`Device Plugin`在上报GPU资源时无法区分GPU型号，统一上报为`nvidia.com/gpu`，AI训推任务无法根据业务特点选择不同型号的GPU，比如A100、T4等型号的GPU，为了解决这一问题，以满足不同类型的AI任务需求，Volcano在`Device Plugin`层面支持上报不同型号的GPU资源到节点，配合`capacity`插件实现更加细粒度的GPU资源共享和回收。 - -关于`Device Plugin`上报不同型号GPU的实现和使用指导，请参考：[GPU Resource Naming](https://github.com/volcano-sh/devices/tree/release-1.1/docs/resource-naming). - -**注意：** - -`capacity`在v1.10.0版本中作为了默认的队列管理插件，`capacity`与`proportion`插件互相冲突，当升级到v1.10.0后，你需要再设置队列的`deserved`字段，以保证队列功能正常工作，具体的使用说明请参考：[Capacity Plugin User Guide](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_capacity_plugin.md). - -`capacity`插件根据用户指定的队列`deserved`值来划分集群资源，而`proportion`插件则根据队列权重动态划分集群资源，用户可以根据实际场景选择使用`capacity`或者`proportion`插件进行队列管理。proportion插件的介绍请参考：[proportion plugin](https://volcano.sh/en/docs/plugins/#proportion). - -#### 支持Pod Scheduling Readiness调度 - -Pod 一旦创建就被认为已准备好进行调度，在 Kube-scheduler 中，它会尽力寻找合适的节点来放置所有Pending的 Pod。然而，在现实情况下，某些 Pod 可能会长时间处于“缺少必要资源”状态，这些 Pod 实际上以不必要的方式干扰调度程序（以及 Cluster AutoScaler 等下游组件）的决策和运行，造成资源浪费等问题。Pod Scheduling Readiness是 Kube-sheduler 的一项新增功能，在Kubernetes v.1.30版本GA，成为了一个稳定特性，它通过设置Pod的schedulingGates字段来控制Pod的调度时机。 - -
{{
}} -Pod SchedulingGates -
- -在前面的版本中，Volcano已集成了K8s默认调度器的所有算法，全面涵盖了Kube-scheduler的原生调度功能。因此，Volcano能够无缝替代Kube-scheduler，作为云原生平台下的统一调度器，支持微服务和AI/大数据工作负载的统一调度。在最新发布的v1.10版本中，Volcano更是引入了Pod Scheduling Readiness调度能力，进一步满足了用户在多样化场景下的调度需求。 - -关于Pod Scheduling Readiness特性的文档，请参考：[Pod Scheduling Readiness | Kubernetes](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/) - -Volcano支持Pod Scheduling Readiness调度的设计文档，请参考：[Proposal for Support of Pod Scheduling Readiness by ykcai-daniel · Pull Request #3581 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3581) - -### 支持Sidecar container调度 - -Sidecar container是一种相对于业务容器而言的辅助容器，通常用来辅助业务容器的运行，比如收集业务容器日志、监控、初始化网络等。 - -在Kubernetes v1.28之前，Sidecar container只是一种概念，并没有单独的API来标识一个容器是否是Sidecar container，Sidecar容器和业务容器处于同等地位，有着相同的生命周期，Kubelet会并发启动所有Sidecar容器和业务容器，这样带来的问题是Sidecar容器可能会在业务容器启动之后才启动，并且在业务容器结束之前先结束，而我们期望的是Sidecar容器先于业务容器启动，并在业务容器结束之后再结束，这样就能保证Sidecar容器收集的日志，监控等信息是完整的。 - -Kubernetes v1.28在API层面支持了Sidecar container，并对init container、Sidecar container、业务container做了统一的生命周期管理，同时调整了Pod的request/limit资源计算方式，该特性在v1.29成为Beta特性。 - -该特性在设计阶段经历了漫长的讨论时间，特性本身并不复杂，主要的考虑点在于兼容旧的使用方式，如果定义一个除了init container、业务容器之外的新的容器类型，会对API有较大的破坏性，同时周边组件适配该特性的话会有较多的侵入式修改，带来很多额外开销，因此Kubernetes社区并没有引入新的容器类型来支持Sidecar container，而是直接复用了init container，通过设置init container的restartPolicy为Always来标识Sidecar container，完美的解决了API兼容性问题和Sidecar容器的生命周期问题。 - -在调度层面，该特性的影响在于Pod申请的request资源计算方式有所变化，因为Sidecar container作为一种特殊的init container是持久运行的，需要将Sidecar container的request值累加到业务容器的request值上，因此需要重新计算init container、Sidecar container和业务容器的资源request值。 - -Volcano调度器在新版本更改了Sidecar container的资源计算方式，支持了Sidecar container的调度，用户可以使用Volcano调度Sidecar container。 - -关于Sidecar container的详细信息，请参考：[Sidecar Containers | Kubernetes](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) - -### 增强vcctl命令行工具功能 - -vcctl是操作Volcano内置CRD资源的一个命令行工具，可以方便的用来查看/删除/暂停/恢复vcjob资源，并支持查看/删除/开启/关闭/更新queue资源。Volcano在新版本对vcctl做了功能增强，新增以下功能： - -- 支持创建/删除/查看/描述`jobflow`和`jobtemplate`资源 - -- 支持查询指定队列里的vcjob - -- 支持通过queue和vcjob过滤查询Pod - -vcctl的详细指导文档，请参考：[`vcctl` Command Line Enhancement](https://github.com/volcano-sh/volcano/blob/master/docs/design/command-line-enhancement.md#new-format-of-volcano-command-line). - -### Volcano支持Kubernetes v1.30 - -Volcano版本紧跟Kubernetes社区版本节奏，对Kubernetes的每个大版本都进行支持，目前最新支持的版本为v1.30，并运行了完整的UT、E2E用例，保证功能和可靠性。 - -如果您想参与Volcano适配Kubernetes新版本的开发工作，请参考：[adapt-k8s-todo](https://github.com/volcano-sh/volcano/blob/master/docs/design/adapt-k8s-todo.md) 进行社区贡献。 - -### 增强Volcano安全性 - -Volcano一直都很重视开源软件供应链的安全，在license合规、安全漏洞披露和修复、仓库分支保护、CI检查等方面遵循OpenSSF定义的规范，Volcano近期在Github Action加入了新的workflow，它会在代码合入时运行OpenSSF安全性检查，并实时更新软件安全评分，持续提升软件安全性。 - -同时Volcano对各个组件的RBAC权限进行了收缩，只保留必要的权限，避免了潜在的越权风险，提升了系统的安全性。 - -相关PR参见： - -[Added the scorecard github action and its badge by harshitasao · Pull Request #3655 · volcano-sh/volcano](https://github.com/volcano-sh/volcano/pull/3655) - -[Shrink permissions of vc scheduler & controller by Monokaix · Pull Request #3545 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3545) - -[Add pre-install&pre-upgrade hook for admission-init job by Monokaix · Pull Request #3504 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3504) - -### 优化Volcano性能 - -在大规模场景下，Volcano做了很多性能优化的工作，主要包括： - -- 优化vcjob更新策略，降低vcjob的更新和同步频次，降低API Server压力，提升提交任务的QPS - -- vc controller新增controller gate开关，用户可以选择关闭不需要的controller，减低内存占用和CPU负载 - -- 所有的controller使用共享的informer，减少内存占用 - -### 提升GPU监控功能 - -新版本的Volcano针对GPU监控指标做了优化和增强，修复了GPU监控不精确的问题，并在GPU的算力和显存监控指标上新增了节点信息，方便用户更加直观的查看每个节点上每一张GPU的算力、显存的总量和已分配量。 - -详细PR参见：[Update volcano-vgpu monitoring system by archlitchi · Pull Request #3620 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3620/) - -### 优化helm chart包安装升级流程 - -Volcano针对helm chart的安装、升级流程进行了优化，并支持安装helm chart包设置更多自定义参数，主要包括： - -- 利用helm的hook机制，在安装成功Volcano之后，自动删除volcano-admission-init这一job，避免后续使用helm upgrade升级失败的问题，相关PR参见：[Add pre-install&pre-upgrade hook for admission-init job by Monokaix · Pull Request #3504 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3504) -- 每次安装成功后更新Volcano admission需要的secret文件，避免在不指定helm包名情况下，重复安装卸载volcano导致volcano admission处理失败的问题，详细PR参见：[Update volcano-admission secret when it already exists by Monokaix · Pull Request #3653 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3653) -- 支持为helm包中的资源对象设置通用label，相关PR参见：[Add common labels for chart objects by Aakcht · Pull Request #3511 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3511) -- 支持通过helm为Volcano组件设置日志等级，相关PR参见：[Expose volcano components (controller, scheduler, etc.) log level control to the helm chat values by chenshiwei-io · Pull Request #3656 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3656) -- 支持通过helm设置Volcano组件的镜像代理仓库，相关PR参见：[add image registry for helm by calvin0327 · Pull Request #3436 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3436) -- 支持通过helm设置容器级别的securityContext，相关PR参加：[feat: Add securityContext support at container level in helm chart templates by lekaf974 · Pull Request #3704 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3704) - -## 致谢贡献者 - -Volcano 1.10.0 版本包含了来自36位社区贡献者的上百次代码提交，在此对各位贡献者表示由衷的感谢： - -**贡献者GitHub ID** - -| **@googs1025** | **@WulixuanS** | **@SataQiu** | -| ------------------- | ----------------- | ------------------ | -| **@guoqinwill** | **@lowang-bh** | **@shruti2522** | -| **@lukasboettcher** | **@wangyysde** | **@bibibox** | -| **@Wang-Kai** | **@y-ykcir** | **@lekaf974** | -| **@yeahdongcn** | **@Monokaix** | **@Aakcht** | -| **@yxxhero** | **@babugeet** | **@liuyuanchun11** | -| **@MichaelXcc** | **@william-wang** | **@lengrongfu** | -| **@xieyanker** | **@lx1036** | **@archlitchi** | -| **@hwdef** | **@wangyang0616** | **@microyahoo** | -| **@snappyyouth** | **@harshitasao** | **@chenshiwei-io** | -| **@TaiPark** | **@Aakcht** | **@ykcai-daniel** | -| **@lekaf974** | **@JesseStutler** | **@belo4ya** | - - -## 参考链接 - -Release note: v1.10.0 - -https://github.com/volcano-sh/volcano/releases/tag/v1.10.0 - -Branch：release-1.10 - -https://github.com/volcano-sh/volcano/tree/release-1.10 - -## 深入了解Volcano - -Volcano 云原生批量计算项目主要用于 AI、大数据、基因、渲染等诸多高性能计算场景，对主流通用计算框架均有很好的支持。社区已吸引5.8万+全球开发者，并获得4.1k+ Star 和900+ Fork，参与贡献企业包括华为、AWS、百度、腾讯、京东、小红书、博云、第四范式等。目前，Volcano在人工智能、大数据、基因测序等海量数据计算和分析场景已得到快速应用，已完成对 Spark、Flink、Tensorflow、PyTorch、Argo、MindSpore、Paddlepaddle 、Kubeflow、MPI、Horovod、mxnet、KubeGene、Ray 等众多主流计算框架的支持，并构建起完善的上下游生态。 diff --git a/content/zh/blog/Volcano-1.11.0-release.md b/content/zh/blog/Volcano-1.11.0-release.md deleted file mode 100644 index fd040c75..00000000 --- a/content/zh/blog/Volcano-1.11.0-release.md +++ /dev/null @@ -1,462 +0,0 @@ -+++ -title = "Volcano v1.11.0正式发布" -description = "新增特性：网络拓扑感知调度、弹性层级队列、多集群AI作业调度、在离线混部与动态资源超卖、负载感知重调度、细粒度的作业故障恢复策略、Volcano Dashboard资源可视化、Volcano支持Kubernetes v1.31、Volcano Job支持Preemption Policy、大规模场景性能优化等" -subtitle = "" - -date = 2025-02-07 -lastmod = 2025-02-07 -datemonth = "Feb" -dateyear = "2025" -dateday = 07 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "新增特性：网络拓扑感知调度、弹性层级队列、多集群AI作业调度、在离线混部与动态资源超卖、负载感知重调度、细粒度的作业故障恢复策略、Volcano Dashboard资源可视化、Volcano支持Kubernetes v1.31、Volcano Job支持Preemption Policy、大规模场景性能优化等" - -# Add menu entry to sidebar. -linktitle = "Volcano v1.11.0正式发布" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ - -# Volcano v1.11 重磅发布！开启AI与大数据的云原生调度新纪元 - -作为云原生批量计算领域的事实标准，Volcano已经在AI、Big Data及高性能计算 (HPC) 等多种场景中获得广泛应用，吸引了来自30多个国家的800多名贡献者，累计代码提交数万次。Volcano已在国内外60+企业进行了生产落地，经受住了实际生产环境的考验，赢得了用户的广泛赞誉，为业界提供了云原生批量计算的卓越实践标准与解决方案。 - -随着用户使用场景的日益复杂，以及对资源利用率极致追求，特别是在AI大模型场景下，对训练与推理任务的性能、GPU资源利用率、可用性提出了更高的要求，促使Volcano不断拓展其应用场景，深入解决用户的核心诉求。Volcano目前的版本历程里共发布了28个release，针对批量计算的场景做了一系列功能增强和优化，帮助用户更好的将业务迁移到云原生平台，解决了诸多痛点问题，赢得了用户的广泛的喜爱与好评，用户与社区之间也形成了良好的互动，approver和reviewer数量累计发展了30+，达成了双赢互利的局面。 - -值此2025新年之际，Volcano新版本将会是一个新的里程碑，社区将在2025年引入一系列重大特性，继续深耕CNAI(Cloud Native AI 云原生AI)和大数据等领域，主要特性包括： - -**AI场景:** - -- **网络拓扑感知调度:** 降低训练任务间的网络传输开销，优化大模型训练场景下的性能。 -- **NPU卡调度和虚拟化能力:** 提升NPU资源利用率。 -- **GPU卡动态切分能力:** 提供MIG与MPS动态切分能力，提升GPU资源利用率。 -- **Volcano Global多集群AI作业调度:** 支持跨集群的AI任务部署与拆分。 -- **断点续训与故障恢复能力优化:** 支持更细粒度的作业重启策略。 -- **支持DRA：**支持动态资源分配，灵活高效的管理异构资源。 - -**大数据场景:** - -- **弹性层级队列能力:** 帮助用户将大数据业务丝滑迁移到云原生平台。 - -**微服务场景：** - -- **在离线混部与动态资源超卖:** 提升资源利用率，同时保障在线业务QoS。 -- **负载感知调度与重调度:** 提供资源碎片整理和负载均衡能力。 - -**[Volcano v1.11](https://github.com/volcano-sh/volcano/releases/tag/v1.11.0)的正式发布**，标志着云原生批量计算迈入全新阶段！本次更新聚焦AI与大数据的核心需求，推出**网络拓扑感知调度、多集群AI作业调度**等重磅特性，显著提升AI训练与推理任务的性能。同时，**在离线混部与动态资源超卖**及**负载感知重调度**功能进一步优化资源利用率，确保在线业务的高可用性。此外，**弹性层级队列**为大数据场景提供了更灵活的调度策略。Volcano v1.11不仅是技术的飞跃，更是云原生批量计算领域的全新标杆！ - -## **重磅特性详解** - -本次发布的v1.11版本针对AI、大数据和资源利用率提升场景提供一系列重磅特性更新，主要包含： - -### **网络拓扑感知调度：优化AI大模型训练性能** - -在AI大模型训练场景中，模型并行（Model Parallelism）将模型分割到多个节点上，训练过程中这些节点需要频繁进行大量数据交互。此时，节点间的网络传输性能往往成为训练的瓶颈，显著影响训练效率。数据中心的网络类型多样，如InfiniBand (IB)、RoCE、NVSwitch等，且网络拓扑复杂，通常包含多层交换机。两个节点间跨的交换机越少，通信延迟越低，吞吐量越高。因此，用户希望将工作负载调度到具有最高吞吐量和最低延迟的最佳性能域，尽可能减少跨交换机的通信，以加速数据交换，提升训练效率。 - -为此，Volcano提出了**网络拓扑感知调度（Network Topology Aware Scheduling）**策略，通过统一的网络拓扑API和智能调度策略，解决大规模数据中心AI训练任务的网络通信性能问题。 - -#### **统一的网络拓扑API：精准表达网络结构** - -为了屏蔽数据中心网络类型的差异，Volcano定义了新的CRD **HyperNode**来表示网络拓扑，提供了标准化的API接口。与传统的通过节点标签（label）表示网络拓扑的方式相比，HyperNode具有以下优势： - -- **语义统一**：HyperNode提供了标准化的网络拓扑描述方式，避免了标签方式的语义不一致问题。 -- **层级结构**：HyperNode支持树状层级结构，能够更精确地表达实际的网络拓扑。 -- **易于管理**：集群管理员可以手动创建HyperNode，或通过网络拓扑自动发现工具维护HyperNode。 - -一个HyperNode表示一个网络拓扑性能域，通常映射到一个交换机。多个HyperNode通过层级连接，形成树状结构。例如，下图展示了由多个HyperNode构成的网络拓扑： - -
{{
}} -
- -- **叶子HyperNode**（s0、s1、s2、s3）：子节点为集群中的真实节点。 -- **非叶子HyperNode**（s4、s5、s6）：子节点为其他HyperNode。 - -在这种结构中，节点间的通信效率取决于它们之间的HyperNode层级跨度。例如： - -- **node0**和**node1**同属于s0，通信效率最高。 -- **node1**和**node2**需要跨两层HyperNode（s0→s4→s1），通信效率较低。 -- **node0**和**node4**需要跨三层HyperNode（s0→s4→s6），通信效率最差。 - -##### **HyperNode配置示例** - -以下是一个叶子HyperNode和非叶子HyperNode的配置示例： - -**叶子HyperNode示例**： - -```yaml -apiVersion: topology.volcano.sh/v1alpha1 -kind: HyperNode -metadata: - name: s0 -spec: - tier: 1 # HyperNode层级，层级越低通信效率越高 - members: # 子节点列表 - - type: Node # 子节点类型为Node - selector: - exactMatch: # 精确匹配 - name: node-0 - - type: Node - selector: - regexMatch: # 正则匹配 - pattern: node-[01] -``` - -**非叶子HyperNode示例**： - -```yaml -apiVersion: topology.volcano.sh/v1alpha1 -kind: HyperNode -metadata: - name: s6 -spec: - tier: 3 # HyperNode层级 - members: # 子节点列表 - - type: HyperNode # 子节点类型为HyperNode - selector: - exactMatch: # 精确匹配 - name: s4 - - type: HyperNode - selector: - exactMatch: - name: s5 -``` - -#### **基于网络拓扑的感知调度策略** - -Volcano Job和PodGroup可以通过`networkTopology`字段设置作业的拓扑约束，支持以下配置： - -- **mode**：支持`hard`和`soft`两种模式。 - - `hard`：硬约束，作业内的任务必须部署在同一个HyperNode内。 - - `soft`：软约束，尽可能将作业部署在同一个HyperNode下。 - -- **highestTierAllowed**：与`hard`模式配合使用，表示作业允许跨到哪层HyperNode部署。 - -例如，以下配置表示作业只能部署在2层及以下的HyperNode内（如s4或s5），否则作业将处于Pending状态： - -``` -spec: - networkTopology: - mode: hard - highestTierAllowed: 2 -``` - -通过这种调度策略，用户可以精确控制作业的网络拓扑约束，确保作业在满足条件的最佳性能域运行，从而显著提升训练效率。 - -#### **未来展望** - -Volcano将持续优化网络拓扑感知调度功能，未来计划： - -- 支持从节点标签自动转换为HyperNode CR，帮助用户迁移到Volcano。 -- 集成底层网络拓扑自动发现工具，简化HyperNode的管理。 -- 提供命令行工具，方便用户查看和管理HyperNode层级结构。 - -关于Network Topology Awre Scheduling的详细设计与使用指导，请参考 - -设计文档：**[Network Topology Aware Scheduling](https://volcano.sh/en/docs/network_topology_aware_scheduling/)**。 - -使用文档：**[Network Topology Aware Scheduling | Volcano](*https://volcano.sh/en/docs/network_topology_aware_scheduling/*)**。 - -由衷感谢社区开发者: **@ecosysbin, @weapons97, @Xu-Wentao,@penggu, @JesseStutler, @Monokaix** 对该特性的贡献！ - -### **弹性层级队列：灵活的多租户资源管理策略** - -在多租户场景中，资源分配的公平性、隔离性以及任务优先级控制是核心需求。不同部门或团队通常需要共享集群资源，同时又要确保各自的任务能够按需获得资源，避免资源争用或浪费。为此，Volcano v1.11 引入了**弹性层级队列**功能，大幅增强了队列的资源管理能力。通过层级队列，用户可以实现更细粒度的资源配额管理、跨层级资源共享与回收，以及灵活的抢占策略，从而构建高效、公平的统一调度平台。同时对于使用YARN的用户，可以使用Volcano无缝将大数据业务迁移到Kubernetes集群之上。 - -#### **弹性层级队列的核心能力** - -Volcano的弹性层级队列具备以下关键特性，满足多租户场景下的复杂需求： - -1. **支持配置队列层级关系:** 用户可以按需创建多级队列，形成树状结构。每个队列可以设置独立的资源配额和优先级，确保资源的合理分配。 - -2. **跨层级资源共享与回收:** 子队列资源空闲时，可以将资源共享给兄弟队列，当子队列提交任务时，可以从兄弟队列回收资源。 - -3. **细粒度的资源配额管理:** 每个队列可以设置以下资源参数： - - `capability`：队列的资源容量上限。 - - `deserved`：队列应得的资源量。如果队列已分配的资源超过`deserved`值，超出的部分可以被回收。 - - `guarantee`：队列的资源预留量，这部分资源不会被其他队列共享，确保队列的最低资源保障。 - -4. **灵活的抢占策略:** 支持基于优先级的资源抢占，确保高优先级任务能够及时获得所需资源。 - -#### **层级队列示意图** - -以下是一个简单的层级队列结构示例： - -{{
}} - -- **根队列**：作为所有队列的父队列，负责全局资源的分配与管理。 -- **部门队列**：隶属于根队列，代表不同部门或团队的资源池。 -- **子队列**：隶属于部门队列，代表具体的项目或任务，用户可以将作业提交到叶子队列。 - -#### **适用场景** - -- **多部门资源共享**：在大型企业中，不同部门共享同一个集群，通过层级队列实现资源的公平分配与隔离。 -- **大数据任务调度**：从YARN迁移到Kubernetes的用户，可以利用Volcano的层级队列功能，无缝迁移大数据业务。 -- **AI训练与推理**：在AI场景中，不同训练任务或推理服务可以通过层级队列实现资源的动态分配与回收。 - -关于弹性层级队列详细设计与使用指导，请参考: - -设计文档: **[hierarchical-queue-on-capacity-plugin](https://github.com/volcano-sh/volcano/blob/master/docs/design/hierarchical-queue-on-capacity-plugin.md)**。 - -使用文档: **[Hierarchica Queue | Volcano](https://volcano.sh/zh/docs/hierarchical_queue/)**。 - -由衷感谢社区开发者: **@Rui-Gan** 对该特性的贡献！ - -### **多集群AI作业调度：跨集群的统一管理与高效调度** - -随着企业业务的快速增长，单个 Kubernetes 集群通常无法满足大规模 AI 训练和推理任务的需求。用户通常需要管理多个 Kubernetes 集群，以实现统一的工作负载分发、部署和管理。目前，已经有许多用户在多个集群中使用 Volcano，并使用 **[Karmada](https://karmada.io/)** 进行管理。为了更好地支持多集群环境中的 AI 任务，支持全局队列管理、任务优先级和公平调度等功能，Volcano 社区孵化了 **[Volcano Global](https://github.com/volcano-sh/volcano-global)**子项目。该项目将 Volcano 在单个集群中的强大调度能力扩展到多集群场景，为多集群 AI 任务提供统一的调度平台，支持跨集群任务分发、资源管理和优先级控制。 - -Volcano Global 在 Karmada 的基础上提供了以下增强功能，以满足多集群 AI 任务调度的复杂需求： - -#### **核心能力** - -Volcano Global在Karmada的基础上，提供了以下增强功能，满足多集群AI作业调度的复杂需求： - -1. **支持Volcano Job的跨集群调度：**用户可以在多集群环境中部署和调度Volcano Job，充分利用多个集群的资源，提升任务执行效率。 -2. **队列优先级调度：**支持跨集群的队列优先级管理，确保高优先级队列的任务能够优先获得资源。 -3. **作业优先级调度与排队：**在多集群环境中，支持作业级别的优先级调度和排队机制，确保关键任务能够及时执行。 -4. **多租户公平调度：**提供跨集群的多租户公平调度能力，确保不同租户之间的资源分配公平合理，避免资源争用。 - -
{{
}} -
- -关于Volcano Global的详细部署和使用指导，请参考: **[Multi-Cluster AI Job Scheduling | Volcano](https://volcano.sh/en/docs/multi_cluster_scheduling/)**。 - -由衷感谢社区开发者: **@Vacant2333, @MondayCha, @lowang-bh, @Monokaix** 对该特性的贡献！ - -### **在离线混部与动态资源超卖：最大化资源利用率，保障业务稳定性** - -#### **背景：资源利用率的挑战** - -随着云原生技术的快速发展，Kubernetes已成为云原生时代的“操作系统”，越来越多的业务迁移到Kubernetes平台。然而，尽管云原生技术带来了灵活性和可扩展性，数据中心的资源利用率仍然较低。在线业务（如微服务）通常具有明显的波峰波谷特征，在波谷时段，大量资源处于闲置状态，而在波峰时段，资源又可能不足。为了提升资源利用率并保障高优先级业务的**SLO（Service Level Objective）**，Volcano推出了**云原生混部解决方案**，通过在离线混部与动态资源超卖，最大化集群资源利用率，同时确保在线业务的稳定性。 - -**云原生混部**的核心思想是将**在线业务**（如实时服务）和**离线业务**（如批处理任务）部署在同一个集群中。当在线业务处于波谷时，离线业务可以利用闲置资源；当在线业务达到波峰时，通过优先级控制压制离线业务，确保在线业务的资源需求。这种动态资源分配机制不仅提升了资源利用率，还保障了在线业务的服务质量。 - -#### **业界实践：Volcano的独特优势** - -业界已有许多公司和用户对在离线混部技术进行了探索与实践，但仍存在一些不足，比如不能做到和Kubernetes完全解耦，超卖资源计算方式粗糙，在离线作业使用方式不一致、用户体验不友好等问题。 - -基于这些问题，Volcano对在离线混部技术进行了深度优化，具备以下独特优势： - -- **天然支持离线作业调度**：Volcano Scheduler原生支持离线作业的调度与管理，无需额外适配。 -- **无侵入式设计**：对Kubernetes无侵入式修改，用户无需调整现有集群架构即可使用。 -- **动态资源超卖**：实时计算节点的可超卖资源，确保资源利用与业务QoS的平衡。 -- **OS层面的隔离与保障**：通过内核级别的资源隔离机制，确保在线业务的优先级和稳定性。 - -#### **Volcano云原生混部解决方案：端到端的资源优化** - -Volcano的云原生混部解决方案从应用层到内核提供了端到端的资源隔离与共享机制，主要包括以下核心组件： - -**Volcano Scheduler：**负责在离线作业的统一调度，提供队列、组、作业优先级、公平调度、资源预留等多种抽象，满足微服务、大数据、AI等多种业务场景的调度需求。 - -**Volcano SLO Agent：**每个节点上部署的SLO Agent实时监控节点的资源使用情况，动态计算可超卖的资源，并将这些资源分配给离线作业。同时，SLO Agent会检测节点的CPU/内存压力，在必要时驱逐离线作业，保障在线业务的优先级。 - -**Enhanced OS：**为了进一步强化资源隔离，Volcano在内核层面实现了精细化的QoS保障。通过cgroup接口，为在线和离线业务设置不同的资源限制，确保在线业务在高负载时仍能获得足够的资源。 - -
{{
}} -架构 -
- -#### **核心能力：资源利用与业务保障的双赢** - -Volcano云原生混部解决方案具备以下关键能力，帮助用户实现资源利用与业务稳定性的双赢： - -- **统一调度**：支持多种工作负载的统一调度，包括微服务、批处理作业和AI任务。 -- **基于QoS的资源模型**：为在线和离线业务提供基于服务质量（QoS）的资源管理，确保高优先级业务的稳定性。 -- **动态资源超卖**：根据节点的实时CPU/内存利用率，动态计算可超卖的资源，最大化资源利用率。 -- **CPU Burst**：允许容器临时超出CPU限制，避免在关键时刻被限流，提升业务响应速度。 -- **网络带宽隔离**：支持整机网络出口带宽限制，保障在线业务的网络使用需求。 - -关于Volcano云原生混部的详细设计和使用文档，请参考: **[Cloud Native Colocation | Volcano](https://volcano.sh/en/docs/colocation/)**。 - -由衷感谢社区开发者: **@william-wang** 对该特性的贡献！ - -### **负载感知重调度：智能均衡集群资源，告别资源热点** - -在Kubernetes集群中，随着工作负载的动态变化，节点资源利用率不均衡的问题时常发生，导致部分节点过热，影响整体集群的稳定性与效率。为了解决这一问题，Volcano v1.11 引入了**负载感知重调度**功能，基于节点的真实负载动态调整Pod分布，确保集群资源的均衡利用，避免资源热点，提升集群的整体性能与可靠性。负载感知重调度通过子项目 [descheduler](https://github.com/volcano-sh/descheduler) 孵化。 - -#### 核心能力： - -- **真实负载感知调度**：通过监控节点的CPU、内存等真实负载指标，动态调整Pod分布，避免仅依赖Pod Request的粗糙调度。 -- **定时与动态触发**：支持按CronTab定时任务或固定时间间隔触发重调度，灵活适应不同场景需求。 - -#### 适用场景： - -- **节点资源不均衡**：当集群中部分节点资源利用率过高，而其他节点资源闲置时，负载感知重调度可自动平衡节点负载。 - -- **热点节点治理**：当节点因高负载出现性能瓶颈或故障风险时，重调度可及时迁移Pod，保障业务稳定性。 - -
{{
}} -
- -#### 技术亮点： - -- **基于真实负载的重调度**：相比传统的基于Pod Request的调度策略，Volcano的负载感知重调度更加精准，能够真实反映节点的资源使用情况。 -- **无缝集成Kubernetes生态**：与Kubernetes原生调度器兼容，无需额外配置即可实现负载感知重调度。 -- **灵活的策略配置**：用户可根据业务需求，自定义重调度的时间间隔或触发条件，确保调度的灵活性与可控性。 - -关于负载感知重调度的使用说明，请参考: **[Load-aware Descheduling | Volcano](https://volcano.sh/en/docs/descheduler/)**。 - -由衷感谢社区开发者: **@Monokaix** 对该特性的贡献！ - -### **细粒度的作业故障恢复策略：高效应对任务中断，提升训练效率** - -在AI、大数据和高性能计算（HPC）场景中，作业的稳定性和故障恢复能力至关重要。传统的作业故障恢复策略通常会在某个Pod失败时重启整个Job，这不仅浪费资源，还可能导致训练任务从头开始，严重影响效率。随着AI场景中**断点续训**和**Checkpoint**技术的普及，单个Pod的失败不再需要重启整个Job。为此，Volcano v1.11 引入了**细粒度的作业故障恢复策略**，支持更灵活的故障处理机制，帮助用户高效应对任务中断，显著提升训练效率。 - -#### 核心能力： - -##### **支持Pod粒度的重启策略** - -用户可以根据需求，设置仅重启失败的Pod或所属的Task，避免不必要的Job重启，减少资源浪费。 - -- **重启单个Pod**：当某个Pod失败时，仅重启该Pod，不影响其他正常运行的任务。 - - ``` - policies: - - event: PodFailed - action: RestartPod - ``` - -- **重启整个Task**：当某个Pod失败时，重启该Pod所属的Task（一组Pod），适用于需要保持任务组一致性的场景。 - - ``` - policies: - - event: PodFailed - action: RestartTask - ``` - -##### **支持为Action设置超时时间** - -Pod失败可能是由临时性故障（如网络抖动或硬件问题）引起的，Volcano允许用户为故障恢复动作设置超时时间。如果在超时时间内Pod恢复正常，则不再执行重启操作，避免过度干预。 - -- **示例配置**：若Pod失败后重启，10分钟内仍未恢复，则重启整个Job。 - - ``` - policies: - - event: PodFailed - action: RestartPod - - event: PodEvicted - action: RestartJob - timeout: 10m - ``` - -##### **新增PodPending事件处理** - -当Pod因资源不足或拓扑约束长期处于Pending状态时，用户可以为Pending事件设置超时时间。若超时后Pod仍未运行，则可以选择终止整个Job，避免资源浪费。 - -- **示例配置**：若Pod处于Pending状态超过10分钟，则终止Job。 - - ``` - policies: - - event: PodPending - action: TerminateJob - timeout: 10m - ``` - -#### **适用场景：** - -- **AI大模型训练**：在分布式训练中，单个Pod的失败不会影响整体训练进度，通过细粒度的故障恢复策略，可以快速恢复任务，避免从头开始训练。 -- **大数据处理**：在批处理任务中，部分任务的失败可以通过重启单个Pod或Task解决，无需重启整个作业，提升处理效率。 -- **高性能计算**：在HPC场景中，任务的稳定性和高效恢复至关重要，细粒度的故障恢复策略可以最大限度地减少任务中断时间。 - -#### 技术亮点： - -- **灵活的策略配置**：用户可以根据业务需求，自定义故障恢复策略，支持Pod、Task和Job级别的重启操作。 -- **超时机制**：通过设置超时时间，避免因临时性故障导致的过度重启行为，提升作业的稳定性。 -- **无缝兼容断点续训**：与AI场景中的断点续训和Checkpoint技术完美结合，确保训练任务的高效恢复。 - -关于Volcano Job的详细设计和说明文档，请参考: **[How to use job policy](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_job_policy.md)**。 - -由衷感谢社区开发者: **@bibibox** 对该特性的贡献！ - -### **Volcano Dashboard：资源管理的可视化利器** - -Volcano dashboard是Volcano官方提供的资源展示仪表盘，用户在部署Volcano后，再部署Volcano dashboard，就可以通过图形界面展示集群中Volcano相关的资源，方便用户查询和操作，项目地址: https://github.com/volcano-sh/dashboard - -目前支持的功能有： - -- 支持查看集群总览，包括Job数量、状态、完成率，Queue数量，Queue的资源利用率等。 -- 支持查看Job列表和详情，支持模糊搜索匹配，支持按照Namespace、Queue、Status等条件过滤，支持Job排序展示。 -- 支持查看Queue列表和详情，支持模糊搜索匹配，支持按照Status等条件过滤，支持Queue排序展示。 -- 支持查看Pod的列表和详情，支持模糊搜索匹配，支持按照Namespace、Status等条件过滤，支持Pod排序展示。 - -由衷感谢社区开发者: **@WY-Dev0, @Monokaix** 对该特性的贡献！ - -### **Volcano支持Kubernetes v1.31** - -Volcano版本紧跟Kubernetes社区版本节奏，对Kubernetes的每个大版本都进行支持，目前最新支持的版本为v1.31，并运行了完整的UT、E2E用例，保证功能和可靠性。 - -如果您想参与Volcano适配Kubernetes新版本的开发工作，请参考：**[adapt-k8s-todo](https://github.com/volcano-sh/volcano/blob/master/docs/design/adapt-k8s-todo.md)**进行社区贡献。 - -由衷感谢社区开发者: **@vie-serendipity, @dongjiang1989** 对该特性的贡献！ - -### **Volcano Job支持Preemption Policy** - -PriorityClass可以表示Pod的优先级，包含一个优先级数值和抢占策略，在调度和抢占的过程中，PriorityClass会被用来作为调度和抢占的依据，高优先级的Pod先于低优先级Pod调度，并且可以抢占低优先级的Pod，Volcano在Pod层面完整支持优先级调度和抢占策略，在Volcano Job层面支持基于priorityClass value的优先级调度和抢占。但在某些场景下，用户希望Volcano Job不通过抢占触发资源回收，而是等待集群资源自动释放，从而整体保障业务稳定性，Volcano在新版本支持了Job级别的PreemptionPolicy，配置了PreemptionPolicy为Never的Volcano Job不会抢占其他Pod。 - -Volcano Job和Job内的task同时支持配置PriorityClass，关于两个PriorityClass的配合关系以及配置样例请参考: **[how to configure priorityclass for job](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_configure_priorityclass_for_job.md)**。 - -由衷感谢社区开发者: **@JesseStutler** 对该特性的贡献！ - -### **性能优化：大规模场景下的高效调度** - -在Volcano中，**Queue**是最基本且最重要的资源之一。Queue的`status`字段记录了其中状态为`Unknown`、`Pending`、`Running`、`Inqueue`、`Completed`的PodGroup。然而，在大规模场景下，当队列中的PodGroup频繁发生变化时（例如，队列中循环提交大量运行时间较短的任务），会导致大量PodGroup状态从`Running`变为`Completed`。这种情况下，Volcano Controller需要频繁刷新Queue的`status`字段，给APIServer带来较大压力。此外，Volcano Scheduler在Job调度完成后会更新Queue的`status.allocated`字段，这在大规模场景下可能导致Queue更新冲突，进一步影响系统性能。 - -为了彻底解决大规模场景下Queue频繁刷新和更新冲突的问题，Volcano v1.11 对Queue的管理机制进行了优化，将Queue中PodGroup的统计数据迁移到**指标（Metrics）**中，不再进行持久化存储。这一优化显著降低了APIServer的压力，同时提升了系统的整体性能和稳定性。 - -#### **优化后的核心改进** - -**PodGroup统计数据迁移到指标**Queue中的PodGroup状态数据（如`Unknown`、`Pending`、`Running`等）不再存储在Queue的`status`字段中，而是通过指标系统进行记录和展示。用户可以通过以下命令查看Queue中PodGroup的统计数据： - -- **查看指定队列的统计数据**： - - ```bash - vcctl queue get -n [name] - ``` - -- **查看所有队列的统计数据**： - - ```bash - vcctl queue list - ``` - -**减少APIServer压力**通过将PodGroup统计数据迁移到指标中，避免了频繁更新Queue的`status`字段，显著降低了APIServer的负载，提升系统吞吐。 - -**解决Queue更新冲突**在大规模场景下，Queue的更新冲突问题得到了有效缓解，确保了调度器的高效运行。 - -关于Queue中PodGroup的状态统计数据迁移到指标的详细设计以及指标名称，请参考: **[Queue podgroup statistics](https://github.com/volcano-sh/volcano/blob/master/docs/design/podgroup-statistics.md)**。 - -由衷感谢社区开发者: **@JesseStutler** 对该特性的贡献！ - -## **总结：Volcano v1.11，云原生批量计算的新标杆** - -Volcano v1.11不仅是技术的飞跃，更是云原生批量计算领域的全新标杆。无论是AI大模型训练、大数据调度，还是资源利用率的提升，Volcano v1.11都提供了强大的功能和灵活的解决方案。我们相信，Volcano v1.11将帮助用户在云原生批量计算领域走得更远、更稳，开启AI与大数据的云原生调度新纪元！ - -**立即体验Volcano v1.11.0，开启高效计算新时代！** - -**v1.11.0 release:** https://github.com/volcano-sh/volcano/releases/tag/v1.11.0 - -## **致谢贡献者** - -Volcano v1.11.0 版本包含了来自39位社区贡献者的上百次代码提交，在此对各位贡献者表示由衷的感谢，贡献者GitHub ID： - -| @QingyaFan | @JesseStutler | @bogo-y | -| :------------- | :-------------- | :--------------- | -| @bibibox | @zedongh | @archlitchi | -| @dongjiang1989 | @william-wang | @fengruotj | -| @SataQiu | @lowang-bh | @Rui-Gan | -| @xovoxy | @wangyang0616 | @PigNatovsky | -| @Yanping-io | @lishangyuzi | @hwdef | -| @bood | @kerthcet | @WY-Dev0 | -| @raravena80 | @SherlockShemol | @zhifanggao | -| @conghuhu | @MondayCha | @vie-serendipity | -| @Prepmachine4 | @Monokaix | @lengrongfu | -| @jasondrogba | @sceneryback | @TymonLee | -| @liuyuanchun11 | @Vacant2333 | @matbme | -| @lekaf974 | @kursataktas | @lut777 | diff --git a/content/zh/blog/Volcano-1.12.0-release.md b/content/zh/blog/Volcano-1.12.0-release.md deleted file mode 100644 index a64d5cc4..00000000 --- a/content/zh/blog/Volcano-1.12.0-release.md +++ /dev/null @@ -1,313 +0,0 @@ -+++ -title = "Volcano v1.12.0正式发布" -description = "新增特性：网络拓扑感知调度达到Alpha阶段、GPU虚拟化动态MIG切分、DRA动态资源分配、Volcano Global队列容量管理、安全性增强、性能优化、通用工作负载Gang调度支持、Job Flow增强、Kubernetes v1.32支持等" -subtitle = "" - -date = 2025-06-12 -lastmod = 2025-06-12 -datemonth = "June" -dateyear = "2025" -dateday = 12 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "新增特性：网络拓扑感知调度达到Alpha阶段、GPU虚拟化动态MIG切分、DRA动态资源分配、Volcano Global队列容量管理、安全性增强、性能优化、通用工作负载Gang调度支持、Job Flow增强、Kubernetes v1.32支持等" - -# Add menu entry to sidebar. -linktitle = "Volcano v1.12.0正式发布" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ - -## Volcano v1.12 正式发布！驱动云原生AI与批量计算向智能高效新阶段演进 - -随着AI大模型技术的快速发展，企业对计算资源利用效率和应用性能的要求日益提高。在AI、大数据及高性能计算（HPC）等复杂应用场景下，如何高效利用GPU等异构加速器、保障系统高可用性并精细化管理资源，是Volcano社区持续探索和创新的核心方向。 - -Volcano的每一次版本迭代，都是对这些挑战的积极回应。在来自全球30余个国家、**超过1000名开发者、近40000次贡献**的共同建设下，Volcano已在国内外60多家企业的生产环境中得到应用，其调度性能与资源管理能力在实践中获得了广泛认可。 - -今天，**Volcano社区正式发布 v1.12 版本。** 新版本聚焦于AI与大数据等前沿场景的核心需求，带来了一系列关键特性与体验优化： - -### **新版本亮点一览** - -* **网络拓扑感知调度 (Alpha):** 优化大规模AI训练与推理任务部署，通过感知网络拓扑减少跨交换机通信，提升运行效率。 -* **GPU虚拟化方案增强:** 在原有vCUDA方案基础上，新增对NVIDIA GPU动态MIG切分的支持，为用户提供软件与硬件两种虚拟化选择，实现更灵活、高效的GPU资源共享。 -* **DRA支持:** 增强异构资源管理的灵活性与能力。 -* **Volcano Global支持队列容量管理:** 在多集群环境下，支持对租户队列的资源配额（capability）进行统一限制和管理。 -* **安全性全面增强:** 从API访问控制到容器运行时权限，实施多维度安全加固，提升系统稳健性。 -* **大规模场景性能优化:** 通过减少不必要的Webhook调用等手段，有效提升高并发任务处理效率。 -* **增强通用工作负载的Gang调度控制:** 现已支持通过Annotation为Deployment、StatefulSet等通用工作负载自定义Gang调度所需的最小成员数（minMember），提供了更精细的Gang Scheduling调度策略。 -* **Job Flow功能增强:** 提升了内置工作流编排引擎的健壮性与可观测性。 -* 以及更多稳定性与易用性改进。 - -我们相信，v1.12版本的这些更新将进一步提升任务调度的智能化水平、资源利用效率和系统的整体性能，帮助用户更好地应对AI和大数据时代的挑战。 - -## 核心功能详解 - -### 网络拓扑感知调度 (Alpha Release) - -Volcano 的网络拓扑感知调度功能，在 v1.11 中作为预览版发布后，现已在 v1.12 中达到 Alpha 发布状态。此功能旨在优化大规模训练和推理场景（如模型并行训练、Leader-Worker 推理）中 AI 任务的部署。它通过将任务调度到同一网络拓扑性能域内，减少跨交换机通信，从而显著提升任务效率。Volcano 使用 HyperNode CRD 来抽象和表示异构硬件网络拓扑，并支持层级结构以方便管理。 - -v1.12 版本集成了以下关键特性： - -* **HyperNode 自动发现 (HyperNode Auto-Discovery)**： Volcano 提供了集群网络拓扑的自动发现能力。用户可配置发现类型，系统将自动创建和维护反映集群真实网络拓扑的层级 HyperNode。目前支持 InfiniBand (IB) 网络下通过 UFM (Unified Fabric Manager) 接口获取网络拓扑信息，并自动更新 HyperNode。未来计划支持 RoCE 等更多网络协议。 -* **HyperNode 优选策略 (Prioritized HyperNode Selection)**： 引入了基于节点级别和 HyperNode 级别的打分策略，累加后作为 HyperNode 的最终得分。 - * **节点级别 (Node-level)**： 建议配置 BinPack 插件以优先填满 HyperNode，减少资源碎片。 - * **HyperNode 级别 (HyperNode-level)**： 优先选择层级更低的 HyperNode 以获得更优性能，因其涉及的跨交换机次数较少；对于相同层级的 HyperNode，包含更多任务的 HyperNode 得分更高，旨在减少 HyperNode 级别的资源碎片。 -* **支持通过 Label Selector 匹配节点 (Support for Label Selector Node Matching)**： HyperNode 叶子节点与集群中的物理节点关联，支持以下三种匹配策略： - * **精确匹配 (Exact Match)**： 直接匹配节点名称。 - * **正则匹配 (Regex Match)**： 通过正则表达式匹配节点名称。 - * **标签匹配 (Label Match)**： 通过标准 Label Selector 匹配节点。 - -相关参考文档： - -* [网络拓扑感知调度介绍与使用](https://volcano.sh/zh/docs/network_topology_aware_scheduling/) -* [网络拓扑感知调度设计文档](https://github.com/volcano-sh/volcano/blob/master/docs/design/Network%20Topology%20Aware%20Scheduling.md) -* [网络拓扑自动发现设计文档](https://github.com/volcano-sh/volcano/blob/master/docs/design/hyperNode-auto-discovery.md) -* [网络拓扑自动发现使用文档](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_hypernode_auto_discovery.md) - -Related PRs: - -* [https://github.com/volcano-sh/volcano/pull/3874](https://github.com/volcano-sh/volcano/pull/3874) -* [https://github.com/volcano-sh/volcano/pull/3894](https://github.com/volcano-sh/volcano/pull/3894) -* [https://github.com/volcano-sh/volcano/pull/3969](https://github.com/volcano-sh/volcano/pull/3969) -* [https://github.com/volcano-sh/volcano/pull/3971](https://github.com/volcano-sh/volcano/pull/3971) -* [https://github.com/volcano-sh/volcano/pull/4068](https://github.com/volcano-sh/volcano/pull/4068) -* [https://github.com/volcano-sh/volcano/pull/4213](https://github.com/volcano-sh/volcano/pull/4213) -* [https://github.com/volcano-sh/volcano/pull/3897](https://github.com/volcano-sh/volcano/pull/3897) -* [https://github.com/volcano-sh/volcano/pull/3887](https://github.com/volcano-sh/volcano/pull/3887) - -由衷感谢社区开发者: **@ecosysbin, @weapons97, @Xu-Wentao, @penggu, @JesseStutler, @Monokaix**对该特性的贡献！ - -### GPU 虚拟化支持动态 MIG 切分 - -Volcano 提供的 GPU 虚拟化功能支持按显存和算力申请部分 GPU 资源，通过与 Device Plugin 配合实现硬件隔离，从而提升 GPU 利用率。传统 GPU 虚拟化通过拦截 CUDA API 方式限制 GPU 使用。NVIDIA Ampere 架构引入的 MIG (Multi-Instance GPU) 技术允许将单个物理 GPU 划分为多个独立实例。然而，通用 MIG 方案通常预先固定实例大小，存在资源浪费和灵活性不足的问题。 - -**Volcano v1.12 提供了动态 MIG 切分与调度能力**，可根据用户申请的 GPU 用量实时选择合适的 MIG 实例大小，并使用 Best-Fit 算法减少资源浪费。同时支持 BinPack 和 Spread 等 GPU 打分策略，以减少资源碎片并提升 GPU 利用率。用户可使用统一的 `volcano.sh/vgpu-number`、`volcano.sh/vgpu-cores`、`volcano.sh/vgpu-memory` API 申请资源，无需关注底层实现。 - -* 设计文档：[Dynamic MIG 设计文档](https://github.com/volcano-sh/volcano/blob/master/docs/design/dynamic-mig.md) -* 使用文档：[Dynamic MIG 使用文档](https://volcano.sh/zh/docs/gpu_virtualization/) - -Related PRs: - -* [https://github.com/volcano-sh/volcano/pull/4290](https://github.com/volcano-sh/volcano/pull/4290) -* [https://github.com/volcano-sh/volcano/pull/3953](https://github.com/volcano-sh/volcano/pull/3953) - -由衷感谢社区开发者: **@sailorvii, @archlitchi** 对该特性的贡献！ - -### 支持 DRA (Dynamic Resource Allocation) - -Kubernetes DRA (Dynamic Resource Allocation，动态资源分配) 是一项内置的 Kubernetes 功能，旨在提供一种更灵活、更强大的方式来管理集群中的异构硬件资源，例如 GPU、FPGA、高性能网卡等。它解决了传统设备插件 (Device Plugin) 在某些高级场景下的局限性。Volcano v1.12 增加了对 DRA 的支持，允许集群动态分配和管理外部资源，增强了 Volcano 与 Kubernetes 生态系统的集成能力及资源管理的灵活性。 - -* 使用文档：[在Volcano中启用DRA](https://volcano.sh/zh/docs/unified_scheduling/#2-1-2-%E5%9C%A8volcano%E4%B8%AD%E5%90%AF%E7%94%A8dra-dynamic-resource-allocation) - -Related PR: - -* [https://github.com/volcano-sh/volcano/pull/3799](https://github.com/volcano-sh/volcano/pull/3799) - -由衷感谢社区开发者: **@JesseStutler** 对该特性的贡献！ - -### Volcano Global 支持队列容量管理 - -队列是 Volcano 的核心概念。为支持多集群和多租户场景下的租户配额管理，Volcano 在 v1.12 中扩展了其全局队列容量管理能力。现在，用户可以在多集群环境中统一限制租户的资源使用，其配置方式与单集群场景一致：通过在队列配置中设置 `capability` 字段来限制租户配额。 - -Related PR: - -* [https://github.com/volcano-sh/volcano-global/pull/16](https://github.com/volcano-sh/volcano-global/pull/16) - -由衷感谢社区开发者: **@tanberBro** 对该特性的贡献！ - -### 安全性增强 - -Volcano 社区持续关注安全性。在 v1.12 中，除了对 ClusterRole 等敏感权限的精细控制外，还修复了以下潜在安全风险并进行了加固： - -* **HTTP Server 设置超时时间**： Volcano 各组件的 Metric 和 Healthz 端点均已设置服务器端的 ReadHeader、Read、Write 超时，避免资源长时间占用。(PR: [https://github.com/volcano-sh/volcano/pull/4208](https://github.com/volcano-sh/volcano/pull/4208)) -* **跳过 SSL 证书验证时增加警告日志**： 当客户端请求设置`insecureSkipVerify`为 true时，添加警告日志，建议生产环境启用 SSL 证书验证。(PR: [https://github.com/volcano-sh/volcano/pull/4211](https://github.com/volcano-sh/volcano/pull/4211)) -* **默认关闭 Volcano Scheduler 的 pprof 端点**： 为避免敏感程序信息泄露，默认关闭用于定位问题的 Profiling 数据端口。(PR: [https://github.com/volcano-sh/volcano/pull/4173](https://github.com/volcano-sh/volcano/pull/4173)) -* **移除不必要的文件权限**： 移除 Go 源文件不必要的执行权限，保持文件最小权限。(PR: [https://github.com/volcano-sh/volcano/pull/4171](https://github.com/volcano-sh/volcano/pull/4171)) -* **为容器设置 Security Context 并以非 Root 权限运行**： 所有 Volcano 组件均以非 Root 权限运行，并增加了 seccompProfile, SELinuxOptions，设置 `allowPrivilegeEscalation`避免容器提权，同时仅保留必要的 Linux Capabilities，全面限制容器权限。(PR: [https://github.com/volcano-sh/volcano/pull/4207](https://github.com/volcano-sh/volcano/pull/4207)) -* **限制 HTTP 请求返回体大小**： 针对 Extender Plugin 和 Elastic Search Service 发送的 HTTP 请求，限制其返回体大小，避免资源过度消耗导致的 OOM 等问题。(披露地址: [https://github.com/volcano-sh/volcano/security/advisories/GHSA-hg79-fw4p-25p8](https://github.com/volcano-sh/volcano/security/advisories/GHSA-hg79-fw4p-25p8)) - -### 大规模场景性能提升 - -Volcano 持续优化性能。新版本在不影响功能的前提下，默认移除和关闭了部分非必要的 Webhook，提升了大规模批创建场景下的性能： - -* **默认关闭 PodGroup 的 Mutating Webhook**： 在创建 PodGroup 未指定队列时，可从 Namespace 读取进行填充。由于该场景不常见，故默认关闭此 Webhook。用户可按需开启。 -* **任务提交时的队列状态校验从 Pod 迁移到 PodGroup**： 当队列处于关闭状态时，不允许提交任务。原校验逻辑在创建 Pod 时进行，而 Volcano 的调度基本单位是 PodGroup，将校验迁移至 PodGroup 创建时更为合理。因 PodGroup 数量少于 Pod，此举可减少 Webhook 调用，提升性能。 - -Related PRs: - -* [https://github.com/volcano-sh/volcano/pull/4128](https://github.com/volcano-sh/volcano/pull/4128) -* [https://github.com/volcano-sh/volcano/pull/4132](https://github.com/volcano-sh/volcano/pull/4132) - -由衷感谢社区开发者: **@Monokaix** 对该特性的贡献！ - -### 多种负载类型支持 Gang 调度 - -Gang 调度是 Volcano 的核心能力。对于 Volcano Job 和 PodGroup 对象，用户可直接设置 `minMember` 来定义所需最小副本数。在新版本中，用户可通过在 Deployment、StatefulSet、Job 等其他类型工作负载上设置 Annotation `scheduling.volcano.sh/group-min-member` 来指定所需最小副本数。这意味着在使用 Volcano 调度时，要么指定数量的副本全部调度成功，要么一个也不调度，从而为多种负载类型实现了 Gang 调度。 - -例如，为 Deployment 设置 minMember=10： - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: volcano-group-deployment - annotations: - # Set min member=10 - scheduling.volcano.sh/group-min-member: "10" -``` - -Related PR: - -* [https://github.com/volcano-sh/volcano/pull/4000](https://github.com/volcano-sh/volcano/pull/4000) - -由衷感谢社区开发者: **@sceneryback** 对该特性的贡献！ - -### Job Flow 功能增强 - -Job Flow 是 Volcano 提供的轻量级 Volcano Job 工作流编排框架。在 v1.12 版本中，Job Flow 进行了以下增强： - -* 新增监控指标： 增加了对成功和失败的 Job Flow 数量的度量支持。 -* DAG 合法性校验： 引入了对 Job Flow DAG (有向无环图) 结构进行合法性校验的功能。 -* 状态同步问题修复： 解决了 Job Flow 状态同步不准确的问题。 - -Related PRs: - -* [https://github.com/volcano-sh/volcano/pull/4169](https://github.com/volcano-sh/volcano/pull/4169) -* [https://github.com/volcano-sh/volcano/pull/4090](https://github.com/volcano-sh/volcano/pull/4090) -* [https://github.com/volcano-sh/volcano/pull/4135](https://github.com/volcano-sh/volcano/pull/4135) - -由衷感谢社区开发者: **@dongjiang1989** 对该特性的贡献！ - -### 多租户场景下更细粒度的权限控制 - -Volcano 原生支持多租户环境，并重视多租户场景下的权限控制。在新版本中，Volcano 增强了对 Volcano Job 的权限控制，增加了只读和读写的 ClusterRole，用户可根据需要为不同租户分配不同的读写权限，以实现权限隔离。 - -Related PR: - -* [https://github.com/volcano-sh/volcano/pull/4174](https://github.com/volcano-sh/volcano/pull/4174) - -由衷感谢社区开发者: **@Hcryw** 对该特性的贡献！ - -### 支持 Kubernetes 1.32 - -Volcano 版本紧随 Kubernetes 社区版本。v1.12 支持最新的 Kubernetes v1.32 版本，并通过完整的 UT 和 E2E 测试用例确保功能和可靠性。 - -如需参与 Volcano 对新 Kubernetes 版本的适配工作，请参考：[adapt-k8s-todo](https://github.com/volcano-sh/volcano/pull/4318)。 - -Related PR: - -* [https://github.com/volcano-sh/volcano/pull/4099](https://github.com/volcano-sh/volcano/pull/4099) - -由衷感谢社区开发者: **@guoqinwill, @danish9039** 对该特性的贡献！ - -### 队列监控指标增强 - -Volcano 队列新增了多项关键资源度量指标。现在支持对 CPU、Memory 及扩展资源的请求量 (request)、已分配量 (allocated)、应得量 (deserved)、容量 (capacity) 和 实际容量 (real\_capacity) 等指标进行监控与可视化，提供队列关键资源状态的详细视图。 - -Related PR: - -* [https://github.com/volcano-sh/volcano/pull/3937](https://github.com/volcano-sh/volcano/pull/3937) - -由衷感谢社区开发者: **@zedongh** 对该特性的贡献！ - -### 支持模糊测试 - -模糊测试 (Fuzz Testing) 是一种自动化软件测试技术。Volcano 在新版本中引入了模糊测试框架，对关键函数单元进行了模糊测试，并使用 Google 开源的 OSS-Fuzz 模糊测试框架进行持续测试，旨在提前发现潜在漏洞和缺陷，增强 Volcano 的安全性和健壮性。 - -Related PR: - -* [https://github.com/volcano-sh/volcano/pull/4205](https://github.com/volcano-sh/volcano/pull/4205) - -由衷感谢社区开发者: **@AdamKorcz** 对该特性的贡献！ - -### 稳定性增强 - -新版本中修复了多项稳定性问题，包括队列容量设置不合理导致的 Panic、层级队列校验失败、PodGroup 无意义刷新以及 StatefulSet 副本为0时仍占用队列资源等问题，进一步提升了系统在复杂场景下的稳定运行能力。 - -Related PRs: - -* [https://github.com/volcano-sh/volcano/pull/4273](https://github.com/volcano-sh/volcano/pull/4273) -* [https://github.com/volcano-sh/volcano/pull/4272](https://github.com/volcano-sh/volcano/pull/4272) -* [https://github.com/volcano-sh/volcano/pull/4179](https://github.com/volcano-sh/volcano/pull/4179) -* [https://github.com/volcano-sh/volcano/pull/4141](https://github.com/volcano-sh/volcano/pull/4141) -* [https://github.com/volcano-sh/volcano/pull/4033](https://github.com/volcano-sh/volcano/pull/4033) -* [https://github.com/volcano-sh/volcano/pull/4012](https://github.com/volcano-sh/volcano/pull/4012) -* [https://github.com/volcano-sh/volcano/pull/3603](https://github.com/volcano-sh/volcano/pull/3603) - -由衷感谢社区开发者: **@halcyon-r, @guoqinwill, @JackyTYang, @JesseStutler, @zhutong196, @Wang-Kai, @HalfBuddhist** 的贡献！ - -#### 升级前注意事项 - -在升级到 Volcano v1.12 之前，请注意以下改动： - -* **PodGroup Mutating Webhook 默认关闭**： 在 v1.12 中，PodGroup 的 Mutating Webhook 默认处于关闭状态。若您有依赖此行为（创建 PodGroup 未指定队列时从 Namespace 填充）的特定工作流，请确保在升级后手动开启此 Webhook。 -* **队列状态校验迁移及行为变更**： 任务提交时的队列状态校验逻辑已从 Pod 创建阶段迁移到 PodGroup 创建阶段。当队列处于关闭状态时，系统将在 PodGroup 创建时即阻止任务提交。然而，如果在队列关闭后继续向该队列提交独立的 Pod（非通过 PodGroup 提交），这些 Pod 可以提交成功，但 Volcano Scheduler 将不会对其进行调度。 -* **Volcano Scheduler pprof 端点默认禁用**： 出于安全增强考虑，Volcano Scheduler 的 pprof 端点在此版本中默认禁用。如需使用，可通过 Helm 参数 `custom.scheduler_pprof_enable=true` 或命令行参数 `--enable-pprof=true` 显式启用。 - -## 总结与展望 - -Volcano v1.12 版本的发布，得益于社区贡献者和用户的共同努力。此版本在 AI 任务调度、GPU 资源利用率、异构资源管理、安全性以及大规模场景下的性能与稳定性等多个方面进行了增强。 - -v1.12 版本旨在提升用户在云原生环境中运行 AI、大数据等批量计算任务的性能和效率。我们建议用户升级并体验新版本，并欢迎通过社区渠道提供使用反馈与改进建议。 - -未来，Volcano 社区将继续关注 CNAI 和大数据等领域的核心需求，持续进行迭代。 - -## **未来展望与需求征集** - -Volcano 社区始终致力于构建更加强大、灵活和易用的批量计算平台，并积极响应快速发展的技术趋势与用户需求。在接下来的版本迭代中，我们计划重点投入以下方向： - -1. **深化网络拓扑感知调度能力**：在v1.12 Alpha版本的基础上，我们将持续演进网络拓扑感知能力。重点包括提供对RoCE网络的自动发现支持、节点标签的智能识别与利用，并向更细粒度的任务级（Task-level）拓扑感知调度迈进。同时，我们也将积极探索和实现更多高级调度特性，以应对复杂AI训练等场景的极致性能需求。相关issue: - * [HyperNode based binpack scheduling policy needed](https://github.com/volcano-sh/volcano/issues/4331) - * [Support task level network topology constraint](https://github.com/volcano-sh/volcano/issues/4188) - * [Support identifying network topology from node labels and converting into hyperNode resources](https://github.com/volcano-sh/volcano/issues/4145) - * [Network-topology-aware scheduling optimization: node reordering for tasks](https://github.com/volcano-sh/volcano/issues/4233) -2. **引入高级资源管理机制**：重点开发和完善作业重调度（Rescheduling）与资源预留（Resource Reservation）功能。这将有助于更灵活地应对集群动态负载变化，保障关键任务的资源确定性，并进一步提升整体集群的资源利用效率。相关issue: - * [GPU fragmentation across nodes and Job/Pod rescheduling strategy request](https://github.com/volcano-sh/volcano/issues/3948) -3. **增强队列调度灵活性**：提供队列级别的调度策略（Queue-level Scheduling Policy）精细化配置能力。用户将能根据不同业务队列的特性、优先级和SLA需求，更灵活地定制其调度行为和资源分配策略。相关issue： - * [volcano supports queue-level scheduling policies](https://github.com/volcano-sh/volcano/issues/3992) -4. **深化生态协同与集成**：我们将积极推进与Kubernetes上游社区及其他云原生项目的协作。例如，推动LWS（Leader Worker Set）与Volcano的集成，以便更好地为分布式应用提供Gang Scheduling能力。相关issue： - * [Support custom scheduler to enable gang scheduling](https://github.com/kubernetes-sigs/lws/issues/407) - 我们热忱欢迎更多优秀的开源项目与Volcano携手，共同构建和繁荣云原生批量计算生态。 -5. **拓展异构硬件支持与合作**：加强与硬件生态伙伴的合作，如昇腾（Ascend）的Device Plugin和DRA Driver的适配与优化，以及与主流GPU厂商在DRA Driver上的协作，确保Volcano能高效、稳定地调度和管理各类前沿异构加速器资源。 -6. **JobFlow工作流能力提升**：持续优化Volcano内置的轻量级工作流引擎JobFlow。计划增强其在复杂作业依赖管理、状态监控、错误处理及用户自定义扩展等方面的能力，为用户提供更强大、更易用的工作流编排解决方案。相关issue： - * [Support JobFlowTemplate CRD](https://github.com/volcano-sh/volcano/issues/4098) - * [Enhance JobFlow Functionality](https://github.com/volcano-sh/volcano/issues/4275) -7. **引入Volcano调度模拟器，提升调度透明度与可测试性**：为提升调度过程的透明度并简化测试验证，Volcano计划引入调度模拟器。这一工具将允许用户在轻量级环境中，通过灵活配置模拟集群状态（节点、Pod、队列配置等），精准复现Volcano核心调度流程——从队列选择、节点过滤与打分到最终绑定。通过输出详尽的调度日志及可选的性能分析，模拟器将极大地便利开发者测试新特性，帮助用户深入理解和验证Volcano在不同场景下的调度行为，并高效评估各类调度策略的实际影响。相关issue： - * [Implement Volcano Scheduler Simulator](https://github.com/volcano-sh/volcano/issues/4276) - -## **社区参与** - -以上 Roadmap 为社区的初步规划。我们欢迎开发者和用户通过以下渠道参与讨论，为 Volcano 的发展贡献新的想法和建议。 - -* **GitHub Issues:** 在 Volcano GitHub 仓库中创建 kind/feature 类型的 Issue，详细说明您的使用场景和功能期望。 -* **社区交流:** 参与社区会议，或在微信交流群/Slack 频道及邮件列表中发起讨论，与开发者和社区成员进行交流。 -* **Roadmap 共建:** 针对我们提出的 Roadmap 或您认为重要的其他特性，欢迎随时提出建议。 - -## **致谢贡献者** - -Volcano v1.12 版本包含了来自43位社区贡献者的上百次代码提交，在此对各位贡献者表示由衷的感谢，贡献者GitHub ID： - -
@daniel-hutao
- - - - - - - - - - - - - - - -
@AdamKorcz@HalfBuddhist@Hcryw
@JackyTYang@JesseStutler@MondayCha
@Monokaix@Poor12@SataQiu
@Wang-Kai@archlitchi@baddoub
@cnmcavoy@co63oc@de6p
@dongjiang1989@ecosysbin@fengruotj
@feyounger@fjq123123@googs1025
@guoqinwill@halcyon-r@hansongChina
@hiwangzhihui@hwdef@kingeasternsun
@linuxfhy@mahdikhashan@mahmut-Abi
@murali1539@ouyangshengjia@qGentry
@sailorvii@sceneryback@sfc-gh-raravena
@wangyang0616@weapons97@xieyanke
@ytcisme@yuyue9284@zedongh
@zhutong196
- diff --git a/content/zh/blog/Volcano-1.13.0-release.md b/content/zh/blog/Volcano-1.13.0-release.md deleted file mode 100644 index a4f19906..00000000 --- a/content/zh/blog/Volcano-1.13.0-release.md +++ /dev/null @@ -1,362 +0,0 @@ -+++ -title = "Volcano v1.13 重磅发布！大模型训练与推理等调度能力全面增强" -description = "新增特性：支持LeaderWorkerSet用于大模型推理场景、新增Cron VolcanoJob、支持基于标签的HyperNode自动发现、新增原生Ray框架支持、新增HCCL插件支持、增强NodeGroup功能、引入ResourceStrategyFit插件、混部能力与操作系统解耦、支持自定义超卖资源名称、支持Kubernetes v1.33等" -subtitle = "" - -date = 2025-09-29 -lastmod = 2025-09-29 -datemonth = "Sep" -dateyear = "2025" -dateday = 29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "新增特性：支持LeaderWorkerSet用于大模型推理场景、新增Cron VolcanoJob、支持基于标签的HyperNode自动发现、新增原生Ray框架支持、新增HCCL插件支持、增强NodeGroup功能、引入ResourceStrategyFit插件、混部能力与操作系统解耦、支持自定义超卖资源名称、支持Kubernetes v1.33等" - -# Add menu entry to sidebar. -linktitle = "Volcano v1.13 重磅发布！大模型训练与推理等调度能力全面增强" -[menu.posts] -parent = "tutorials" -weight = 5 -+++ - -# Volcano v1.13 重磅发布！大模型训练与推理等调度能力全面增强 - -北京时间2025年9月29日，Volcano v1.13 版本[1]正式发布。本次更新在多方面进行了功能增强，为用户提供更完善的云原生批量计算解决方案。 - -## 版本亮点 - -v1.13.0版本的主要更新包括： - -**AI训练与推理增强** - -- [支持LeaderWorkerSet用于大模型推理场景](#支持leaderworkerset用于大模型推理场景) -- [新增Cron VolcanoJob](#新增cron-volcanojob) -- [支持基于标签的HyperNode自动发现](#支持基于标签的hypernode自动发现) -- [新增原生Ray框架支持](#新增原生ray框架支持) -- [新增HCCL插件支持](#新增hccl插件支持) - -**资源管理与调度增强** - -- [引入ResourceStrategyFit插件](#引入resourcestrategyfit插件) - - [按资源类型独立评分策略](#按资源类型独立评分策略) - - [稀缺资源避让(SRA)](#稀缺资源避让sra) -- [增强NodeGroup功能](#增强nodegroup功能) - -**混部能力增强** - -- [混部能力与操作系统解耦](#混部能力与操作系统解耦) -- [支持自定义超卖资源名称](#支持自定义超卖资源名称) - -## 支持LeaderWorkerSet用于大模型推理场景 - -[LeaderWorkerSet (LWS)](https://github.com/kubernetes-sigs/lws) 是一个用于在Kubernetes上部署一组Pod的API。它主要用于解决AI/ML推理工作负载中的多主机推理问题，特别是需要对大型语言模型(LLM)进行分片并在多个节点的多个设备上运行的场景。 - -自开源以来，Volcano一直积极与上下游生态集成，构建了涵盖AI和大数据等批量计算的全面社区生态。在LWS的[v0.7](https://github.com/kubernetes-sigs/lws/releases/tag/v0.7.0)版本中，它原生集成了Volcano的AI调度能力。与新版本的Volcano配合使用时，LWS会自动创建PodGroup，然后由Volcano进行调度和管理，从而为大模型推理场景实现Gang调度等高级能力。 - -展望未来，Volcano将继续扩展其生态集成能力，为更多致力于在Kubernetes上实现分布式推理的项目提供强大的调度和资源管理支持。 - -使用文档：[LeaderWorkerSet With Gang](https://github.com/kubernetes-sigs/lws/tree/main/docs/examples/sample/gang-scheduling)。 - -相关PRs：https://github.com/kubernetes-sigs/lws/pull/496, https://github.com/kubernetes-sigs/lws/pull/498 - -感谢社区开发者：@[JesseStutler](https://github.com/JesseStutler) - -## 新增Cron VolcanoJob - -本次版本引入了对Cron Volcano Job的支持。用户现在可以根据预定义的调度计划定期创建和运行Volcano Job，类似于Kubernetes原生的CronJob，以实现批量计算任务(如AI和大数据)的定期执行。详细功能如下： - -- **定时执行**：使用标准的Cron表达式(`spec.schedule`)定义作业的执行周期。 -- **时区支持**：在`spec.timeZone`中设置时区，确保作业在预期的本地时间执行。 -- **并发策略**：通过`spec.concurrencyPolicy`控制并发行为： - - `AllowConcurrent`：允许多个作业并发执行(默认)。 - - `ForbidConcurrent`：如果前一个作业尚未完成，则跳过当前的调度执行。 - - `ReplaceConcurrent`：如果前一个作业仍在运行，则终止它并启动新的作业。 -- **历史管理**：配置要保留的成功(`successfulJobsHistoryLimit`)和失败(`failedJobsHistoryLimit`)作业历史记录的数量；旧作业会自动清理。 -- **错过调度处理**：`startingDeadlineSeconds`字段允许在一定时间范围内容忍调度延迟；超时被视为错过执行。 -- **状态跟踪**：CronJob状态(`status`)跟踪当前活动的作业、上次调度时间和上次成功完成时间，便于监控和管理。 - -相关PRs：https://github.com/volcano-sh/apis/pull/192, https://github.com/volcano-sh/volcano/pull/4560 - -感谢社区开发者：@[GoingCharlie](https://github.com/volcano-sh/volcano/commits?author=GoingCharlie), @[hwdef](https://github.com/hwdef), @[Monokaix](https://github.com/volcano-sh/volcano/commits?author=Monokaix) - -使用示例：[Cron Volcano Job Example](https://github.com/volcano-sh/volcano/blob/master/example/cronjob/cronjob.yaml)。 - -## 支持基于标签的HyperNode自动发现 - -Volcano在v1.12版本中正式推出了网络拓扑感知调度能力，并率先实现了基于InfiniBand (IB)网络的UFM自动发现机制。然而，对于不支持IB网络或使用其他网络架构(如以太网)的硬件集群，手动维护网络拓扑仍然繁琐。 - -为解决这一问题，新版本引入了**基于标签的HyperNode自动发现机制**。此功能为用户提供了一种通用且灵活的方式来描述网络拓扑，将复杂的拓扑管理任务转化为简单的节点标签管理。 - -该机制允许用户在volcano-controller-configmap中定义拓扑层级与节点标签之间的对应关系。Volcano控制器会定期扫描集群中的所有节点，并根据它们的标签自动执行以下任务： - -- **自动拓扑构建**：根据节点上的一组标签，自动从上到下构建多层HyperNode拓扑结构(例如，机架 -> 交换机 -> 节点)。 -- **动态维护**：当节点标签发生变化，或添加、删除节点时，控制器会自动更新HyperNode的成员和结构，确保拓扑信息与集群状态保持一致。 -- **支持多种拓扑类型**：允许用户同时定义多个独立的网络拓扑，以适应不同的硬件集群(例如，GPU集群、NPU集群)或不同的网络分区。 - -配置示例： - -```yaml -# volcano-controller-configmap.yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: volcano-controller-configmap - namespace: volcano-system -data: - volcano-controller.conf: | - networkTopologyDiscovery: - - source: label - enabled: true - interval: 10m # 发现间隔 - config: - networkTopologyTypes: - # 定义一个名为topology-A的拓扑类型 - topology-A: - # 定义拓扑层级，从上到下排序 - - nodeLabel: "volcano.sh/hypercluster" # 顶层HyperNode - - nodeLabel: "volcano.sh/hypernode" # 中间层HyperNode - - nodeLabel: "kubernetes.io/hostname" # 底层物理节点 -``` - -此功能通过在Volcano控制器的ConfigMap中添加标签源来启用。上述配置定义了一个名为`topology-A`的三层拓扑结构： - -- 顶层(Tier 2)：由`volcano.sh/hypercluster`标签定义。 -- 中间层(Tier 1)：由`volcano.sh/hypernode`标签定义。 -- 底层：物理节点，由Kubernetes内置的`kubernetes.io/hostname`标签标识。 - -当节点被标记如下时，它将被自动识别并分类到拓扑路径`cluster-s4 -> node-group-s0`： - -```yaml -# 节点node-0的标签 -labels: - kubernetes.io/hostname: node-0 - volcano.sh/hypernode: node-group-s0 - volcano.sh/hypercluster: cluster-s4 -``` - -基于标签的网络拓扑自动发现功能提供了出色的通用性和灵活性。它不依赖于特定的网络硬件(如IB)，适用于各种异构集群，并允许用户通过标签灵活定义任意深度的层级结构。它将复杂的拓扑维护任务自动化为简单的节点标签管理，显著降低了运维成本和错误风险。此外，该机制可以动态适应集群节点和标签的变化，实时保持拓扑信息的准确性，无需手动干预。 - -相关PR：https://github.com/volcano-sh/volcano/pull/4629 - -感谢社区开发者：@[zhaoqi612](https://github.com/zhaoqi612) - -使用文档：[HyperNode Auto Discovery](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_hypernode_auto_discovery.md)。 - -## 新增原生Ray框架支持 - -[Ray](https://docs.ray.io/) 是一个开源的统一分布式计算框架，其核心目标是简化从单机到大规模集群的并行计算，特别适合扩展Python和AI应用。为了在Kubernetes上管理和运行Ray，社区提供了KubeRay——一个专为Kubernetes设计的操作器。它充当Kubernetes和Ray框架之间的桥梁，极大地简化了Ray集群和作业的部署和管理。 - -历史上，在Kubernetes上运行Ray工作负载主要依赖于KubeRay Operator。KubeRay在其[v0.4.0版本(2022年发布)](https://docs.ray.io/en/master/cluster/kubernetes/k8s-ecosystem/volcano.html)中集成了Volcano，用于Ray集群的调度和资源管理，解决了分布式训练场景中的资源死锁等问题。随着Volcano新版本的推出，用户现在可以直接通过原生Volcano Job创建和管理Ray集群并提交计算任务。这为Ray用户提供了另一种使用方案，允许他们更直接地利用Volcano的Gang调度、队列管理和公平调度以及作业生命周期管理等能力来运行Ray工作负载。 - -相关PR：https://github.com/volcano-sh/volcano/pull/4581 - -感谢社区开发者：@[Wonki4](https://github.com/Wonki4) - -设计文档：[Ray Framework Plugin Design Doc](https://github.com/volcano-sh/volcano/blob/master/docs/design/distributed-framework-plugins.md)。 - -使用文档：[Ray Plugin User Guide](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_ray_plugin.md)。 - -## 新增HCCL插件支持 - -新版本为Volcano Job添加了HCCL Rank插件(`hcclrank`)，用于在分布式任务中自动为Pod分配HCCL Rank。这包括： - -- Volcano Job的`hcclrank`插件的新实现，支持根据任务类型(master/worker)和索引自动计算HCCL Rank并注入到Pod注解中。 -- 该插件支持自定义master/worker任务名称，允许用户在分布式任务中指定master/worker角色。 - -此功能增强了Volcano对HCCL通信场景(如华为昇腾)的原生支持，便于AI训练任务中Rank的自动管理和分配。 - -相关PR：https://github.com/volcano-sh/volcano/pull/4524 - -感谢社区开发者：@[kingeasternsun](https://github.com/kingeasternsun) - -## 增强NodeGroup功能 - -在层级队列结构中，为每个子队列重复配置与其父队列相同的节点组亲和性(`nodeGroupAffinity`)会导致配置冗余且难以维护。 - -为解决这个问题，Nodegroup插件添加了对层级队列内亲和性继承的支持。启用后，调度器根据以下规则解析队列的有效亲和性： - -1. **优先自身配置**：如果队列定义了`spec.affinity`，则直接使用此配置。 -2. **向上继承**：如果队列没有定义`spec.affinity`，则向上搜索其父队列，并继承最近的祖先队列定义的亲和性配置。 -3. **覆盖能力**：子队列可以通过定义自己的`spec.affinity`来覆盖继承的配置，确保灵活性。 - -此功能允许管理员在父队列(例如部门级别)设置统一的节点组亲和性，所有子队列(例如团队级别)将自动继承此设置，简化管理。 - -对于没有NodeAffinity配置的队列，插件中的"strict"参数控制调度行为。当`strict`设置为`true`(默认值)时，这些队列中的任务无法调度到任何节点。当`strict`设置为`false`时，允许这些任务调度到没有`volcano.sh/nodegroup-name`标签的常规节点。 - -在调度器配置文件的nodegroup插件参数中，设置`enableHierarchy: true`启用层级队列模式，设置`strict: false`配置非严格模式。示例配置如下： - -```yaml -actions: "allocate, backfill, preempt, reclaim" -tiers: -- plugins: - - name: nodegroup - arguments: - enableHierarchy: true # 启用层级支持 - strict: false # 设置为非严格模式，允许队列中的任务调度到没有"volcano.sh/nodegroup-name"标签的节点 -``` - -相关PRs：https://github.com/volcano-sh/volcano/pull/4455 - -感谢社区开发者：@[JesseStutler](https://github.com/JesseStutler), @[wuyueandrew](https://github.com/wuyueandrew) - -NodeGroup设计文档：[NodeGroup Design.](https://github.com/volcano-sh/volcano/blob/master/docs/design/node-group.md) - -NodeGroup使用文档：[NodeGroup User Guide.](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_nodegroup_plugin.md) - -## 引入ResourceStrategyFit插件 - -在Kubernetes原生的`noderesources` fit策略中，只能对所有资源应用单一的聚合(`MostAllocated`)或分散(`LeastAllocated`)策略。这在复杂的异构计算环境(如AI/ML集群)中存在局限性。为满足差异化的调度需求，Volcano引入了增强的`ResourceStrategyFit`插件。 - -该插件现在集成了两个核心功能：按资源类型独立评分策略和稀缺资源避让(SRA)。 - -### 按资源类型独立评分策略 - -此功能允许用户为不同的资源(例如cpu、memory、nvidia.com/gpu)独立指定`MostAllocated`(binpack)或`LeastAllocated`(spread)策略，并为它们分配不同的权重。调度器根据每种资源的独立配置精细计算节点分数。 - -为简化同一系列资源的管理(例如，同一供应商的不同型号GPU)，此功能还支持资源名称的后缀通配符(`*`)匹配。 - -- **语法规则**：仅支持后缀通配符，例如`nvidia.com/gpu/*`。像`*`或`vendor.*/gpu`这样的模式被视为无效。 -- **匹配优先级**：使用"最长前缀匹配"原则。精确匹配具有最高优先级；当不存在精确匹配时，选择具有最长前缀的通配符模式。 - -配置示例：以下配置为特定的V100 GPU型号设置高优先级binpack策略，为所有其他NVIDIA GPU设置通用binpack策略，为CPU资源设置spread策略。还支持Pod级别的资源评分策略配置。 - -```yaml -actions: "enqueue, allocate, backfill, reclaim, preempt" -tiers: -- plugins: - - name: resource-strategy-fit - arguments: - resourceStrategyFitWeight: 10 - resources: - # 精确匹配，最高优先级 - nvidia.com/gpu-v100: - type: MostAllocated - weight: 3 - # 通配符匹配，适用于所有其他NVIDIA GPU - nvidia.com/gpu/*: - type: MostAllocated - weight: 2 - # CPU资源的精确匹配 - cpu: - type: LeastAllocated - weight: 1 -``` - -### 稀缺资源避让(SRA) - -SRA是一种"软"策略，旨在提高昂贵或稀缺资源(如GPU)的整体利用率。它影响节点评分，引导不需要特定稀缺资源的普通任务(例如，仅需CPU的任务)尽可能避免包含这些资源的节点。这有助于为真正需要稀缺资源的任务"保留"稀缺资源节点，从而减少资源争用和任务等待时间。 - -机制： - -1. 用户在配置中定义一组"稀缺资源"(例如`nvidia.com/gpu`)。 -2. 当调度不请求任何定义的稀缺资源的Pod时，SRA策略生效。 -3. 调度器降低拥有这些稀缺资源的节点的分数。节点拥有的稀缺资源类型越多，其分数越低。 -4. 对于请求稀缺资源的Pod，SRA策略不会对其调度决策产生负面影响。 - -配置示例：以下配置将`nvidia.com/gpu`定义为稀缺资源。当调度仅需CPU的任务时，拥有GPU的节点的分数将降低，使任务更有可能调度到没有GPU的节点。 - -```yaml -actions: "enqueue, allocate, backfill, reclaim, preempt" -tiers: -- plugins: - - name: resource-strategy-fit - arguments: - # ... resourceStrategyFit的binpack/spread策略配置 ... - resources: - nvidia.com/gpu: - type: MostAllocated - weight: 2 - cpu: - type: LeastAllocated - weight: 1 - # SRA策略配置 - sra: - enable: true - resources: "nvidia.com/gpu" # 定义稀缺资源列表，逗号分隔 - weight: 10 # SRA策略在总分中的权重 - resourceWeight: - nvidia.com/gpu: 1 # 将nvidia.com/gpu定义为稀缺资源及其权重 -``` - -通过结合ResourceStrategyFit的binpack/spread策略和SRA的避让策略，用户可以实现对异构资源更精细和高效的调度。 - -相关PRs：https://github.com/volcano-sh/volcano/pull/4391, https://github.com/volcano-sh/volcano/pull/4454, https://github.com/volcano-sh/volcano/pull/4512 - -感谢社区开发者：@[LY-today](https://github.com/LY-today), @[XbaoWu](https://github.com/XbaoWu), @[ditingdapeng](https://github.com/ditingdapeng), @[kingeasternsun](https://github.com/kingeasternsun) - -设计文档：[ResourceStrategyFit Design](https://github.com/volcano-sh/volcano/blob/master/docs/design/resource-strategy-fit-scheduling.md) - -使用文档：[ResourceStrategyFit User Guide](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_resource_strategy_fit_plugin.md) - -## 混部能力与操作系统解耦 - -Volcano的混部能力由两部分组成：应用级和内核级。应用级混部为在线和离线工作负载提供统一调度、动态资源超卖、节点压力驱逐等功能。内核级混部涉及在内核级别对CPU、内存和网络等资源的QoS保证，通常需要特定操作系统(如OpenEuler)的支持。在新版本中，Volcano将混部能力与操作系统解耦。对于使用不支持内核级混部的操作系统的用户，他们可以选择使用Volcano的应用级混部能力，实现在线和离线任务的统一调度、动态资源超卖以及高优先级任务保证。 - -具体使用：安装Volcano agent时，指定`--supported-features`参数： - -```shell -helm install volcano . --create-namespace -n volcano-system --set custom.colocation_enable=true --set "custom.agent_supported_features=OverSubscription\,Eviction\,Resources" -``` - -相关PRs：https://github.com/volcano-sh/volcano/pull/4409, https://github.com/volcano-sh/volcano/pull/4630 - -感谢社区开发者：@[ShuhanYan](https://github.com/ShuhanYan), @[Monokaix](https://github.com/Monokaix) - -混部文档：https://volcano.sh/en/docs/colocation/ - -## 支持自定义超卖资源名称 - -Volcano混部Agent添加了参数`--extend-resource-cpu-name`和`--extend-resource-memory-name`，允许用户自定义超卖资源的名称。这支持CPU和内存资源的自定义命名(默认值分别为`kubernetes.io/batch-cpu`和`kubernetes.io/batch-memory`)，增强了设置超卖资源名称的灵活性。 - -具体使用：安装Volcano时，指定`--extend-resource-cpu-name`和`--extend-resource-memory-name`参数： - -```shell -helm install volcano . --create-namespace -n volcano-system --set custom.colocation_enable=true --set custom.agent_extend_resource_cpu_name=example.com/cpu --set custom.agent_extend_resource_memory_name=example.com/memory -``` - -相关PRs：https://github.com/volcano-sh/volcano/pull/4413, https://github.com/volcano-sh/volcano/pull/4630 - -感谢社区开发者：@[ShuhanYan](https://github.com/ShuhanYan), @[Monokaix](https://github.com/Monokaix) - -混部文档：https://volcano.sh/en/docs/colocation/ - -## 新增Kubernetes 1.33支持 - -Volcano版本与Kubernetes社区发布保持同步。v1.13支持最新的Kubernetes v1.33版本，通过全面的UT和E2E测试用例确保功能和可靠性。 - -参与Volcano适配新Kubernetes版本工作，请参考：[adapt-k8s-todo](https://github.com/volcano-sh/volcano/blob/v1.13.0/docs/design/adapt-k8s-todo.md)。 - -相关PR：https://github.com/volcano-sh/volcano/pull/4430 - -感谢社区开发者：@[mahdikhashan](https://github.com/mahdikhashan) - -## **总结：Volcano v1.13.0，持续引领云原生批量计算发展** - -Volcano v1.13.0不仅是技术的提升，更是云原生批量计算领域的持续创新。无论是AI大模型训练与推理、大数据调度，还是资源优化，Volcano v1.13.0都提供了强大的功能和灵活的解决方案。我们相信Volcano v1.13.0将帮助用户在云原生批量计算领域取得更大的成就，开启AI与大数据调度的新篇章！ - -**立即体验Volcano v1.13.0，步入高效计算的新时代！** - -**v1.13.0发布地址：** https://github.com/volcano-sh/volcano/releases/tag/v1.13.0 - -## **致谢** - -Volcano v1.13.0包含了36位社区成员的贡献。衷心感谢所有贡献者： - -| @ElectricFish7 | @philandstuff | @junzebao | -| :------------- | :-------------- | :--------------- | -| @ShuhanYan | @GautamBytes | @coldzerofear | -| @houyuting | @lhlxc | @cyf-2002 | -| @neo502721 | @suyiiyii | @dafu-wu | -| @ditingdapeng | @GoingCharlie | @Wonki4 | -| @zhaoqi612 | @huntersman | @JesseStutler | -| @LY-today | @XbaoWu | @kingeasternsun | -| @Monokaix | @wuyueandrew | @mahdikhashan | -| @bibibox | @archlitchi | @guoqinwill | -| @ouyangshengjia| @Poor12 | @dongjiang1989 | -| @zhifei92 | @halcyon-r | @Xu-Wentao | -| @hajnalmt | @kevin-wangzefeng| @linuxfhy | diff --git a/content/zh/blog/Volcano-1.7.0-release.md b/content/zh/blog/Volcano-1.7.0-release.md deleted file mode 100644 index c431ec81..00000000 --- a/content/zh/blog/Volcano-1.7.0-release.md +++ /dev/null @@ -1,161 +0,0 @@ -+++ -title = "Volcano v1.7.0正式发布" -description = "新增特性：Pytorch Job强化插件功能、Ray on Volcano、增强Volcano对Kubernetes通用服务的调度能力、支持Volcano的多架构镜像、优化队列状态信息等" -subtitle = "" - -date = 2023-01-12 -lastmod = 2023-01-12 -datemonth = "Jan" -dateyear = "2023" -dateday = 12 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "新增特性：Pytorch Job强化插件功能、Ray on Volcano、增强Volcano对Kubernetes通用服务的调度能力、支持Volcano的多架构镜像、优化队列状态信息等" - -# Add menu entry to sidebar. -linktitle = "Volcano v1.7.0正式发布" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ - -

-北京时间2023年1月9日，Volcano社区v1.7.0版本正式发布。此次版本增加了以下新特性： - -- **Pytorch Job强化功能插件** -- **Ray on Volcano** -- **增强Volcano对Kubernetes通用服务的调度能力** -- **支持Volcano的多架构镜像** -- **优化队列状态信息等** - -{{
}} -Volcano是业界首个云原生批量计算项目，于2019年6月在上海KubeCon正式开源，并在2020年4月成为CNCF官方项目。2022年4月，Volcano正式晋级为CNCF孵化项目。Volcano社区开源以来，受到众多开发者、合作伙伴和用户的认可和支持。截止目前，累计有490+全球开发者向项目贡献了代码。 - -### Volcano v1.7.0 关键特性介绍 - -#### 1. Pytorch Job 强化功能插件 -Pytorch是当下最流行的AI框架之一，已被广泛应用于计算机视觉，自然语言处理等深度学习领域，并且已有越来越多的用户开始以容器化的方式在Kubernetes上运行Pytorch来获得更高资源利用率和并行效率。 - -本次1.7版本提供了Pytorch Job强化功能插件，免去了容器端口、MASTER_ADDR、MASTER_PORT、WORLD_SIZE、RANK环境变量等众多繁琐的手动配置环节，让用户可以用极简的方式在Volcano上运行Pytorch Job。 - -Volcano社区当前提供了TensorFlow、MPI和Pytorch等Job强化功能插件，帮助用户更简洁高效的运行对应的训练框架和计算作业。 - -此外，Volcano还提供了Job插件的扩展开发框架，满足高阶用户针对复杂场景定制Job插件的需求。 - - -设计文档：[Pytorch-plugin](https://github.com/volcano-sh/volcano/blob/master/docs/design/distributed-framework-plugins.md#pytorch-plugin)
-用户手册：[Pytorch-plugin-user-guide](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_pytorch_plugin.md#pytorch-plugin-user-guide)
-Issue：[#2292](https://github.com/volcano-sh/volcano/issues/2292)
- - -#### 2. Ray on Volcano -Ray 是一个用于扩展 AI 和 Python 应用程序的统一框架，可以在任何机器、集群、云提供商和 Kubernetes 上运行，并具有不断发展完善的社区生态。 - -当前，机器学习工作负载的计算密集度越来越高，单节点开发环境已无法满足训练任务在资源上的需求，Ray可以将同一套代码从单机运行无缝扩展到集群中运行。Ray是面向通用场景进行的设计，可以高效运行任何类型的工作负载。 - -该特性实现Volcano和Ray的协同工作，使用Volcano为Ray提供批量调度功能。目前该特性已在[KubeRay v0.4](https://github.com/ray-project/kuberay/releases/tag/v0.4.0)版本正式发布。 - -用户手册：[KubeRay-integration-with-Volcano](https://ray-project.github.io/kuberay/guidance/volcano-integration/#kuberay-integration-with-volcano)
-Issue：[#2429](https://github.com/volcano-sh/volcano/issues/2429)， [#213](https://github.com/ray-project/kuberay/issues/213)
- -#### 3. 增强Volcano对Kubernetes通用服务的调度能力 -在不同的业务场景下，调度器发挥的价值与能力各有不同。比如，在批量计算场景中，Volcano的调度策略和能力更为丰富；通用服务调度场景中，Kubernetes默认调度器的能力更为均衡。但用户的集群中通常不会只运行一种类型的任务，在既有批量计算任务运行，又需要调度通用服务的场景下，如何兼顾不同类型任务的调度是一项值得研究的工作。 - -从1.7版本开始，Volcano提供对Kubernetes 默认调度器完全兼容的能力，可用于调度管理长期运行的服务。通过本次功能强化，用户可以使用 Volcano 统一调度长时间运行的服务和批处理工作负载。 - -**重点增强项如下：** -
    -
  • Volcano的 scheduler 和 webhook 支持多调度器
    • -
  • 支持 NodeVolumeLimits 插件
    • -
  • 支持 VolumeZone 插件
    • -
  • 支持 PodTopologySpread 插件
    • -
  • 支持 SelectorSpread插件
    • -
- -另外，本次版本将对Kubernetes的支持升级到了v1.25。 - -Issue： [#2394](https://github.com/volcano-sh/volcano/issues/2394)，[#2510](https://github.com/volcano-sh/volcano/issues/2510) - -#### 4. 支持Volcano的多架构镜像 -该特性通过交叉编译实现一键化编译Volcano多架构镜像功能，比如，可以在amd64架构机器上，一键化编译amd64和arm64架构的Volcano基础镜像，并上传至镜像仓库。用户安装部署时，会根据自身环境中机器架构自动选择合适的镜像运行，对开发者和用户更友好。 - -使用手册：[building-docker-images](https://github.com/volcano-sh/volcano/blob/master/docs/development/development.md#building-docker-images)
-Issue：[#2435](https://github.com/volcano-sh/volcano/pull/2435)
- -#### 5. 优化队列状态信息 -该特性在队列状态信息中增加已分配资源实时统计功能，通过该功能，用户可以实时查看队列的资源分配情况，方便管理员动态规划资源，使集群资源得到最大化利用。 - -Volcano通过队列的方式对集群资源进行分配管理，使用Capability字段来限制每个队列的资源上限，这是我们进行队列资源分配的硬约束。 - -当前，在集群运行过程中，用户无法清楚的了解到队列中已分配资源和Capability中空闲资源的详细信息，有可能会在队列空闲资源不足时提交大量工作负载，导致作业挂起和autoScaler非预期的触发集群扩容，从而增加云资源的使用成本。通过该特性中新增加的状态信息，用户可以更清晰有效的管理集群资源，控制使用成本。 - -Issue：[#2571](https://github.com/volcano-sh/volcano/issues/2571) - -### 致谢贡献者 -Volcano v1.7.0 版本包含了来自29位贡献者的数百次代码提交，在此对各位贡献者表示由衷的感谢： - -**贡献者 GitHub ID：**
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
@xiaoxubeii@jsolbrig@Yikun
@tgaddair@william-wang@elinx
@Abirdcfly@xiaoanyunfei@qiankunli
@wpeng102@waiterQ@hwdef
@WingkaiHo@Monokaix@kerthcet
@WulixuanS@autumn0207@jinzhejz
@lucming@jimoosciuc@LY-today
@dontan001@wangyang0616@Akiqqqqqqq
@zhoumingcheng@jiangkaihua@Thor-wl
@ccchenjiahuan@zhifanggao 
- -#### 相关链接 -Release note：[v1.7.0](https://github.com/volcano-sh/volcano/releases/tag/v1.7.0)
-Branch：[release-1.7](https://github.com/volcano-sh/volcano/tree/release-1.7)
- -### 深入了解Volcano -Volcano云原生批量计算项目主要用于 AI、大数据、基因、渲染等诸多高性能计算场景，对主流通用计算框架均有很好的支持。社区已吸引2.6万+全球开发者，并获得2.8k Star和670+ Fork，参与贡献企业包括华为、AWS、百度、腾讯、京东、小红书等。目前，Volcano在人工智能、大数据、基因测序等海量数据计算和分析场景已得到快速应用，已完成对Spark、Flink、Tensorflow、PyTorch、Argo、MindSpore、Paddlepaddle 、Kubeflow、MPI、Horovod、mxnet、KubeGene、Ray等众多主流计算框架的支持，并构建起完善的上下游生态。 \ No newline at end of file diff --git a/content/zh/blog/Volcano-1.8.2-release.md b/content/zh/blog/Volcano-1.8.2-release.md deleted file mode 100644 index 477f5a8b..00000000 --- a/content/zh/blog/Volcano-1.8.2-release.md +++ /dev/null @@ -1,263 +0,0 @@ -+++ -title = "Volcano v1.8.2正式发布" -description = "新增特性：支持vGPU调度及隔离、支持vGPU和用户自定义资源的抢占能力、新增JobFlow工作流编排引擎、节点负载感知调度与重调度支持多样化的监控系统、优化Volcano对通用服务调度的能力、优化Volcano charts包的发布与归档等" -subtitle = "" - -date = 2024-01-31 -lastmod = 2024-01-31 -datemonth = "Jan" -dateyear = "2024" -dateday = 31 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "新增特性：支持vGPU调度及隔离、支持vGPU和用户自定义资源的抢占能力、新增JobFlow工作流编排引擎、节点负载感知调度与重调度支持多样化的监控系统、优化Volcano对通用服务调度的能力、优化Volcano charts包的发布与归档等" - -# Add menu entry to sidebar. -linktitle = "Volcano v1.8.2正式发布" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ - - -北京时间2024年1月9日，Volcano 社区 v1.8.2 版本正式发布，此次版本增加了以下新特性： - -- **支持vGPU调度及隔离** - -- **支持vGPU和用户自定义资源的抢占能力** - -- **新增JobFlow工作流编排引擎** - -- **节点负载感知调度与重调度支持多样化的监控系统** - -- **优化Volcano对通用服务调度的能力** - -- **优化Volcano charts包的发布与归档** - -{{
}} -Volcano是业界首个云原生批量计算项目，于2019年6月在上海 KubeCon 正式开源，并在2020年4月成为 CNCF 官方项目。2022年4月，Volcano 正式晋级为CNCF 孵化项目。Volcano 社区开源以来，受到众多开发者、合作伙伴和用户的认可和支持。截至目前，累计有600+全球开发者参与社区贡献。 - -### Volcano v1.8.2 关键特性介绍 - -#### 支持vGPU调度及隔离 -自 ChatGPT 爆火之后，AI大模型的研发层出不穷，不同种类的AI大模型也相继推出，由于其庞大的训练任务需要大量算力，以 GPU 为核心的算力供给已成为大模型产业发展的关键基础设施。在实际使用场景中，用户对于 GPU 资源的使用存在资源利用率低，资源分配不灵活等痛点问题，必须采购大量冗余的异构算力才能满足业务需求，而异构算力本身成本高昂，为企业的发展带来了很大的负担。 -从 1.8版本开始，Volcano 为可共享设备（GPU、NPU、FPGA...）提供一个抽象的通用框架，开发者可以基于该框架自定义多种类型的共享设备；当前，Volcano 已基于该框架实现 GPU 虚拟化特性，支持 GPU 设备复用、资源隔离等能力，详情如下： - -- GPU共享：每个任务可以申请使用一个 GPU 卡的部分资源，GPU 卡可以在多个任务之间共享。 - -- 设备显存控制：GPU 可以按照设备显存分配（比如：3000M）或者按比例分配（比如：50%），实现 GPU 虚拟化资源隔离能力。 - -关于 vGPU 的更多信息，请参考： - -- 如何使用 vGPU 功能： - - https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_vgpu.md - -- 如何增加新的异构算力共享策略： - - https://github.com/volcano-sh/volcano/blob/master/docs/design/device-sharing.md - -#### 支持vGPU和用户自定义资源的抢占能力 -当前 Volcano 支持 CPU、Memory 等基础资源抢占功能，对于 GPU 资源和用户基于 Volcano 框架二次开发调度插件，并自主管理的资源（如：NPU、网络资源等）尚不能很好的支持抢占能力。 -在1.8版本中，Volcano 对节点过滤相关处理（ PredicateFn 回调函数）进行重构，返回结果中增加 Status 类型，用于标识在调度、抢占等场景下，当前节点是否满足作业下发条件。GPU 抢占功能已基于优化后的框架实现发布，用户基于Volcano 进行二次开发的调度插件可以结合业务场景适配升级。 -在1.8.2版本中，Volcano支持节点CSI挂盘数量和节点Pods数量的抢占。 - -关于支持扩展资源抢占的更多信息，请参考： - -https://github.com/volcano-sh/volcano/pull/2916 - -#### 新增JobFlow工作流编排引擎 -工作流编排引擎广泛应用于高性能计算、AI 生物医药、图片处理、美颜、游戏AGI、科学计算等场景，帮助用户简化多个任务并行与依赖关系的管理，大幅度提升整体计算效率。 -JobFlow 是一种轻量化的任务流编排引擎，专注于 Volcano 的作业编排，为Volcano 提供作业探针、作业完成依赖，作业失败率容忍等多样化作业依赖类型，并支持复杂的流程控制原语，具体能力如下： - -- 支持大规模作业管理以及复杂任务流编排 - -- 支持实时查询到所有关联作业的运行情况以及任务进度 - -- 支持作业自动运行、定时启动释放人力成本 - -- 支持不同任务可以设置多种动作策略，当任务满足特定条件时即可触发对应动作，如超时重试、节点故障漂移等 - -JobFlow 任务运行演示如下： - -
{{
}}
- -关于 JobFlow 的更多信息，请参考： - -https://github.com/volcano-sh/volcano/blob/master/docs/design/jobflow/README.md - -#### 节点负载感知调度与重调度支持多样化的监控系统 -Kubernetes 集群状态随着任务的创建和结束实时变化，在某些场景（如：增加、删除节点，Pod、Node 的亲和性改变，作业生命周期动态变化等），出现集群节点间资源利用率不均衡，节点性能瓶颈掉线等问题，此时基于真实负载的调度与重调度可以帮助我们解决上述问题。 -Volcano 1.8版本之前，真实负载调度与重调度的指标获取仅支持 Prometheus，从1.8版本开始，Volcano 优化监控指标获取框架，新增 ElasticSearch 监控系统支持，并支持以较小适配工作量平滑对接更多类型监控系统。 - -关于支持多种监控系统的更多信息，请参考： - -- 基于节点负载感知调度： - - https://github.com/volcano-sh/volcano/blob/master/docs/design/usage-based-scheduling.md - -- 重调度： - - https://github.com/volcano-sh/volcano/blob/master/docs/design/rescheduling.md - -#### 优化Volcano对微服务调度的能力 - -##### 增加Kubernetes默认调度器插件开关 -Volcano 是一个统一的融合调度系统，不仅支持 AI、BigData 等计算类作业，也支持微服务工作负载，兼容 Kubernetes 默认调度器的 PodTopologySpread、VolumeZone、VolumeLimits、NodeAffinity、PodAffinity等调度插件，Kubernetes 默认调度插件能力在 Volcano 中默认开启。 -自 Volcano 1.8 版本开始，Kubernetes 默认调度插件可以通过配置文件的方式自由选择打开和关闭，默认全部打开，如果选择关闭部分插件，比如：关闭PodTopologySpread 和 VolumeZone插件，可以在 predicate 插件中将对应的值设置为 false，配置如下： - -```yaml -actions: "allocate, backfill, preempt" -tiers: -- plugins: - - name: priority - - name: gang - - name: conformance -- plugins: - - name: drf - - name: predicates - arguments: - predicate.VolumeZoneEnable: false - predicate.PodTopologySpreadEnable: false - - name: proportion - - name: nodeorder -``` - -更多信息，请参考： - -https://github.com/volcano-sh/volcano/issues/2748 - -##### 增强ClusterAutoscaler兼容性 -在 Kubernetes 平台中，Volcano 除了作为批量计算业务的调度器之外，也被越来越多的用作通用服务的调度器。Node 水平伸缩（ClusterAutoscaler）是Kubernetes 的核心功能之一，在面对用户业务量激增和节省运行成本方面发挥重要作用。Volcano 优化作业调度等相关逻辑，增强与 ClusterAutoscaler 的兼容互动能力，主要为以下两个方面： -调度阶段进入 pipeline 状态的 pod 及时触发扩容 -候选节点分梯度打分，减少集群 terminating pod 对调度负载的影响，避免pod 进入无效 pipeline 状态，从而导致集群误扩容 - -更多信息，请参考： - -https://github.com/volcano-sh/volcano/issues/3000 -https://github.com/volcano-sh/volcano/issues/2782 - -##### 精细化管理Node资源，增强韧性 -当节点中由于某种原因比如 device-plugin 上报信息异常，出现节点的某种资源总量小于已分配资源量时，Volcano 认为该节点数据不一致，会隔离节点，停止向该节点调度任何新的工作负载。在1.8版本中，对于节点资源进行精细化管理，比如：当节点的 GPU 总资源容量小于已分配资源量时，申请 GPU 资源的 pod 禁止再调度至该节点，申请非 GPU 资源的作业，将仍然允许正常向该节点调度。 - -更多信息，请参考： - -https://github.com/volcano-sh/volcano/issues/2999 - -#### 优化Volcano charts包的发布与归档 -随着 Volcano 在用户越来越多的生产环境和云环境中使用，简洁标准的安装动作至关重要。自1.8版本开始，Volcano 优化 charts 包发布归档动作，标准化安装使用流程，并完成历史版本（v1.6、v1.7）向新 helm 仓库的迁移，使用方式如下： - -- 添加 Volcano charts 仓地址 -```shell -helm repo add volcano-sh https://volcano-sh.github.io/helm-charts -``` - -- 查询所有可安装的 Volcano 版本 -```shell -helm search repo volcano -l -``` - -- 安装最新版 Volcano -```shell -helm install volcano volcano-sh/volcano -n volcano-system --create-namespace -``` - -- 安装指定版本 Volcano，比如：1.8.2 -```shell -helm install volcano volcano-sh/volcano -n volcano-system --create-namespace --version 1.8.2 -``` - - -关于 Volcano charts 包的更多信息，请参考： - -https://github.com/volcano-sh/helm-charts - -### 致谢贡献者 - -Volcano 1.8.2 版本包含了来自 33 位贡献者的数百次代码提交，在此对各位贡献者表示由衷的感谢： - -**贡献者 GitHub ID：**
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
@shaobo76@william-wang@gengwg
@kingeasternsun@Aakcht@waiterQ
@Shoothzj@hwdef@halegreen
@wulixuan@Monokaix@medicharlachiranjeevi
@WulixuanS@rayoluo@lowang-bh
@gj199575@noyoshi@Tongruizhe
@jinzhejz@Cdayz@Mufengzhe
@renwenlong-github@wangyang0616@jiamin13579
@zbbkeepgoing@jiangkaihua@z2Zhang
@archlitchi@lixin963@xiao-jay
@Yanping-io@Lily922@shusley244
- -**参考链接** - -Release note: v1.8.0 - -https://github.com/volcano-sh/volcano/releases/tag/v1.8.0 - -Release note: v1.8.1 - -https://github.com/volcano-sh/volcano/releases/tag/v1.8.1 - -Release note: v1.8.2 - -https://github.com/volcano-sh/volcano/releases/tag/v1.8.2 - -Branch：release-1.8 - -https://github.com/volcano-sh/volcano/tree/release-1.8 - -### 深入了解Volcano - -Volcano 云原生批量计算项目主要用于 AI、大数据、基因、渲染等诸多高性能计算场景，对主流通用计算框架均有很好的支持。社区已吸引5.8万+全球开发者，并获得3.5k+ Star 和800+ Fork，参与贡献企业包括华为、AWS、百度、腾讯、京东、小红书、博云、第四范式等。目前，Volcano在人工智能、大数据、基因测序等海量数据计算和分析场景已得到快速应用，已完成对 Spark、Flink、Tensorflow、PyTorch、Argo、MindSpore、Paddlepaddle 、Kubeflow、MPI、Horovod、mxnet、KubeGene、Ray 等众多主流计算框架的支持，并构建起完善的上下游生态。 \ No newline at end of file diff --git a/content/zh/blog/Volcano-1.9.0-release.md b/content/zh/blog/Volcano-1.9.0-release.md deleted file mode 100644 index cbc2d11c..00000000 --- a/content/zh/blog/Volcano-1.9.0-release.md +++ /dev/null @@ -1,217 +0,0 @@ -+++ -title = "Volcano v1.9.0正式发布" -description = "新增特性：支持弹性队列容量capacity调度、支持队列与节点间的亲和调度、Volcano支持Kubernetes v1.29、GPU共享支持节点打分调度、增强scheduler metrics指标、新增License合规性检查、提升调度稳定性等" -subtitle = "" - -date = 2024-05-21 -lastmod = 2024-05-21 -datemonth = "May" -dateyear = "2024" -dateday = 21 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "新增特性：支持弹性队列容量capacity调度、支持队列与节点间的亲和调度、Volcano支持Kubernetes v1.29、GPU共享支持节点打分调度、增强scheduler metrics指标、新增License合规性检查、提升调度稳定性等" - -# Add menu entry to sidebar. -linktitle = "Volcano v1.9.0正式发布" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ - - -北京时间2024年5月21日，Volcano 社区 v1.9.0 版本正式发布，此次版本增加了以下新特性： - -- **支持弹性队列容量capacity调度** - -- **支持队列与节点间的亲和调度** - -- **Volcano支持Kubernetes v1.29** - -- **GPU共享支持节点打分调度** - -- **增强scheduler metrics指标** - -- **新增License合规性检查** - -- **提升调度稳定性** - -{{
}} -Volcano是业界首个云原生批量计算项目，于2019年6月在上海 KubeCon 正式开源，并在2020年4月成为 CNCF 官方项目。2022年4月，Volcano 正式晋级为CNCF 孵化项目。Volcano 社区开源以来，受到众多开发者、合作伙伴和用户的认可和支持。截至目前，累计有600+全球开发者参与社区贡献。 - -### Volcano v1.9.0 关键特性介绍 - -#### 支持弹性队列容量capacity调度 - -Volcano现在使用proportion插件来进行队列管理，用户可以设置队列的guarantee、capability等字段来设置队列的预留资源和容量上限。并通过设置队列的weight值来实现集群内的资源共享，队列按照weight值按比例划分集群资源，但这种队列管理方式存在以下问题： - -- 队列划分的资源容量通过权重体现，不够直观。 -- 队列内的所有资源使用相同的比例进行划分，不能为队列的每一维资源单独设置容量。 - -基于以上考虑，Volcano实现了新的队列弹性容量管理能力，它支持： - -- 用户可以直接为队列设置每一维度资源的容量，而不是设置weigh值来实现。 -- 基于deserved的队列弹性容量调度，支持队列的资源共享和回收。 - -比如在AI大模型训练中分别为队列中不同的GPU型号如A100和V100，设置不同的资源容量。同时在集群资源空闲时，队列可以复用其他空闲队列的资源，并在需要时进行资源回收，直到回收到用户为队列设置的资源容量为止，即应得资源量deserved，从而实现弹性容量能力。 - -使用改功能时需要设置队列的deserved字段，为每一维资源设置应得资源量。同时需要在调度配置中打开capacity插件，并关闭proportion插件。 - -```yaml -apiVersion: scheduling.volcano.sh/v1beta1 -kind: Queue -metadata: - name: demo-queue -spec: - reclaimable: true - deserved: # set the deserved field. - cpu: 64 - memeory: 128Gi - nvidia.com/a100: 40 - nvidia.com/v100: 80 -``` - -队列弹性容量调度的完整使用例子，请参考：[How to use capacity plugin](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_capacity_plugin.md). - -关于弹性队列容量设计文档，请参考[Capacity scheduling Design](https://github.com/volcano-sh/volcano/blob/master/docs/design/capacity-scheduling.md). - -#### 支持队列与节点间的亲和调度 - -队列通常关联着公司内的部门，而不同部门通常需要使用不同的异构资源类型，比如大模型训练团队需要使用NIVDIA的Tesla GPU，而推荐团队需要使用AMD的GPU，当用户提交作业到队列时，需要根据队列的属性将作业自动调度到对应资源类型的节点上。 - -为此Volcano实现了队列和节点的亲和调度能力，用户只需在队列的affinity字段设置需要亲和的节点标签，Volcano会自动将提交到当前队列的作业调度到队列关联的节点上，用户无需单独设置作业的亲和性，而只需统一设置队列的亲和性，提交到队列的作业都会根据队列与节点的亲和性将作业调度到对应的节点。 - -该特性同时支持硬亲和、软亲和、反亲和调度，使用时需要为节点设置key为`volcano.sh/nodegroup-name`的标签，然后设置队列的affinity字段，指定硬亲和、软亲和和反亲和的标签值。例如如下的队列设置，表示提交到该队列的作业需要调度到标签值为groupname1和groupname2的节点，并优先调度到标签值为groupname2的节点，同时，作业不能调到到标签值为groupname3和groupname4的节点，当资源不足时则也可以调度到标签值为groupname3的节点上。 - -```yaml -apiVersion: scheduling.volcano.sh/v1beta1 -kind: Queue -metadata: - name: default - spec: - reclaimable: true - weight: 1 - affinity: # added field - nodeGroupAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - - - - preferredDuringSchedulingIgnoredDuringExecution: - - - nodeGroupAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - - - - preferredDuringSchedulingIgnoredDuringExecution: - - -``` - -该功能对应的调度插件名为nodegroup，完整使用例子请参考：[How to use nodegroup plugin](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_nodegroup_plugin.md). - -详细设计文档请参考：[The nodegroup design](https://github.com/volcano-sh/volcano/blob/master/docs/design/node-group.md). - -#### GPU共享功能支持节点打分调度 - -GPU共享是Volcano v1.8版本推出的GPU共享与隔离方案，提供GPU共享、设备显存控制能力，以提升AI训练推理场景下GPU资源利用率低的问题。v1.9在该功能基础上新增了对GPU节点打分的策略，从而可以在作业分配时选择最优的节点，进一步提升资源利用率，用户可以设置不同的打分策略。目前支持以下两种策略： - -- Binpack：提供GPU卡粒度的binpack算法，优先把一个节点上的已经分配了资源的GPU卡占满，避免资源碎片和浪费。 - - -- Spread：优先使用空闲的GPU卡而不是已经分配了资源的共享卡。 - - -详细使用文档请参考：[How to use gpu sharing](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_gpu_sharing.md) - -#### Volcano支持Kubernetes v1.29 - -Volcano版本紧跟Kubernetes社区版本节奏，对Kubernetes的每个大的基数版本都进行支持，目前最新支持的版本为v1.29，并运行了完整的UT、E2E用例，保证功能和可靠性。如果您想参与Volcano适配Kubernetes新版本的开发工作，请参考：https://github.com/volcano-sh/volcano/pull/3459 进行社区贡献。 - -#### 增强scheduler metrics指标 - -Volcano使用了client-go客户端和Kubernetes交互，尽管客户端可以设置QPS来避免请求被限流，但是客户端实际使用的QPS到底达到了多少却很难观察到，为了实时观测到客户端请求的频率，Volcano新增了client-go客户端的metrics指标，用户可以通过访问metrics接口，查看GET、POST等请求在每秒钟的请求数量，从而确定每秒钟实际使用的QPS，以此决定是否需要调整客户端设置的QPS。同时client-go的相关指标还包括客户端证书轮转周期统计、每个请求的response size统计等。 - -用户可以使用curl http://$volcano_scheduler_pod_ip:8080/metrics 来获取volcano scheduler的所有详细指标。 - -详细PR见：[[feat\] Add rest client metrics by Monokaix · Pull Request #3274 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3274) - -#### 新增License合规性检查 - -为了增强Volcano社区开源license合规治理规范，避免引入传染性开源协议，规避潜在风险，Volcano社区引入了开源license合规检查工具，所谓传染性协议指的是使用了该协议作为开源许可的软件在修改、使用、复制之后生成的衍生作品，也必须以该协议进行开源。开发者提交的PR会引入的三库如果包含了传染性开源协议比如GPL，LGPL等，CI门禁会进行拦截，开发者需要将三方库替换为松自由软件许可协议比如MIT、Apache 2.0，BSD等，以通过开源license合规检查。 - -#### 提升调度稳定性 - -Volcano v1.9.0版本在抢占、调度失败重试、避免内存泄露、安全性增强等方面做了较多优化，具体内容包括： - -- 修复极端情况下deployment频繁扩缩容导致的pod无法调度的问题，详见PR：[[cherry-pick for release-1.9\]fix PodGroup being incorrectly deleted due to frequent creation and deletion of pods by guoqinwill · Pull Request #3376 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3376) - - -- 修复Pod抢占问题：详见PR：[ignore PredicateFn err info for preempt & reclaim scheduler plugin by LivingCcj · Pull Request #3458 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3458) - - -- 优化Pod调度失败重试机制：详见PR：[fix errTask channel memory leak by bibibox · Pull Request #3435 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3435) - - -- metrics指标优化：详见PR：[Fix queue metrics when there are no jobs in it by Monokaix · Pull Request #3463 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3463) - - -- 安全性增强：详见PR：[Remove list secret in controller ClusterRole by lekaf974 · Pull Request #3449 · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/pull/3449) - -### 致谢贡献者 - -Volcano 1.9.0 版本包含了来自多位贡献者的代码提交，在此对各位贡献者表示由衷的感谢： - -**贡献者 GitHub ID：**
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
@daniel-hutao@wuyueandrew@googs1025
@7sunarni@flyingfang@LivingCcj
@guoqinwill@panoswoo@william-wang
@lekaf974@yangqz@lowang-bh
@loheagn@hwdef@archlitchi
@Lily922@bibibox@Monokaix
@belo4ya
- -**参考链接** - -Release note: v1.9.0 - -https://github.com/volcano-sh/volcano/releases/tag/v1.9.0 - -Branch：release-1.9 - -https://github.com/volcano-sh/volcano/tree/release-1.9 - -### 深入了解Volcano - -Volcano 云原生批量计算项目主要用于 AI、大数据、基因、渲染等诸多高性能计算场景，对主流通用计算框架均有很好的支持。社区已吸引5.8万+全球开发者，并获得3.5k+ Star 和800+ Fork，参与贡献企业包括华为、AWS、百度、腾讯、京东、小红书、博云、第四范式等。目前，Volcano在人工智能、大数据、基因测序等海量数据计算和分析场景已得到快速应用，已完成对 Spark、Flink、Tensorflow、PyTorch、Argo、MindSpore、Paddlepaddle 、Kubeflow、MPI、Horovod、mxnet、KubeGene、Ray 等众多主流计算框架的支持，并构建起完善的上下游生态。 diff --git a/content/zh/blog/Volcano-community-co-construction-program.md b/content/zh/blog/Volcano-community-co-construction-program.md deleted file mode 100644 index 61b9abe8..00000000 --- a/content/zh/blog/Volcano-community-co-construction-program.md +++ /dev/null @@ -1,113 +0,0 @@ -+++ -title = "Volcano 社区共建计划" -description = "华为云携手11家合作伙伴启动Volcano社区共建计划" -subtitle = "" - -date = 2023-08-11 -lastmod = 2023-08-11 -datemonth = "Aug" -dateyear = "2023" -dateday = 11 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Tutorials"] -summary = "华为云携手11家合作伙伴启动Volcano社区共建计划" - -# Add menu entry to sidebar. -linktitle = "华为云携手11家合作伙伴启动Volcano社区共建计划" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ - - 随着人工智能技术的高速发展，以及大语言模型的推广应用，各行各业对智能算力的需求呈爆发式增长，除了AI芯片等硬件设备的支持，对于AI任务的高性能智能调度也是重中之重。 - -{{
}} - - - - -Volcano是业界首个云原生批量计算项目，2019年由华为云捐献给云原生计算基金会（CNCF），也是CNCF首个和唯一的孵化级容器批量计算项目。主要包含以下能力： - -- 面向人工智能、大数据、HPC场景提供统一的高性能作业管理，支持丰富的高阶调度策略，包括在离线统一调度、AI弹性训练调度、SLA、作业拓扑、Fairness、负载感知、重调度、抢占、回收等； -- 对Spark、Flink、Pytorch、MPI、TensorFlow等工作负载实现统一生命周期管理，作业依赖、作业内任务依赖管理； -- 在细粒度资源管理方面，Volcano支持min-max队列资源管理，提供队列资源预留、多租户资源租借/抢占的动态资源共享等能力； -- 在异构资源管理方面，Volcano实现 x86、Arm、GPU、昇腾、昆仑等多元算力的统一调度，并提供CPU/GPU资源的精细化调度能力，用户可根据业务需求灵活搭配资源，实现最高性价比。 - -Volcano目前已吸引 **5.8万+全球开发者**，并获得 **3.2k Star** 和 **730+ Fork**，参与贡献企业包括华为、AWS、IBM、百度、腾讯、京东、小红书、第四范式、博云、DaoCloud、锐天投资、七牛云、银清科技、字节跳动、快手、云知声、Infosys、Visa、网易、Red Hat、金山云、浪潮、中兴、Oracle、爱奇艺等。 - -Volcano已实现 **50+生产落地案例**，广泛分布于互联网、先进制造、金融、生命科学、科研、自动驾驶、医药等行业，覆盖人工智能、大数据、基因测序、渲染等海量数据计算和分析场景。主要用户包括：腾讯、亚马逊、荷兰ING银行、百度、小红书、滴滴、360、爱奇艺、中科类脑、鹏程实验室、Curise、理想汽车、云知声、喜马拉雅、唯品会、希望组、BOSS直聘等，**随着Volcano社区的生态圈不断扩大，越来越多的用户表达了加入社区的强烈意愿**，为帮助用户快速融入社区，加速落地实践，共同打造繁荣的社区生态， **华为云携手11家合作伙伴启动Volcano社区共建计划**。 - - -**华为云开源业务总经理邓明昆表示：** “云原生批量计算项目Volcano自2019年6月开源以来，在人工智能、大数据、基因测序、渲染、转码、音视频、金融等领域得到越来越广泛的应用，一批行业标杆用户不仅积极地推动Volcano落地生产环境，也基于自身实践反哺社区，华为云希望携手合作伙伴启动Volcano社区共建计划，共同打造繁荣的社区生态，帮助更多企业加速云原生的进程。” - -**首批加入的合作伙伴有：百度、博云、第四范式、唯品会、锐天投资、中科类脑、品览、360、网易数帆、喜马拉雅、BOSS直聘** - -
{{
}}
- - -**百度飞浆开源技术生态负责人 周倜表示：** “为了提升飞桨框架的计算效率，飞桨团队和Volcano团队联合发布PaddlePaddle on Volcano方案。作为一个面向高性能计算场景的平台，Volcano弥补了Kubernetes 在机器学习、深度学习、HPC、大数据计算等场景下的基本能力缺失，并在原有Kubernetes能力基础上对计算任务的批量创建及生命周期管理、Fair-share调度等方面做了增强，从而满足飞桨对资源创建，资源调度的基本要求。” - - -**博云PAAS产品线总经理 赵安全表示：** “博云高性能算力解决方案为多个客户提供了运行 AI、big data、仿真计算等应用的高并发计算基础平台，切实解决了众多行业痛点，方案支持的CNCF Volcano调度引擎得到了众多客户的认可。秉持“取之于社区，回馈于社区”的理念，我们将行业首个用于高性能算力的作业编排组件JobFlow捐献给 Volcano 社区，便于用户更好地应用云原生技术。” - - -**第四范式异构算力虚拟化负责人 李孟轩表示：** “Volcano项目使得我们能以较低的成本解决AI项目在云原生落地的过程中遇到的痛点，尤其是在设备复用领域，使用Volcano将会为集群资源使用率带来显著的提升。第四范式将持续的为volcano项目贡献代码，目的是将volcano打造成一个支持所有主流异构算力（NPU，GPU，MLU，DCU等）的复用平台。 - - -**唯品会AI云平台负责人 何颖鹏表示：** “作为国内TOP级的电商平台，随着业务发展，唯品会面临业务体量快速增长、产品快速迭代、产品模式多样化等问题。通过构建基于Volcano的AI训练平台，结合queue动态资源共享、gang-scheduling等Volcano提供的高阶调度能力，可支持系统10多万核的节点调度，加速了业务创新步伐。” - - -**中科类脑研发中心负责人 常峰表示：** “Volcano是最早针对批量计算场景开源的云原生项目之一，其动态可配的高级调度策略和优秀的资源管理能力解决了AI场景下作业调度、生命周期管理、异构硬件支持等多个问题。在落地实践的过程中，我们基于Volcano的能力做扩展，有效提升了系统稳定性和资源利用效率。” - - -**品览联合创始人&CTO 彭靖田表示：** “CNCF Volcano 项目成功应用于品览的 云原生智能建筑设计产品：筑绘通AlphaDraw。在CAD二维图纸的AI识图翻模和三维建筑模型的智能设计等业务场景中，Volcano为AlphaDraw的算法服务提供了批处理和弹性伸缩负载的能力，大幅提升了Kubernetes集群资源利用率，同时优化了工作负载性能。作为Volcano社区共建计划的首批成员，品览会将 Cloud+AI 在建筑设计领域的最佳实践，持续贡献给开源社区。期待AlphaDraw与Volcano项目共同成长，未来不断为智能云计算和产业云落地提供更多优秀的产品和解决方案。” - - -**网易数帆云原生技术专家 王新勇表示：** “Volcano为Kubernetes的原生能力提供了许多有益的补充，使其能够更好地编排AI训练、大数据计算等批处理任务。Volcano良好的任务抽象和管理能力，多种场景化的调度机制支持，以及与多种常见的开源计算框架集成，开箱即用，使我们能够更加专注于为用户提供业务价值，而无需花费大量精力去重复造轮子。” - - -**锐天投资基础架构团队负责人表示：** “Volcano补全了原生Kubernetes批处理任务调度、资源共享、公平调度策略等能力，并提供统一的接口降低学习和维护成本。在生产环境中，Volcano配合我们自研的二级调度，实现了每日数万任务的需求，大大提升了策略研究的效率。” - - - -**360容器团队负责人表示：** “Volcano弥补了Kubernetes原生调度在机器学习，大数据计算任务上的能力缺失，丰富的调度插件，以解决不同场景下的任务调度，极大提升集群整体利用率。且支持大部分主流的计算框架，诸如Spark,Tensorflow,Flink等。整体设计遵循Kubernetes的设计理念和风格，降低学习成本。” - - -**喜马拉雅AI云团队负责人表示：** “Volcano补全了原生Kubernetes批处理任务调度、资源共享、公平调度策略等能力，并提供弹性调度能力。在生产环境中，作为机器学习平台资源调度的基础组件，提升了GPU利用率。” - - -**BOSS直聘AI基础平台团队负责人表示：** “BOSS直聘基于Volcano在AI、大数据计算等场景构建基础设施，其强大的批处理能力和丰富的调度策略，为我们支撑复杂业务场景提供了极大的便利，大幅提升集群资源利用率和稳定性。在其丰富的生态和广泛的社区的支持下，为我们的技术和业务发展提供了极大的帮助。” - - - - -我们期待着与更多的组织共建Volcano社区的繁荣！ -
- **Volcano社区共建计划介绍** -
- - -为帮助用户快速融入社区，加速落地实践，共同打造繁荣的社区生态，Volcano社区现推出Volcano社区共建计划。 - -通过Volcano社区共建计划，你将获得技术指导、宣传推广等支持，线上/线下技术布道分享等机会！如果你的公司、组织认可Volcano的技术路线，希望在使用Volcano的过程中获得帮助，并有意和Volcano社区共建技术影响力，请考虑加入该计划。 - -关于加入该计划的要求与权益详情，请参考：https://github.com/volcano-sh/community/blob/master/community-building-program.md - - -## 申请流程 -- 扫描二维码或点击阅读全文填写申请表单 - -
{{
}}
- -- 结果将通过邮件通知，请耐心等待 - - - **任何疑问和问题，请联系Volcano社区 Maintainer: wang.platform@gmail.com** - - - diff --git a/content/zh/blog/_index.md b/content/zh/blog/_index.md deleted file mode 100644 index eddc1306..00000000 --- a/content/zh/blog/_index.md +++ /dev/null @@ -1,16 +0,0 @@ -+++ -title = "Blogs" -date = 2017-01-01T00:00:00 -type = "blogList" - -# List format. -# 0 = Simple -# 1 = Detailed -# 2 = Stream -list_format = 2 - -# Optional featured image (relative to `static/img/` folder). -[header] -image = "" -caption = "" -+++ diff --git a/content/zh/blog/aiqiyi.md b/content/zh/blog/aiqiyi.md deleted file mode 100644 index 7e7656ca..00000000 --- a/content/zh/blog/aiqiyi.md +++ /dev/null @@ -1,282 +0,0 @@ -+++ -title = "揭秘爱奇艺深度学习平台云原生迁移实践" -description = "Volcano在互联网领域的落地案例" -subtitle = "" - -date = 2020-09-30 -lastmod = 2021-08-25 -datemonth = "Aug" -dateyear = "2021" -dateday = 25 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["李灏/爱奇艺资深研发工程师"] - -tags = ["Practice"] -summary = "深入介绍Volcano在锐天投资落地实践过程中遇到的挑战和解决方案" - -# Add menu entry to sidebar. -linktitle = "揭秘爱奇艺深度学习平台云原生迁移实践" -[menu.posts] -parent = "tutorials" -weight = 9 -+++ - ->本文2020年9月30日首发于容器魔方微信公众号，原文链接[揭秘爱奇艺深度学习平台云原生迁移实践](https://mp.weixin.qq.com/s/YtP-ZURRBr5-ba1eWfKS2A) - -## Jarvis深度学习平台介绍 - -__平台整体架构__ - -平台支持 GPU 和 CPU 的训练/推理，支持 S3、HDFS 和 NFS 作为训练数据的存储、模型的存储。平台支持 Tensorflow、Pytorch、Caffe、Caffe2、MxNet，但主要以 Tensorflow 和 Pytorch 为主，支持 tensorflow 从 1.X 到 2.X 的各个版本。 - - - -支撑公司的广告、搜索、推荐、NLP 等业务。我们基于 mesos + marathon 作为我们的弹性容器平台。这是因为我们容器弹性平台起步比较早，当时 k8s 还不是很成熟。因此在很长一段时间内，我们的容器并不是运行在 K8s 平台上，这点需要大家注意。 - - - - - -{{
}} - -__一站式平台服务__ - -这里主要是通过4个小平台实现的，第一个是数据预处理平台：通过数据预处理平台可以对训练数据进行可视化分析，帮助用户合理的调参，并及时发现异常数据。 - -{{
}} - -第二个是编写训练代码平台：用户可以通过 runonce 训练或者 notebook 训练，得到一个和训练环境相同的环境编写训练代码，并上传 gitlab。 - - - -第三个是执行训练任务平台：用户通过 Jarvis 训练平台执行训练任务，然后训练任务将执行用户的训练代码，输出算法模型。 - - - -最后用户通过 Jarvis 推理平台创建推理服务，并对外提供服务。 - -__平台发展__ - -{{
}} - -我们是从推理平台入手，首先解决让用户训练好的模型能对外提供服务的能力，而后再逐步的将平台功能扩展到支持训练、开发及数据预处理，目前我们正处在将容器弹性平台从 mesos + marathon 迁移至 k8s + volcano 的过程中。 - -__使用Volcano之前训练平台架构__ - -如图所示为我们在使用 Volcano 之前的训练平台架构： - -{{
}} - -其运行的流程为： - -（1）用户编写训练代码并提交到公司内部 gitlab。 - - - -（2）用户通过 web 页面、命令行工具去创建训练任务。创建训练任务需要用户填写的信息有： - -- 所需的资源 - -- 使用的镜像，每一个框架的每一个版本都通过一个镜像来支持，选择镜像就等于选择了算法框架 - -- 我们可能会有多个集群，需要用户指定在哪个集群上运行任务 - -- 需要 Gitlab project 的 url，这个 project 中包含了用户编写的训练代码 - - - -（3）Jarvis cli/web 将请求转化为 grpc 发送给 Jarvis core。 - - - -（4）Core 中将请求转换，调用 marathon api 创建容器。 - - - -（5）容器在对应的集群中启动，执行训练任务 - -__训练平台迁移到K8s的挑战__ - -主要有3个方面的挑战： - -- 原生 Pod/Deployment/Job 无法满足分布式训练的要求 - -- 无队列管理、配额管理。 - -- 调度能力缺失、如 Gang Scheduling - -__引入 Volcano__ - -实际上，Volcano 对于我们来说最重要的是这么几个概念，一个是 vcjob，可以简单的理解成 vcjob 是对 K8s job 的一种扩展，或者是对 pob 的一种封装。 - - - -第二个对于我们比较重要的是 Queue ,也就是队列，因为在 Queue 上可以分配一些配额，可以做一些配额的管理、队列的管理； - - - -第三个对我们比较重要的是 podgroup，可以把它看成是 pod 的一个集合，因为有了 pod 集合这个概念，所以它才能做一些更高级的上层调度。 - -{{
}} - -按照我的理解： - -- Volcano 是 K8S 原生的 batch System，高度符合 AI 训练场景。 - -- 不侵入 k8s 源码，符合 k8s 的开发规范。（简单来说，方便我们二次开发） - -- 项目加入 CNCF，成熟度高 - - -## Power of Volcano - -__Volcano 要如何解决迁移到 K8s 上遇到的问题：__ - -__Gang Scheduling 的问题__ - -Gang Scheduling ：可以简单的理解为要么同时被调度，要么同时不被调度，这对于 AI 训练场景来说是非常重要的，因为我们的大部分的训练都是基于分布式训练，分布式训练的特点是一次启动的 pod 非常多，可能最多的话是有四五十个 pod, 如果一个任务下有个别的 pod 被调度，部分 pod 没有被调度，那显然这个任务是不能正常运行的，那这些运行起来的 pod 是没有任何意义的，会造成资源的浪费，同时可能会引发死锁的问题。 - -{{
}} - -比如，我们整个资源池只有 4 张 GPU 卡，有训练任务 A 和 B，A 任务有 4 个 pod，每个 pod 需要一张卡；另外B任务也是同样的情况，当 A 和 B 同时被创建，如果没有 gang scheduling，A 可能拿到了2张卡，而 B 拿到了2张卡，那这个时候无论哪个任务都不能顺利完成，这个时候系统就产生死锁了，除非你增加资源，否则就一直保持死锁的状态。 - - - -如果有了 Gang scheduling , 就能避免上述问题。Volcano 通过 podgroup 这个 CRD，能够以 podgroup 为单位对 job 进行整体调度，实现 Gang scheduling的功能。 - -__分布式任务原生支持__ - -我们以 Tensorflow 分布式训练为例，它主要有这几个角色，PS、master、worker。Parameter Server（PS）是用来存储参数的，master、worker 简单理解为进行计算梯度的，在每个迭代过程，master、worker 从 parameter sever 中获得参数，然后将计算的梯度更新给 parameter server，parameter server 聚合从 master、worker 传回的梯度，然后更新参数，并将新的参数广播给 master、worker。 - -{{
}} - -当然我们在这里不是分析 Tensorflow 分布式训练的细节。我们只讨论它的一个网络结构。比如 master、worker它要和这个 Parameter Server去做通信，那么要做通信的话就存在一个问题，如果我去创建一个 pod，在创建的时候我可能并不知道这个 pod 的 IP 是什么，如果我在一个 Deployment 创建多个 pod， pod 之间也互相不知道对方的IP地址或域名是什么，我们需要通过其他的办法来做这样一件事情，对于我们来说相当的复杂。 - - - -这里的每一个角色都要互相知道对方的 IP 地址或者域名，最终要组成 TF_CONFIG 这样一个配置文件，这个配置文件里面其实写的很清楚，至少包括 master、worker、ps 的IP地址或者域名，这些都要写在这个配置文件里面，每一个节点都需要知道这个东西，除此之外还需要知道自己所担任的是什么角色，Index 是多少，对于 K8s 的话是很难实现的，但有了 Volcano 后就变得非常简单了。 - -{{
}} - -Volcano 会帮你在一个 vcjob 下多个 pod 去注入一个文件夹（etc/volcano），这个文件夹下面就会有所有的 master、volcano、ps 的域名，都会填在这里面，这样的话每个 pod 都知道整体集群里都有哪些 peer ,非常方便去组成 TF_CONFIG 这个文件，只要组成了这个 TF_CONFIG 文件，我们就能进行 Tensorflow 分布式训练。 - - - -现在的话 Tensorflow 提供一些高层的 API，比如说 TF estimator ,这个estimator 里代码的单机和分布式代码是一模一样的，只不过这个 TF_CONFIG 的配置是不一样的，所有说只要有那样一个格式的环境变量或者配置文件传进去的话，就可以做分布式训练，对于我们平台方来说,帮助用户构建 TF_CONFIG，用户拿来直接运行就可以了。Volcano 通过注入文件，可以方便构建 TF_CONFIG，以支持 TF 分布式训练。 - -__Horovod/mpi__ - -Volcano支持 Horovod 训练，Horovod 训练其实和 Tensorflow 分布式训练有点类似，因为大家都是分布式训练，区别是更新参数的方式不一样，大家可以注意一下 Horovod 训练简单来说它更新参数是环形的更新方式。 - -{{
}} - -但这个并不重要，因为对我们平台侧的话，我们主要想要做的事情是我们要构建好基础环境，让上层应用来使用，这种 Horovod 网络架构，它对于我们基础环境有什么要求呢？ - - - -它的要求很简单，它除了要保证每一个节点都要知道对方的域名和之前的需求是一模一样的之外，它还额外的 ssh 互信，因为它经过 22 端口做 ssh 登录，去做一些工作，所有在这里面需要做好互性，互性的工作如果让我们来做，就很麻烦，我们要想一堆办法来做这个事情，Volcano 可以通过设置 ssh plugin 自动完成容器里的互性，就可以达到 Horovod 对于训练任务的网络要求。 - -__配额系统、排队系统__ - -Volcano 其实是通过 Queue 这么一个 CRD 来支持的，在图中这个是表示资源池里的一个资源，我们假设有两个 Queue，一个是 Queue1，一个是 Queue2，一个的配额有20个 GPU，一个有10个 GPU，当 Queue 已经使用的资源不太多时，新任务来了之后是可以被调度的，但是如果 Queue2 已经被使用了非常多，就已经使用完了，下面的新任务来了之后那就不可以被调度，只能继续排队，你 podgroup 的状态就处于 pending 的状态，就一直处于排队中。 - -{{
}} - -Volcano 通过 Queue 支持排队及配额系统，正好与平台中的 team 对应，因为我们的结构也是一个组有一个独立的配额，这个配额与配额之间是相互独立的，你只能用这么多，使用量超过配额后，任务就将排队，当然排队的任务支持优先级策略，高优先级的任务将在有资源后首先被执行，正好和我们系统的设计是一致的，所以就非常的好对接。 - -__与 volcano 集成__ - -新增了 volcano_plugin，其封装了 vcjob、queue、podgroup 的 restful api，将 grpc 请求的内容转成 k8s api 规范的 yaml 配置，并调用 k8s api 创建容器。 - -{{
}} - -Jarvis Core 根据传入的集群信息，决定使用哪个 backend。 - -## 实际使用中遇到问题 - -__问题一__ - -现象：当升级 volcano 版本的时候，直接修改https://github.com/volcano_x0002_sh/volcano/blob/master/installer/volcano-development.yaml 中的 image，然后执行 kubectl apply -f ，会导致已经存在的 queue/vcjob 等全部消失。 - - - -原因：yaml 中的 volcano-admission-init 会重复执行，导致 volcano 整体被 reset 现象 - - - -解决办法：升级的时候想清楚升级对应的组件就可以了 - -{{
}} - -__问题二__ - -现象：通过 list_and_watch 监控 vcjob 状态变化的程序进行遇到 watch 连接无故断开的问题，即如果没有新的 events 产生，大约 80~90s 就会断开一次，每次时间还不固定，但是同样的代码 watch pod 就没有问题。 - - - -原因：通过阅读 k8s 源码，K8s 对于 CRD 资源，默认的 http timeout 的时间是time.Duration(float64(minRequestTimeout) * (rand.Float64() + 1.0))，其 minRequestTimeout 为 1 分钟，因此会出现上述的问题。可以通过客户端指定 timeoutSecond 来避免该问题。 - -{{
}} - -__问题三__ - -现象：Jarvis 训练平台中容器入口地址是一个 bash 脚本，在 k8s 下运行时，会出现 stop 命令下发后，等约 30s 才退出。 - - - -原因：bash 不会把 signal 传递给子进程中。当 graceful stop timeout 到了之后，守护进程发现容器还没有退出，会再发 SIGKILL，此时会将 bash 脚本杀掉，容器退出，但是容器中的其他进程将无法主动完成清理工作。 - - - -解决方案： - -使用 dumb-init，比如入口脚本： - -``` -#!/usr/bin/dumb-init /bin/bash - -my-web-server & # launch a process in the background - -my-other-server # launch another process in the foreground - -``` - -__对 Volcano 的修改__ - -- SVC plugin 支持传入参数，参数为 nodeport 的端口号，当创建 vcjob 并传入 SVC 参数时，将创建对应的 nodeport，这是因为我们的 tensorboard 及其他服务需要让外部访问。 - - - -- ssh plugin 的名字超过 63 字节则会创建失败，我们自己修复了这个 bug。 - - - -- Queue 的 capability 存在 bug，用户可以突破 capability 来使用资源，目前官方已经修复了这个问题。https://github.com/volcano-sh/volcano/issues/921 - - - -- 给 vcjob annotation 后，某个 pod 失败时，无法触发删除 vcjob，https://github.com/volcano_x0002_sh/volcano/issues/805 - - -## 总结 - -（1）Volcano 弥补了 kubernetes 深度学习场景下的基本能力的缺失 - -- gang scheduler - -- 队列管理 - - - -（2）Volcano 代码遵循 kubernetes 的标准，采用非侵入式方案 - -- 减少开发者的开发对接成本 - -- 便于二次开发 - - - -（3） 基于 Volcano 的 Jarvis 训练平台目前已经上线并运行良好 diff --git a/content/zh/blog/hpc.md b/content/zh/blog/hpc.md deleted file mode 100644 index e8f03cef..00000000 --- a/content/zh/blog/hpc.md +++ /dev/null @@ -1,291 +0,0 @@ -+++ -title = "HPC on Volcano：容器在气象行业HPC高性能计算场景的应用" -description = "本文以传统的HPC应用WRF为例，探讨Volcano是如何支持HPC应用的。" -subtitle = "" - -date = 2020-10-27 -lastmod = 2021-08-24 -datemonth = "Aug" -dateyear = "2021" -dateday = 24 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["金喆/华为云容器批量计算资深工程师"] - -tags = ["Practice"] -summary = "本文以传统的HPC应用WRF为例，探讨Volcano是如何支持HPC应用的。" - -# Add menu entry to sidebar. -linktitle = "HPC on Volcano：容器在气象行业HPC高性能计算场景的应用" -[menu.posts] -parent = "tutorials" -weight = 7 -+++ - ->本文2020年10月27日首发于容器魔方微信公众号，原文链接[HPC on Volcano：容器在气象行业HPC高性能计算场景的应用](https://mp.weixin.qq.com/s/wLIoJeUSey9tzOCV6GZRig) - - - -前言：Kubernetes已经成为云原生应用编排、管理的事实标准， 越来越多的应用选择向K8S迁移。HPC作为传统的分布式计算模式，在很多领域都有着广泛的应用，很多用户都希望能将HPC应用迁移到容器中运行，通过Kubernetes强大的功能来进行作业管理。Volcano作为CNCF首个面向批量计算的分布式调度系统，也支持MPI作业的调度，__本文以传统的HPC应用WRF为例，探讨Volcano是如何支持HPC应用的。__ - - -## HPC简介 - -HPC是High Performance Computing（高性能计算）的缩写。平时提到的HPC，一般指代高性能计算机群（HPCC），它将大量的计算机软件/硬件整合起来，将大的计算作业分解成一个个小部分，通过并行计算的方式加以解决。HPC高性能计算在CAE仿真、动漫渲染、物理化学、石油勘探、生命科学、气象环境等领域有广泛的应用。 - -__一般来说，高性能计算集群（HPCC）包含如下部分：__ - -{{
}} - -- PBS：Protable Batch System，资源管理器，负责管理集群中所有节点的资源。除了PBS意外，常用的资源管理系统还有Slurm，LSF等 - -- Maui：第三方任务调度器，支持资源预留，支持各种复杂的优先级策略，支持抢占机制等。资源管理器中内置了默认的任务调取器，但功能往往比较简单 - -- OpenMPI：上层通信环境，兼顾通信库，编译，分布式启动任务的功能 - - -上述三部分中，PBS和Maui对于用户来说是完全透明的，用户只需要按照PBS提供的方式提交作业即可，不需要了解内部细节。而OpenMPI则需要用户进行相关了解，来编写能够并行计算的应用。 - - - -__下面以mpirun -np 4 ./mpi_hello_world为例介绍mpi作业是如何运行的：__ - -{{
}} - -- 调用openmpi或者其他mpi的库来编写源代码，例子里就是输出hello world字符串了 - -- 使用支持MPI的编译器来编译出可执行程序mpi_hello_world - -- 将mpi_hello_world分发到各个节点，也可以通过共享文件系统来实现对mpi_hello_world的访问 - -- 运行mpirun来并行执行mpi_hello_world - - -## WRF简介 - -WRF是Weather Research and Forecasting Model（天气研究和预报模型）的简称，是一种比较常见的HPC应用。WRF是一种中尺度数值天气预报系统，设计用于大气研究和业务预报应用，可以根据实际的大气条件或理想化的条件进行模拟。 - - - -由于WRF包含多个模块，因此处理流程可能不尽相同，这里仅以WPS和WRF这两个模块为例介绍一下完整的WRF流程： - -{{
}} - -该处理流程包括4部分： - -- 外部数据源 - -- 前处理系统（WPS） - -- 核心模拟系统（WRF） - -- 后处理系统 - -__外部数据源__ - -包含静态地理数据，网络数据等。静态地理数据可以理解为某区域内的地理信息，例如山川，河流，湖泊，森林等等。网络数据是某区域内的气象环境数据，例如气温，风速风向，空气湿度，降雨量等等。 - -__前处理系统__ - -——WPS，WRF Pre-processing System） - - - -前处理系统用于载入地理和气象数据，对气象数据进行插值，为WRF提供输入数据。该部分包含3个程序： - -- geogrid.exe：定义模型投影、区域范围，嵌套关系，对地表参数进行插值，处理地形资料和网格数据 - -- ungrib.exe：从grib数据中提取所需要的气象参数 - -- metgrid.exe：将气象参数插值到模拟区域 - - - -经过这3个程序处理后，生成可以用来进行气象模拟的数据。这3个处理程序目前不支持mpi并行运算。 - -__核心模拟系统（WRF）__ - -核心模拟系统对前处理系统生成的气象信息进行模拟和预报，是WRF的核心模块。该部分包含2个程序： - -- real.exe：初始化实际气象数据 - -- wrf.exe：模拟及预报结果 - - - -real.exe和wrf.exe可以通过mpi并行运算来提升计算速度，例如： - -{{
}} - -上图中wrfinput_d0X和wrfbdy_d0X为real.exe的运算结果，wrf.exe以该结果为输入进行模拟演算，生成最终的气象模拟结果wrfout_dxx_yyyy-mm-dd_hh:mm:ss，并由后处理系统进行验证展示。 - -__后处理系统__ - -后处理系统用来验证和显示核心模拟系统的计算结果。主要由各种第三方图像和验证工具组成。下图展示了Conus 2.5km算例中各个地区相对湿度的模拟预报结果： - -{{
}} - -Conus 2.5km是指美国本土气象数据，分辨率为2.5km（将整个区域分成一个个2.5km*2.5km*2.5km的方格，每个方格中的气象信息被认为是完全一致的）。 - -## HPC on Volcano - -{{
}} - -上面介绍了一个HPCC包括资源管理器，调度器和mpi并行计算库三部分，其中资源管理器由Kubernetes负责，调度器由Volcano负责。 - - - -__在Kubernetes+Volcano环境中运行HPC应用，本质上就是在容器中运行HPC作业，示意图如下：__ - -{{
}} - -将运行的容器分为Master容器和Worker容器两种。Master容器负责启动mpirun/mpiexec命令，Worker容器负责运行真正的计算作业。 - - - -__因此Volcano为了支持MPI作业运行，添加了如下功能：__ - -- Volcano job支持定义多个pod模板，能够同时定义master pod和worker pod - -- 支持 Gang scheduling，保证作业中所有的pod能够同时启动 - -- Master/Worker pod内部主机IP映射 - -- Master/Workerpod之间ssh免密登录 - -- 作业生命周期管理 - - - -__Volcano mpi作业配置mpi_sample.yaml：__ - -``` -apiVersion: batch.Volcano.sh/v1alpha1 -kind: Job -metadata: - name: mpi-job - labels: - # 根据业务需要设置作业类型 - "Volcano.sh/job-type": "MPI" -spec: - # 设置最小需要的服务 (小于总replicas数) - # 这里等于mpimaster和mpiworker的总数 - minAvailable: 3 - # 指定调度器为Volcano - schedulerName: Volcano - plugins: - # 提供 ssh 免密认证 - ssh: [] - # 提供运行作业所需要的网络信息，hosts文件，headless service等 - svc: [] - # 如果有pod被 杀死，重启整个作业 - policies: - - event: PodEvicted - action: RestartJob - tasks: - - replicas: 1 - name: mpimaster - # 当 mpiexec 结束，认为整个mpi作业结束 - policies: - - event: TaskCompleted - action: CompleteJob - template: - spec: - # Volcano的信息会统一放到 /etc/Volcano 目录下 - containers: - # master容器中 - # 1. 启动sshd服务 - # 2. 通过/etc/Volcano/mpiworker.host获取mpiworker容器列表 - # 3. 运行mpirun/mpiexec - - command: - - /bin/sh - - -c - - | - MPI_HOST=`cat /etc/Volcano/mpiworker.host | tr "\n" ","`; - mkdir -p /var/run/sshd; /usr/sbin/sshd; - mpiexec --allow-run-as-root --host ${MPI_HOST} -np 2 mpi_hello_world; - image: Volcanosh/example-mpi:0.0.1 - imagePullPolicy: IfNotPresent - name: mpimaster - ports: - - containerPort: 22 - name: mpijob-port - workingDir: /home - resources: - requests: - cpu: "100m" - memory: "1024Mi" - limits: - cpu: "100m" - memory: "1024Mi" - restartPolicy: OnFailure - imagePullSecrets: - - name: default-secret - - replicas: 2 - name: mpiworker - template: - spec: - containers: - # worker容器中只需要启动sshd服务 - - command: - - /bin/sh - - -c - - | - mkdir -p /var/run/sshd; /usr/sbin/sshd -D; - image: Volcanosh/example-mpi:0.0.1 - imagePullPolicy: IfNotPresent - name: mpiworker - ports: - - containerPort: 22 - name: mpijob-port - workingDir: /home - resources: - requests: - cpu: "100m" - memory: "2048Mi" - limits: - cpu: "100m" - memory: "2048Mi" - restartPolicy: OnFailure - imagePullSecrets: - - name: default-secret -``` - -__提交mpi Volcano job：__ - -{{
}} - - -__作业执行完毕：__ - -{{
}} - -__查看master pod的结果：__ - -{{
}} - - -通过上述执行结果可以看出，在作业执行结束后，Volcano只清理worker pod，保留master pod，这样用户kubectl命令获取执行结果。 - - - -此外，由于网络构建可能会出现延迟，在作业运行开始时，master pod会出现连接worker pod失败的情况。对于这种情况，Volcano会自动重启master pod，保证作业能够正确运行。 - - - -通过以上示例我们可以看出，Volcano想要运行WRF作业的话，理论上需要将其中的mpi_hello_world替换为real.exe/wrf.exe，此外，用户还需要进行如下准备： - -- 自建docker images，包含完整的WRF运行环境 - -- 将计算所需要的数据（原生数据或者中间结果数据）挂载到相应的容器中 - - - -__这样就能在Kubernetes+Volcano上运行气象模拟作业了。__ - - - - - diff --git a/content/zh/blog/iflytek_case_study.md b/content/zh/blog/iflytek_case_study.md deleted file mode 100644 index c74ba168..00000000 --- a/content/zh/blog/iflytek_case_study.md +++ /dev/null @@ -1,59 +0,0 @@ -+++ -title = "科大讯飞凭借Volcano实现AI基础设施突破，赢得CNCF最终用户案例研究竞赛" -description = "科大讯飞凭借在 Volcano 上的创新应用，成功斩获 CNCF 最终用户案例研究竞赛大奖，并在 KubeCon + CloudNativeCon China 2025 大会上分享了其大规模 AI 模型训练的成功经验。" -subtitle = "" - -date = 2025-06-13 -lastmod = 2025-06-13 -datemonth = "June" -dateyear = "2025" -dateday = 13 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["News"] -summary = "科大讯飞凭借在 Volcano 上的创新应用，成功斩获 CNCF 最终用户案例研究竞赛大奖，并在 KubeCon + CloudNativeCon China 2025 大会上分享了其大规模 AI 模型训练的成功经验。" - -# Add menu entry to sidebar. -linktitle = "科大讯飞凭借Volcano实现AI基础设施突破，赢得CNCF最终用户案例研究竞赛" -[menu.posts] -parent = "news" -weight = 6 -+++ - -
{{
}} -
- -[ 中国，香港，2025年6月10日 ] 云原生计算基金会（CNCF）宣布，科大讯飞赢得 CNCF 最终用户案例研究竞赛。CNCF致力于构建可持续的云原生软件生态，科大讯飞凭借其在 Volcano 上的创新应用脱颖而出，获得本次殊荣，于6月10日至11日在香港举行的KubeCon + CloudNativeCon China 大会上，分享其大规模 AI 模型训练的成功经验。 - -### 科大讯飞的挑战 - -作为专注于语音和语言 AI 的中国科技公司，科大讯飞在业务快速增长过程中遇到了扩展难题。调度效率低导致 GPU 资源利用不足，工作流管理复杂，团队间资源争抢激烈，这些问题拖慢了研发进度，也给基础设施带来压力。 - -**使用 Volcano 后，科大讯飞实现了弹性调度、基于 DAG 的工作流和多租户隔离，简化了操作流程，显著提升了资源利用率。** - -“在使用 Volcano 之前，跨团队协调大规模 GPU 集群训练就像不断‘灭火’，资源瓶颈、任务失败和复杂的训练管道调试层出不穷，”**科大讯飞高级平台架构师 DongJiang** 表示。“Volcano 让我们拥有更灵活的控制权，能够高效可靠地扩展 AI 训练。CNCF 对我们的认可令我们倍感荣幸，我们也很期待在 KubeCon + CloudNativeCon China 现场与更多同行分享我们的实践经验。” - - -### 关于 Volcano - -Volcano 是基于 Kubernetes 构建的云原生批处理系统，专为 AI/机器学习训练、大数据处理和科学计算等高性能工作负载设计。它提供先进的调度功能，如任务编排、资源公平分配和队列管理，能够高效管理大规模分布式任务。自 2020 年加入 CNCF Sandbox 项目，2022 年晋升为 Incubating 阶段项目，Volcano 已成为处理计算密集型任务的关键工具。 - -### Volcano 为科大讯飞带来的显著成效 - -随着 AI 需求不断增长，科大讯飞选择 Volcano 来应对训练基础设施日益复杂和庞大的挑战。工程团队需要更高效的资源分配方案，管理多阶段复杂训练工作流，减少任务中断，并保障不同团队的公平资源使用。**借助 Volcano，他们实现了：** - -* **GPU 利用率提升 40%**，显著降低基础设施成本和计算资源闲置。 -* **任务失败恢复速度提升 70%**，确保训练过程不中断。 -* **资源干扰率降低 50 %**，保障业务稳定性和资源使用灵活性 - -**CNCF 首席技术官 Chris Aniszczyk** 表示：“科大讯飞的案例展示了开源技术如何解决复杂且关键的规模化挑战。通过 Volcano 提升 GPU 效率和优化训练工作流，他们降低了成本，加快了开发，并在 Kubernetes 平台上构建了更可靠的 AI 基础设施，这对所有致力于 AI 领先的组织都至关重要。” - -随着 AI 工作负载变得更加复杂和资源密集，科大讯飞的实践证明，Volcano 等云原生工具能够帮助团队简化运营、提升扩展能力。其在 KubeCon + CloudNativeCon China 的分享[1]，带来如何在 Kubernetes 环境下更有效管理分布式训练的实用经验。 - -### 参考资料 - -[1] 分享: [https://kccncchn2025.sched.com/event/23EWS?iframe=no](https://kccncchn2025.sched.com/event/23EWS?iframe=no) \ No newline at end of file diff --git a/content/zh/blog/introducing_kthena.md b/content/zh/blog/introducing_kthena.md deleted file mode 100644 index 51b42869..00000000 --- a/content/zh/blog/introducing_kthena.md +++ /dev/null @@ -1,143 +0,0 @@ -+++ -title = "Volcano社区发布Kthena子项目: 重新定义大模型智能推理" -description = "Volcano 社区迎来了一个新的子项目 Kthena！Kthena 是一个专为 Kubernetes 设计的、云原生、高性能的 LLM 推理路由和编排、调度系统。" -subtitle = "" - -date = 2025-12-29 -lastmod = 2025-12-29 -datemonth = "Dec" -dateyear = "2025" -dateday = 29 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["volcano"] - -tags = ["Practice"] -summary = "Volcano 社区迎来了一个新的子项目 Kthena！Kthena 是一个专为 Kubernetes 设计的、云原生、高性能的 LLM 推理路由和编排、调度系统。" - -# Add menu entry to sidebar. -linktitle = "Volcano社区发布Kthena子项目: 重新定义大模型智能推理" -[menu.posts] -parent = "tutorials" -weight = 5 -+++ - -今天，我们激动地向全球开发者和 MLOps 工程师宣布，Volcano 社区迎来了一个新的子项目 Kthena！Kthena 是一个专为 Kubernetes 设计的、云原生、高性能的 LLM 推理路由和编排、调度系统。它旨在解决在生产环境中大规模编排、部署和服务 LLM 所面临的核心挑战，通过其独特的超节点拓扑感知的亲和性调度，KV Cache 感知的流量调度、Prefill/Decode 分离路由等高级功能，显著提升 GPU/NPU 资源利用率和吞吐，降低推理延迟，并赋予企业前所未有的灵活性和控制力。作为 Volcano 的子项目，Kthena将致力于帮助 Volcano 扩展除 AI 训练之外的边界，打造训推一体的完整解决方案。 - -## LLM 服务化的“最后一公里”困境 - -大语言模型（LLM）正在以前所未有的速度重塑各行各业，但将其高效、经济地部署在生产环境中，特别是基于 Kubernetes 的云原生平台上，仍然困难重重。开发者们普遍面临以下挑战： -1. **资源利用率低**：LLM 推理，尤其是其独特的 KV Cache 机制，对 GPU、NPU 显存的占用是动态且巨大的。传统的负载均衡一般采用Round-Robin算法，无法感知这种负载特性，导致 GPU、NPU 资源闲置与请求排队并存，成本高昂。 -2. **延迟与吞吐量难以兼顾**：LLM 推理分为“Prefill”（处理输入提示）和“Decode”（生成 Token）两个阶段，前者是计算密集型，后者是访存密集型。将两者混合调度，常常导致无法针对性优化，影响整体服务的响应速度和吞吐能力。因此PD分离的部署已经成为主流，但如何高效路由和调度，仍是一个难题。 -3. **多租户与多模型管理复杂**：在企业环境中，通常需要同时提供多个不同模型、不同版本或经过 LoRA 微调的模型。如何实现请求的公平调度、优先级管理以及动态路由，是一个复杂的工程难题，业界甚至有些方案将AI网关与大模型一一对应。 -4. **缺乏K8s原生集成**：许多现有的解决方案要么是外部系统，与 Kubernetes 生态割裂；要么过于复杂，无法满足生产级所需的简单易用性和灵活运维。 - -## Kthena：云原生 LLM 推理的智能大脑 - -为了攻克上述难题，Kthena 应运而生。它并非要取代现有的 LLM 服务框架（如 vLLM, sgLang），而是作为它们上层的智能“交通枢纽”和“调度中心”，深度集成于 Kubernetes 之中。 - -
{{
}} -
- - -Kthena 的核心由两大组件构成： - -1）Kthena Router：一个独立、高性能面向多模型的router，负责接收所有推理请求，并根据 `ModelRoute` 规则，智能地将请求分发到后端的 `ModelServer`。 - -2）Kthena Controller Manager：Kubernetes 控制平面的控制器，它主要包含多种控制器，负责 LLM 工作负载的编排与生命周期管理。它持续调谐并联动多类 CRD（如 `ModelBooster`、`ModelServing`、`AutoScalingPolicy`/`AutoScalingPolicyBinding`、以及 `ModelRoute`/`ModelServer`），将声明式API转化为运行时资源：ModelServing 控制器编排 `ServingGroup` 与 `Prefill/Decode` 角色分组；支持网络拓扑亲和调度和Gang调度、滚动升级与故障恢复；基于 `AutoScalingPolicy` 实现弹性扩缩容。 - -这种架构使得 Kthena 成为连接用户请求与 LLM 模型的高度可编程的桥梁。 - -## 核心特性与优势 - -Kthena 的强大之处在于其专为 LLM 推理场景设计的核心功能： - -**1) 生产级推理编排（ModelServing）** - -
{{
}} -
- -- **LLM工作负载三层架构设计**：ModelServing -> ServingGroup -> Role，一个API，支持LLM原生部署、PD分离部署，乃至大EP部署等多种部署形态，简化管理多LWS的负担。例如对于PD分离的大规模部署，可用一个ModelServing表示，根据负载的大小每个ModelServing可以包含任意数目的 ServingGroup（xPyD 分组）， 每个ServingGroup包含多个角色（Prefill Decode，他们通常部署在同一个超节点内以提升推理性能），相同的角色可以等价为一个LeaderWorkerSet，支持TP/PP/EP等多节推理并行计算。 -- **原生支持Prefill-Decode分离部署**：将计算密集型的 Prefill 实例调度到配备高性能计算卡的节点组，而将访存密集型的 Decode 实例调度到配备高带宽显存的节点组，实现资源的最佳匹配和极致的端到端延迟优化。另可以独立伸缩，动态调整Prefill-Decode的比例，更灵活的应对各种复杂的业务场景（如长短句混合、实时推理等）。 -- **多并行范式支持**：TP/PP/DP/EP 等并行模式灵活配置，最大化提升资源利用率和SLO -- **内置拓扑感知、Gang 调度支持**：Gang调度确保ServingGroup/Role“成组原子化”落地，避免资源浪费；拓扑感知调度通过将Role内的一组Pod调度到网络拓扑更优的节点，提升并行计算的数据传输时延。 - -**2) 开箱即用的模型上线（ModelBooster）** - -- 针对主流的大模型，提供包括PD分离在内的多种部署范式模板，自动生成ModelRoute/ModelServer/ModelServing/Autoscaling等路由策略和生命周期管理资源 -- 覆盖通用的部署场景，至于更灵活的编排可通过ModelServing进行细粒度的控制 - -**3) 智能、模型感知的路由（Kthena Router）** - -- **多模型路由**：兼容OpenAI API，根据请求头或Body体内容，将流量调度到不同的基础模型。 -- **插件化调度算法**：提供最少请求、最小时延、KV Cache 感知、Prefix Cache 感知、LoRA 亲和、GPU 利用率感知、公平调度等多种负载均衡算法，满足用户不同业务场景和部署形态的需求 -- **LoRA 模型热插拔无中断**：感知推理引擎加载的LoRA 适配器，提供无中断的插拔和路由能力 -- **丰富的流量治理策略**：基于权重的模型路由，金丝雀发布、Token级流控、故障转移 -- **All-in-one实现架构**，无需部署Envoy Gateway，原生支持PD分离的流量调度，将多层路由合并成一层，易于维护 - -**4) 成本驱动的自动扩缩容（Autoscaler）** - -- **同构伸缩**：支持稳定、突发双模式，按业务指标（CPU/GPU/内存/自定义）精准扩缩 -- **异构部署优化**：在多推理引擎/异构加速器组合中按“成本-能力”贪心分配，最大化性价比 - -**5) 主流推理引擎与异构硬件支持** - -- 支持多种主流推理引擎vLLM、SGLang、Triton/TGI 等，统一API抽象、标准化指标 -- **支持GPU/NPU 等异构混部**，配合异构 Autoscaling 实现成本与 SLO 的动态平衡 - -**6) 内置流量控制与公平性调度** - -- **公平调度**：支持基于优先级和历史Token消耗的的公平调度，既兼顾用户的优先级，对高优先级用户提供更好的服务，又防止低优先级用户“饿死” -- **流量控制**：支持按照用户、模型、token长度进行精细化流量控制 - -## 极致的性能提升 - -基于 Kthena Router 的调度插件架构，在长系统提示词场景（如 4096 tokens）下，采用“KV Cache 感知 + 最少请求”策略相较随机基线： - -- 吞吐可提升约 2.73 倍 -- TTFT 降低约 73.5% -- 端到端时延降低超过 60% - -| Plugin Configuration | Throughput (req/s) | TTFT (s) | E2E Latency (s) | -| :---- | :---- | :---- | :---- | -| Least Request + KVCacheAware | **32.22** | **9.22** | **0.57** | -| Least Request + Prefix Cache | 23.87 | 12.47 | 0.83 | -| Random | 11.81 | 25.23 | 2.15 | - -短提示词场景差距会随提示词长度收敛，但在多轮对话、模板化生成、前缀高度相似的业务中，KV Cache 感知策略优势显著。实际收益与模型规模、Prompt长短、硬件紧密相关，但“按需组合、按场景选型”已被验证有效。 - -## 社区展望 - -Kthena 在项目规划和发展的初期便得到了部分社区用户单位的关注和支持，但这只是一个开始。我们计划在未来支持更高效的调度算法、更广泛的大模型最佳部署实践，并持续深耕 LLM 推理的大规模部署和性能优化。 - -“开源是技术创新的源头活水，也是推动产业标准化的最强引擎。作为Volcano项目的发起单位，华为云很荣幸能够与社区其他伙伴一起推出全新的Kthena分布式推理项目。这不仅是Volcano社区技术演进的重要里程碑，更是华为云在云原生AI领域长期投入与持续创新的有力见证。它将与华为云CCE（云容器引擎）、CCI（云容器实例）等基础设施深度结合，进一步释放包括昇腾（Ascend）在内的多元算力价值，为客户提供极致的算力性价比。我们希望通过Kthena，与全球开发者与伙伴，共建、共享一个开放、繁荣的云原生AI生态，为千行万业的智能化升级构筑最坚实的算力底座。” -
—— 祁小波，华为云通用计算服务产品部部长
- -“Kthena进一步巩固了Volcano在智能计算调度领域的领先地位。我们的平台利用Volcano的统一调度与资源池化能力，一站式满足通用计算与智能计算中训练、推理等多类算力需求。这使得算力资源能够在不同场景间灵活流转，有效避免了资源割裂的问题。展望未来，我们期待 Kthena结合Volcano的弹性伸缩能力与Volcano Global的跨集群调度特性，共同推动算力资源利用率进一步提升！” -
—— 杨磊，中电信人工智能公司 PaaS研发总监
- -“Volcano 项目自诞生之日起，便始终与社区以及各类 AI 场景深度共建、同频演进，逐步沉淀出一整套面向 AI 工作负载的调度与批处理生态。今天，Kthena 的出现，不仅将这条共建链路进一步拓展到大模型推理领域，把推理这一关键一环真正纳入 Volcano 生态之中，更是在统一编排与智能路由层面，将 Volcano 在调度、弹性伸缩以及多算力适配上的多年实践，凝练成一个令人振奋的里程碑式能力。 -借助既有的 Kubernetes / Volcano 生态，更多团队可以用更低的成本，获得更智能的调度决策和更高效的算力利用，并在开放协作的基础上持续演进。这不仅为道客解决了在推理场景中遇到的实际问题，也是我们所期待的云原生 AI 形态——一个足够开放、足够智能、值得我们长期投入和深度参与的社区方向。” -
—— 徐俊杰，DaoCloud 开源团队负责人、Kubernetes 社区指导委员会成员
- -“自建大模型推理服务的生产级部署和运维难题，是一个覆盖推理服务全生命周期管理（部署、运维、弹性、故障恢复等），GPU集群稳定性，资源调度效率、推理服务性能提升，推理流量智能调度、AI可观测等领域的系统工程。而这也正是Kthena项目的技术定位。 -早在Kthena的规划阶段，小红书云原生团队就和Kthena贡献者做了深度的沟通，在推理流量智能调度方向，一起设计了多种流量调度策略和路由实现。未来，双方将继续在AI网关方向合作，结合小红书内部业务经验，一起为社区提供更精细化的AI流量智能调度能力，模型API管理能力，MCP协议支持等多种生产可用能力。” -
—— 空古(陈华昌)，小红书云原生业务网关负责人
- -“在深入调研并试用Kthena这一云原生AI推理平台后，联通云对其展现出的前瞻能力印象深刻。我们尤为看好其与Volcano实现的联合调度特性，其网络拓扑感知与Gang Scheduling功能，能够有效解决大规模分布式模型推理场景下中，关于效率与可靠性的核心诉求，为破解复杂调度难题提供了极具潜力的解决方案。我们相信，Kthena卓越的低延迟、高吞吐与多模型智能路由能力，将为开源社区带来真正具备生产级的AI推理解决方案，助力开发者更高效地构建和管理云原生环境下的智能应用。” -
—— 卢照旭，联通云智算能力中心团队长
- -“开放和协作是构建社区的未来、加速技术创新的核心动力。在CNCF，我们持续致力于推动基础设施向‘AI Native’演进，为整个云原生生态提供标准、中立且可扩展的基础能力。Volcano社区通过孵化Kthena子项目，将其在大规模批量计算和调度上积累的拓扑感知、Gang调度等核心经验，精准地应用到了LLM在线推理这一关键场景。 -Kthena的价值在于，它提供了一套专为大模型设计、可供业界参考借鉴的云原生调度原语和抽象，这有助于将复杂的LLM推理工作负载，真正以Kubernetes原生的一等公民身份进行高效管理。 -这不仅是Volcano项目技术演进的重要一步，更是社区生态在解决AI规模化部署挑战中贡献的一份重要实践经验。我们诚挚邀请全球的开发者、研究人员和所有云原生爱好者加入，共同贡献智慧，完善这些关键AI基础设施，加速 AI Native 进程。” -
—— Kevin Wang, Volcano Maintainer、CNCF TOC 副主席
- -## 立即开始探索 Kthena - -**GitHub 仓库**: [https://github.com/volcano-sh/kthena](https://github.com/volcano-sh/kthena) -**官网**: [https://kthena.volcano.sh/](https://kthena.volcano.sh/) -**社区**: 加入我们的 Slack [https://cloud-native.slack.com/archives/C011GJDQS0N](https://cloud-native.slack.com/archives/C011GJDQS0N) - -让我们一起，为 LLM 插上云原生的翅膀，释放 AI 的全部潜能！ \ No newline at end of file diff --git a/content/zh/blog/kube-batch-customers.md b/content/zh/blog/kube-batch-customers.md deleted file mode 100644 index 0d5014bd..00000000 --- a/content/zh/blog/kube-batch-customers.md +++ /dev/null @@ -1,33 +0,0 @@ -+++ -title = "Kube-Batch的客户群体" -description = "基于Kubernetes的批量调度器" -subtitle ="" - -date = 2019-01-28 -lastmod = 2020-09-07 -datemonth = "Sep" -dateyear = "2020" -dateday = 07 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Volcano"] - -tags = ["Tutorials"] -summary = "基于kube-batch的广泛客户群体" - -# Add menu entry to sidebar. -linktitle = "Kube-Batch的客户群体" -[menu.posts] - parent = "customers" - weight = 1 -+++ -## 谁正在使用kube-batch? - -| 组织 | 联系方式 (Github用户名) | 环境 | 用户描述 | -|------------------------------------| ------------- | ------------- | ------------- | -| [百度](https://www.baidu.com) |[@tizhou86](https://github.com/tizhou86)| 生产环境 | 自研机器学习框架PaddlePaddle的离线训练调度器 | -| [图森未来](https://www.tusimple.com) | [@suleisl2000](https://github.com/suleisl2000) | | MXNet离线训练的调度器 | -| [IBM](https://github.com/IBM/FfDL) | [@animeshsingh](https://github.com/animeshsingh)| | | -| [蘑菇街](https://www.mogujie.com) | [@jiaxuanzhou](https://github.com/jiaxuanzhou)| 生产环境 | Tiny+离线训练的调度器| \ No newline at end of file diff --git a/content/zh/blog/kube-batch-startup.md b/content/zh/blog/kube-batch-startup.md deleted file mode 100644 index 854de5c0..00000000 --- a/content/zh/blog/kube-batch-startup.md +++ /dev/null @@ -1,194 +0,0 @@ -+++ -title = "Kube-Batch新手教程" -description = "基于Kubernetes的调度器" -subtitle ="" - -date = 2019-01-28 -lastmod = 2020-09-07 -datemonth = "Sep" -dateyear = "2020" -dateday = 07 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["Volcano"] - -tags = ["Tutorials"] -summary = "批量工作负载调度新手教程" - -# Add menu entry to sidebar. -linktitle = "Kube-Batch新手教程" -[menu.posts] - parent = "tutorials" - weight = 2 -+++ -# Kube-Batch新手速成 - -本文档将展示如何将`kube-batch`作为基于Kubernetes的批量调度引擎运行起来。代码请参考[master](https://github.com/kubernetes-sigs/kube-batch/tree/master) 。 - -## 1. 前置条件 - -运行`kube-batch`之前，必须先启动一个Kubernetes集群。关于如何搭建集群请参考[使用kubeadm创建一个集群](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)。 -另外，为了本地调试的目的，您可以使用Minikube，请参考[使用Minikube在本地运行Kubernetes](https://kubernetes.io/docs/getting-started-guides/minikube/)。 -您还可以使用[kind](https://github.com/kubernetes-sigs/kind) 。 它是一个将Docker容器作为节点并运行本地Kubernetes集群的工具。 - -接下来将展示怎样快速将`kube-batch`作为Kubernetes调度器运行起来。请参考[配置多调度器](https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/) 获取更多细节。 - -## 2. 为Kubernetes配置kube-batch - -### (1) kube-batch镜像 - -kube-batch提供了官方镜像，您可以通过[DockerHub](https://hub.docker.com/r/kubesigs/kube-batch/) 下载。当前版本为`v0.4`。 - -```html -# docker pull kubesigs/kube-batch:v0.4 -``` - -### (2) 为kube-batch创建一个Kubernetes Deployment - -#### 下载kube-batch - -```html -# mkdir -p $GOPATH/src/github.com/kubernetes-sigs/ -# cd $GOPATH/src/github.com/kubernetes-sigs/ -# git clone https://github.com/kubernetes-sigs/kube-batch -``` - -#### 使用Helm部署`kube-batch` - -将`kube-batch`作为Kubernetes调度器运行起来 - -```html -# helm install $GOPATH/src/github.com/kubernetes-sigs/kube-batch/deployment/kube-batch --namespace kube-system -``` - -验证版本 - -```html -# helm list -NAME REVISION UPDATED STATUS CHART NAMESPACE -dozing-otter 1 Thu Jun 14 18:52:15 2018 DEPLOYED kube-batch-0.4.0 kube-system -``` - -请注意：`kube-batch`需要收集集群信息（如Pod、Node、CRD等），所以用于该deployment的serviceaccount必须有权限访问这些集群资源，否则`kube-batch` -将无法启动。对于不太了解Kubernetes RBAC的用户，请将example/role.yaml拷贝到`$GOPATH/src/github.com/kubernetes-sigs/kube-batch/deployment/kube-batch/templates/` -并重装。 - -### (3) 创建Job - -创建一个名为`job-01.yaml`的文件，内容如下： - -```html -apiVersion: batch/v1 -kind: Job -metadata: - name: qj-1 -spec: - backoffLimit: 6 - completions: 6 - parallelism: 6 - template: - metadata: - annotations: - scheduling.k8s.io/group-name: qj-1 - spec: - containers: - - image: busybox - imagePullPolicy: IfNotPresent - name: busybox - resources: - requests: - cpu: "1" - restartPolicy: Never - schedulerName: kube-batch ---- -apiVersion: scheduling.incubator.k8s.io/v1alpha1 -kind: PodGroup -metadata: - name: qj-1 -spec: - minMember: 6 -``` - -yaml文件表示一个名为`qj-01`的Job将创建6个pod（由`parallelism`指定），这些pod将由调度器`kube-batch`调度（由`schedulerName`指定）。`kube-batch` -将监视`PodGroup`和名为`scheduling.k8s.io/group name`的annotation，该annotation标识pod属于哪个组。`kube-batch`将为Job同时启动数量为 -`.spec.minMember`的Pod；否则，在诸如资源不足等情况下，`kube-batch`将不会为该Job启动任何pod。 - -创建该Job - -```html -# kubectl create -f job-01.yaml -``` - -检查Job状态 - -```html -# kubectl get jobs -NAME DESIRED SUCCESSFUL AGE -qj-1 6 6 2h -``` - -检查pods状态 - -```html -# kubectl get pod --all-namespaces -``` - - -## 4. 为Pod创建PriorityClass - -`kube-batch`将根据优先级启动同一个QueueJob中的Pod。高优先级的Pod会先被启动。下面是个展示`PriorityClass`用法的例子： - -创建一个名为`priority_1000.yaml`的文件，内容如下： - -```html -apiVersion: scheduling.k8s.io/v1beta1 -kind: PriorityClass -metadata: - name: high-priority - namespace: batch-ns01 -value: 1000 -``` - -创建PriorityClass，优先级值设为1000. - -``` -# kubectl create -f priority_1000.yaml -``` - -创建一个Pod配置文件（假设名为`pod-config-ns01-r01.yaml`）： - -```html -apiVersion: v1 -kind: Pod -metadata: - name: pod-ns01-r01 -spec: - containers: - - name: key-value-store - image: redis - resources: - limits: - memory: "1Gi" - cpu: "1" - requests: - memory: "1Gi" - cpu: "1" - ports: - - containerPort: 6379 - priorityClassName: high-priority -``` - -创建该Pod，优先级设置为1000。 - -``` -# kubectl create -f pod-config-ns01-r01.yaml -``` - - -请注意: - -* `PriorityClass` 仅在kubernetes 1.9+中支持. -* 在同一个Deployment/RS/Job中的Pod共享该Pod模板，因此它们有同样的`PriorityClass`。为了给同一个QueueJob中的Pod设置不同的`PriorityClass`， -用户需要自己创建控制器。 diff --git a/content/zh/blog/leinao-cloud-os.md b/content/zh/blog/leinao-cloud-os.md deleted file mode 100644 index c44ea7de..00000000 --- a/content/zh/blog/leinao-cloud-os.md +++ /dev/null @@ -1,155 +0,0 @@ -+++ -title = "Volcano在中科类脑云OS中的落地实践" -description = "深入介绍Volcano在中科类脑云OS落地实践过程中遇到的挑战和解决方案" -subtitle = "" - -date = 2020-12-24 -lastmod = 2021-08-23 -datemonth = "Dec" -dateyear = "2020" -dateday = 23 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["朱建/中科类脑平台事业部高级系统架构师"] - -tags = ["Practice"] -summary = "深入介绍Volcano在中科类脑云OS落地实践过程中遇到的挑战和解决方案" - -# Add menu entry to sidebar. -linktitle = "Volcano在中科类脑云OS中的落地实践" -[menu.posts] -parent = "tutorials" -weight = 3 -+++ - ->本文2020年12月24日首发于容器魔方微信公众号，原文链接[Volcano在中科类脑云OS中的落地实践](https://mp.weixin.qq.com/s/HS6RzzqztBJsHQX7P5T5ww) - -## 类脑AI云平台介绍 - -AI开发平台 ：面向AI科研人员，提供不同领域不同应用场景相匹配的全流程技术支撑和解决方案 - -公共服务平台：面向政府和社会人员，提供人工智能相关的新闻咨询、运营内容等支撑 - -AI可视化平台：面向管理人员，提供统一的运营数据可视化展示，运筹帷幄 - -AI社区：面向AI开发者和相关企业的社区平台，提供AI交流、问答、培训服务 - - - -## 类脑云OS架构 - -{{
}} - - - -在OS架构中，最底层为硬件平台，硬件平台之上为引擎层。引擎层主要包括调度引擎和数据引擎两部分，提供任务调度和数据管理能力。 - -引擎层之上为计算层，它提供一系列面向通用场景创建分布式训练任务的接口。 - -应用层由面向业务的一系列业务系统组成，如模型管理系统、资源管理系统、运维监控系统等。 - - -## 为什么选择volcano？ - -在搭建类脑云OS 2.0时，我们尝试寻找一款生态环境比较好的调度引擎。经过一系列的调研，我们发现Volcano的分布式训练任务重试机制是基于失败场景的，而我们原有的任务重试机制是基于失败次数的；其次，**Volcano对深度学习框架的支持比较友好**；此外，**Volcano有大量高级调度策略** 。 - -另外，我们发现K8s默认的调度机制对批量调度的支持并不友好。Hadoop虽然支持批量任务，但是它的架构比较笨重。因此，基于上面几点我们最终选择了Volcano。 - -{{
}} - -{{
}} - - -由于在决定引入Volcano调度服务时，我们系统已经完成开发，如何在现有系统里引入Volcano呢？如果直接引入会导致系统计算层和应用层修改较大。 - - - 另外，我们在引入前也对Volcano需求进行了分析，发现单机任务和分布式训练任务上Volcano已经支持，但是对调试任务、调试工具集需要做一系列定制开发，所以需要引入一个组件job-server做接口的适配和类似调试工具集的集成开发。 - -{{
}} - -在确定使用Volcano和通过 job server去做任务适配后，我们梳理了Volcano的场景，发现上层的业务是需要知道task详细的运行信息、状态变化信息、历史信息，而Volcano Job可以提供job的运行状态信息，但暂未提供task运行相关信息。 - - - -因此我们就面临一个问题：**去定制修改Volcano还是实现Volcano task监控？**当时发现Volcano对新功能的版本迭代是非常快的，如果去做深度的修改，这种源码会给后续版本的升级带来很大的困难。并且我们也希望这个版本随着Volcano升级功能的增强，和它版本迭代一起往上升级，所以我们采用了第二种方案，去监控Volcano task的一些运行状态信息。 - - - -下图是一个简单的监控机制，通过Watch的方式去监控API Server里job相关信息的变化和Pod相关信息的变化。 - -{{
}} - - - -## 实践思路 - -__场景__ - - -- 创建批量任务 - -- 批量任务需要具备jupyter lab、tensorboard、code-server、wetty等调试工具集 - -- 批量任务支持数据存储集优化策略 - -- 支持训练、量化和模型转换 - - -基于上面的场景，我们做如下的设计： - -{{
}} - -{{
}} - - - -#### 任务的清理 - -__场景__ - -- Job运行完成后，清理任务 - -- 清理该Job附加的能力（jupyter、code-server） - -**1）当Job结束时，volcano可以自动清理** - -{{
}} - -**2）清理能力相关资源（pod、service、ingress）** - -{{
}} - -{{
}} - - - - -#### 任务重试 - -我们1.0版本设计的初衷是基于次数进行限制，但是后来发现Volcano基于事件触发的机制更加灵活，业务场景也更加合理。因此我们在任务重试方面进行了原有方案的一系列改动，直接采用Volcano的任务重试机制。 - -{{
}} - -{{
}} - -- TaskRole中的策略优先级高于Job中的重试策略 - -- 重试策略中包含Event和Action，Event表示触发条件， - -- Action表示条件满足后执行的动作 - -- maxRetry限定最大重试次数 - - -在实现以上特性的过程中，我们得到了Volcano社区的大力支持。当时遇到的一个问题是RestartTask失效，在当天遇到这个问题时反馈给了社区，当天就得到了解决，这个响应速度是非常快的。 - -{{
}} - - -## 后期计划 - -在拓扑调度、基于单机多卡如何选择最好的GPU拓扑结构进行调度这两方面，希望社区能够给予更好的支持。也希望我们和社区共同努力，深化双方在技术、运营等方面的合作，使得社区生态更加繁荣！ - - \ No newline at end of file diff --git a/content/zh/blog/paddlepaddle.md b/content/zh/blog/paddlepaddle.md deleted file mode 100644 index ce66461f..00000000 --- a/content/zh/blog/paddlepaddle.md +++ /dev/null @@ -1,614 +0,0 @@ -+++ -title = "百度飞桨(PaddlePaddle)分布式训练在Volcano系统上的实践" -description = "深入基于Volcano和PaddlePaddle的分布式训练最佳实践" -subtitle = "" - -date = 2019-11-06 -lastmod = 2021-08-23 -datemonth = "Dec" -dateyear = "2020" -dateday = 23 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["PaddlePaddle团队", "Volcano团队"] - -tags = ["Practice"] -summary = "深入基于Volcano和PaddlePaddle的分布式训练最佳实践" - -# Add menu entry to sidebar. -linktitle = "百度飞桨(PaddlePaddle)分布式训练在Volcano系统上的实践" -[menu.posts] -parent = "tutorials" -weight = 4 -+++ - ->本文2019年11月6日首发于容器魔方微信公众号，原文链接[百度飞桨(PaddlePaddle)分布式训练在Volcano系统上的实践](https://mp.weixin.qq.com/s/SnUUEEy9OfNghzoel7FtUg) - -飞桨(PaddlePaddle)是百度于2016年9月开源的深度学习框架，旨在提供一款安全高效、灵活易用、可扩展的深度学习平台。 - -2018年10月，飞桨团队发布Paddle Fluid 1.0版本，对神经网络描述、大规模分布式训练、高性能推理引擎等核心能力进行了全面升级。以工业界应用必需的分布式训练能力为例，在最新的Paddle Fluid 1.5.2版本中，飞桨支持数据并行、模型并行、流水线并行等多种并行模式，参数服务器架构和点对点同步训练架构全面支持在CPU、GPU等硬件资源设备上的大规模训练。本文将介绍飞桨分布式训练在Kubernetes社区中的Volcano系统上进行实践的案例。 - - - -__Kubernetes是当今最火的容器化应用自动部署、伸缩和资源管理的开源系统。__ 随着Kubernetes的崛起，越来越多的公司愿意将自己的业务应用部署在Kubernetes上。除了典型的Web服务、数据库等服务会基于Kubernetes进行部署以外，深度学习框架的分布式训练也不例外。 - - - -然而，在Kubernetes系统中提交深度学习训练任务的功能并不像传统的高性能计算MPI平台那样直观。在2017年，Kubernetes社区就有文章Run Deep Learning with PaddlePaddle on Kubernetes分析了运行PaddlePaddle对底层资源的诉求，基于PaddlePaddle对计算容错性、弹性伸缩、资源隔离的要求，提出在Kubernetes平台上运行PaddlePaddle是最佳实践。 - - - -自Paddle Fluid 1.0 发布以来，飞桨在平台部署和任务调度管理上已经取得了长足的进步。借助Kubernetes平台，飞桨可以实现CPU/GPU等硬件资源的合理调度分配、训练任务的弹性扩缩容，并能显著提升计算资源的利用率。但是，在并行创建和调度任务、训练任务的生命周期管理、计算资源亲和性调度、调度策略优化等方面还有提升空间。__为了提升飞桨框架的计算效率，飞桨团队和Volcano团队联合发布PaddlePaddle on Volcano方案。__ - -Volcano是一款构建于Kubernetes之上的增强型高性能计算任务批量处理系统。 - -作为一个面向高性能计算场景的平台，它弥补了kubernetes 在机器学习、深度学习、HPC、大数据计算等场景下的基本能力缺失，其中包括gang-schedule的调度能力、计算任务队列管理、GPU亲和性调度。另外，Volcano在原有Kubernetes能力基础上对计算任务的批量创建及生命周期管理、Fair-share调度等方面做了增强。 - - - -__Volcano平台可以满足飞桨对资源创建，资源调度的基本要求。__ Volcano的批量创建批量调度计算任务为飞桨作业提供计算任务的自动化生命周期管理，gang-scheduler调度策略可以满足PServer和Trainer “all or nothing”的业务调度约束，Queue和priority逻辑可以管理集群下计算作业的执行顺序，Fair-share和GPU亲和度调度使计算任务调度更贴合PServer和Trainer对节点资源和网络拓扑结构的要求而提升任务计算效率。 - - - -Volcano借助Kubernetes创建CRD能力，在Kubernetes中引入“apiVersion”为“batch.volcano.sh/v1alpha1 ”,“kind”为“Job”的资源对象，用于描述计算任务。通过配置和创建Volcano job可以使用Volcano平台创建、管理和调度计算任务。使用volcano平台，需要先在Kubernetes集群下安装Volcano，安装Volcano的方法可参考Volcano 官网。 - -选择一个飞桨框架任务分别使用Kubernetes原生资源和Volcano job执行计算任务并对比分析，以下对比将着重体现两者在使用方法、任务管理、调度策略方面进行比较。选择飞桨官网分布式训练CTR（Click-Through-Rate） demo进行对比测试。CTR demo将运行两个PServer任务和两个Trainer任务。 - - - -首先使用飞桨官网推荐模式执行分布式计算任务，先创建一个副本数为2的Kubernetes ReplicaSet对象，用于运行PServer业务，然后创建一个并行度为2的Kubernetes Job对象，用于运行Trainer任务。 - - - -创建PServer任务 -``` -root@volcano-paddlepaddle:~# kubectl apply -f pserver.yaml - - -replicaset.extensions/fluid-ctr-pserver create -``` - -查看pserver ReplicaSet组件 -``` -root@volcano-paddlepaddle:~# kubectl get rs - -NAME DESIRED CURRENT READY AGE -fluid-ctr-pserver 2 2 2 5 -``` - - -查看pserver pods -``` -root@volcano-paddlepaddle:~# kubectl get pods | grep fluid - -fluid-ctr-pserver-b9w99 1/1 Running 0 9m18s -fluid-ctr-pserver-pb9vd 1/1 Running 0 9m18 -``` - -查看pserver日志，PServer已经开始工作，并对外提供服务 -``` -root@volcano-paddlepaddle:~# kubectl logs fluid-ctr-pserver-b9w99 - - -+ case "$1"in -+ start_fluid_process -+ pserver_label=paddle-job-pserver=fluid-ctr -+ trainer_label=paddle-job=fluid-ct -+ hostname=c-rlnrdybm-muamumvq-1 -+ task_index= -+ '[' PSERVER == TRAINER '] -+ '[' PSERVER == PSERVER ']' -+ stdbuf -oL python /root/k8s_tools.py wait_pods_running paddle-job-pserver=fluid-ctr 2 -label selector: paddle-job-pserver=fluid-ctr, desired: 2 -current cnt: 0 sleep for 5 seconds... -+ '[' PSERVER == TRAINER ']' -+ '[' PSERVER == WORKER '] -++ python /root/k8s_tools.py fetch_endpoints paddle-job-pserver=fluid-ctr 30236 -+ export PADDLE_PSERVERS=192.168.48.24:30236,192.168.48.25:30237 -+ PADDLE_PSERVERS=192.168.48.24:30236,192.168.48.25:30237 -++ python /root/k8s_tools.py fetch_ips paddle-job=fluid-ctr -+ export PADDLE_TRAINER_IPS= -+ PADDLE_TRAINER_IPS= -+ '[' PSERVER == TRAINER ']' -+ '[' PSERVER == WORKER ']' -++ python /root/k8s_tools.py fetch_id paddle-job-pserver=fluid-ctr -+ task_index=0 -+ export PADDLE_TRAINER_ID=0 -+ PADDLE_TRAINER_ID=0 -+ export PADDLE_PSERVER_ID=0 -+ PADDLE_PSERVER_ID=0 -+ stdbuf -oL sh -c 'cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1' -2019-09-03 06:43:10,661 - INFO - run dist training -2019-09-03 06:43:10,715 - INFO - run pserver -get_pserver_program() is deprecated, call get_pserver_programs() to get pserver main and startup in a single call. -I0903 06:43:10.826609 41 grpc_server.cc:435] Server listening on 192.168.48.24:30236 selected port: -``` - -创建trainer任务 -``` -root@volcano-paddlepaddle:~# kubectl apply -f trainer.yaml - -job.batch/fluid-ctr-trainer create -``` - -查看trainer pods -``` -root@volcano-paddlepaddle:~# kubectl get pod | grep fluid - - -fluid-ctr-pserver-b9w99 1/1 Running 0 87m -fluid-ctr-pserver-pb9vd 1/1 Running 0 87m -fluid-ctr-trainer-lg9n5 1/1 Running 0 12s -fluid-ctr-trainer-tvr99 1/1 Running 0 12 -``` - -查看trainer任务日志，看到任务已经开始执行 - -``` -root@volcano-paddlepaddle:~# kubectl logs fluid-ctr-trainer-lg9n5 - -+ case "$1" in -+ start_fluid_process -+ pserver_labe=paddle-job-pserver=fluid-ctr -+ trainer_label=paddle-job=fluid-ctr -+ hostname=c-rlnrdybm-muamumvq-2 -+ task_index= -+ '[' TRAINER == TRAINER ']' -+ stdbuf -oL python /root/k8s_tools.py wait_pods_running paddle-job-pserver=fluid-ctr 2 -label selector: paddle-job-pserver=fluid-ctr, desired: 2 -+ '[' TRAINER == TRAINER ']' -+ stdbuf -oL python /root/k8s_tools.py wait_pods_running paddle-job=fluid-ctr 2 -label selector: paddle-job=fluid-ctr, desired: 2 -++ python /root/k8s_tools.py fetch_endpoints paddle-job-pserver=fluid-ctr 30236 -+ export PADDLE_PSERVERS=192.168.48.24:30236,192.168.48.25:30237 -+ PADDLE_PSERVERS=192.168.48.24:30236,192.168.48.25:30237 -++ python /root/k8s_tools.py fetch_ips paddle-job=fluid-ctr -+ export PADDLE_TRAINER_IPS=192.168.48.24,192.168.48.25 -+ PADDLE_TRAINER_IPS=192.168.48.24,192.168.48.25 -+ '[' TRAINER == TRAINER ']' -+ check_failed_cnt 1 -+ max_failed=1 -++ python /root/k8s_tools.py count_pods_by_phase paddle-job=fluid-ctr Failed -+ failed_count=0 -+ '[' 0-gt 1 ']' -++ python /root/k8s_tools.py fetch_id paddle-job=fluid-ctr -+ task_index=0 -+ export PADDLE_TRAINER_ID=0 -+ PADDLE_TRAINER_ID=0 -+ export PADDLE_PSERVER_ID=0 -+ PADDLE_PSERVER_ID=0 -+ stdbuf -oL sh -c 'cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1' -2019-09-03 08:10:20,888 - INFO - run dist training -2019-09-03 08:10:20,951 - INFO - download the training materials - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed -100 433M 100 433M 0 0 70.9M 0 0:00:06 0:00:06 --:--:-- 97.0M -2019-09-03 08:11:04,522 - INFO - run trainer -2019-09-03 08:11:04,591 - WARNING - -I0903 08:11:04.594007 25 parallel_executor.cc:329] The number of CPUPlace, which is used in ParallelExecutor, is 2. And the Program will be copied 2 copies -I0903 08:11:04.875757 25 rpc_client.h:101] init rpc client with trainer_id 0 -2019-09-03 08:11:38,625 - INFO - TRAIN --> pass: 0 batch: 0 loss: 0.697331115723 auc: 0.500826068453, batch_auc: 0.500826068453 -2019-09-03 08:11:38,967 - INFO - TRAIN --> pass: 0 batch: 1 loss: 0.652093688965 auc: 0.48451329672, batch_auc: 0.48451329672 -2019-09-03 08:11:39,242 - INFO - TRAIN --> pass: 0 batch: 2 loss: 0.629092956543 auc: 0.485173881519, batch_auc: 0.485173881519 -2019-09-03 08:11:39,577 - INFO - TRAIN --> pass: 0 batch: 3 loss: 0.603850708008 auc: 0.482131778494, batch_auc: 0.482131778494 -2019-09-03 08:11:39,874 - INFO - TRAIN --> pass: 0 batch: 4 loss: 0.591485412598 auc: 0.479737304993, batch_auc: 0.479737304993 -2019-09-03 08:11:40,133 - INFO - TRAIN --> pass: 0 batch: 5 loss: 0.58376159668 auc: 0.478554220739, batch_auc: 0.478554220739 -2019-09-03 08:11:40,385 - INFO - TRAIN --> pass: 0 batch: 6 loss: 0.561969116211 auc: 0.481465857424, batch_auc: 0.481465857424 -2019-09-03 08:11:40,637 - INFO - TRAIN --> pass: 0 batch: 7 loss: 0.557065185547 auc: 0.486014931119, batch_auc: 0.486014931119 -2019-09-03 08:11:40,890 - INFO - TRAIN --> pass: 0 batch: 8 loss: 0.562498413086 auc: 0.489651573333, batch_auc: 0.489651573333 -2019-09-03 08:11:41,158 - INFO - TRAIN --> pass: 0 batch: 9 loss: 0.566428283691 auc: 0.489853260221, batch_auc: 0.49137884426 -2019-09-03 08:11:41,452 - INFO - TRAIN --> pass: 0 batch: 10 loss: 0.564840087891 auc: 0.492880386228, batch_auc: 0.494013763938 -2019-09-03 08:11:41,742 - INFO - TRAIN --> pass: 0 batch: 11 loss: 0.564809204102 auc: 0.493201528907, batch_auc: 0.498872381582 -2019-09-03 08:11:42,056 - INFO - TRAIN --> pass: 0 batch: 12 loss: 0.584479736328 auc: 0.494151972036, batch_auc: 0.503926628391 -2019-09-03 08:11:42,329 - INFO - TRAIN --> pass: 0 batch: 13 loss: 0.615677246094 auc: 0.49252557362, batch_auc: 0.5028352489 -``` - - -等待trainer任务执行完成，查看pserver和trainer pods状态，trainer已经执行完成，pserver仍然于运行中 -``` -root@volcano-paddlepaddle:~# kubectl get pods | grep fluid - - -fluid-ctr-pserver-b9w99 1/1 Running 0 177m -fluid-ctr-pserver-pb9vd 1/1 Running 0 177m -fluid-ctr-trainer-lg9n5 0/1 Completed 0 90m -fluid-ctr-trainer-tvr99 0/1 Completed 0 90 -``` - -__将上述计算任务迁移到volcano平台上进行测试。__ - - - -Volcano支持Multi-pod jobs，拓展“tasks”字段，tasks下可以定义多个pod描述，其中“replicas” 字段描述task将要生成的pod数量，“name”描述task名称，pod名称将根据task名称生成。Template字段与kubernetes “podTemplate”一致。ctr的demo中含有两个task： “pserver”和“trainer”，每个task的replicas都是2，将会创建两个PServer任务，两个Trainer任务。 - - - -使用Volcano调度器，在job的配置中需要指定“schedulerName”为“volcano”，如果schedulerName没有指定为“volcano”，job下的任务调度将会使用kubernetes的默认调度器“default”调度器。 - - - -Volcano通过指定“minAvailable”字段保证计算任务的gang-scheduler调度策略。“minAvailable”数值指明在对当前计算任务下的pods进行调度时，需保证多少计算任务都能够调度才会执行调度任务，“minAvailable”的数值需要小于或等于计算任务下的所有任务数量的总和。对于PaddlePaddle框架计算任务，只有当所有的PServer和Trainer任务都处于运行中，才能开始计算任务。因此对于飞桨计算任务，“minAvailable”的数值需要与计算任务下的所有计算任务总和相等。 - - - -对于使用飞桨分布式训练的应用，在计算过程中，如果PServer任务或者Trainer任务被驱逐或失败，PServer和Trainer形成的计算集群将会失效，所有的PServer任务和Trainer任务都需要重启，以形成新的集群开始新的计算。Volcano可以通过设置“policies”实现上述目的。设置“PodEvicted”事件对应“RestartJob”动作，设置“PodFailed”事件对应“RestartJob”动作，在设置了这两个“policies”之后，当计算任务被驱逐或者失败，所有的计算任务将会重启。 - - - -下面是使用Volcano平台执行CTR任务的配置ctr-volcano.yaml，配置文件可从Volcano代码库获取 - -Volcano代码仓库地址： - -https://github.com/volcano-sh/volcano/blob/master/example/integrations/paddlepaddle/ctr-paddlepaddle-on-volcano.yaml - - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: ctr-volcano -spec: - minAvailable: 4 -schedulerName: volcano - policies: -- event: PodEvicted - action: RestartJob - - event: PodFailed - action: RestartJob - tasks: -- replicas: 2 - name: pserver - template: - metadata: - labels: -paddle-job-pserver: fluid-ctr - spec: - imagePullSecrets: - - name: default-secret - volumes: - - hostPath: - path: /home/work/ - type: "" - name: seqdata - containers: - - image: volcanosh/edlctr:v1 - command: - - paddle_k8s - - start_fluid - imagePullPolicy: IfNotPresent - name: pserver - volumeMounts: - - mountPath: /mnt/seqdata - name: seqdata - resources: - limits: - cpu: 10 - memory: 30Gi - ephemeral-storage: 10Gi - requests: - cpu: 1 - memory: 100M - ephemeral-storage: 1Gi - env: - - name: GLOG_v - value: "0" - - name: GLOG_logtostderr - value: "1" - - name: TOPOLOGY - value: "" - - name: TRAINER_PACKAGE - value: /workspace - - name: NAMESPACE - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.namespace - - name: POD_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: POD_NAME - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.name - - name: PADDLE_CURRENT_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP -- name: PADDLE_JOB_NAME - value: fluid-ctr - - name: PADDLE_IS_LOCAL - value: "0" - - name: PADDLE_TRAINERS_NUM - value: "2" - - name: PADDLE_PSERVERS_NUM - value: "2" - - name: FLAGS_rpc_deadline -" value: " - - name: ENTRY - value: cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1 - - name: PADDLE_PORT - value: "30236" - - name: LD_LIBRARY_PATH - value: /usr/local/lib:/usr/local/nvidia/lib64:/usr/local/rdma/lib64:/usr/lib64/mlnx_ofed/valgrind -- name: PADDLE_TRAINING_ROLE - value: PSERVER -- name: TRAINING_ROLE - value: PSERVER - restartPolicy: OnFailure - - replicas: 2 - policies: - - event: TaskCompleted - action: CompleteJob - name: trainer - template: - metadata: - labels: - paddle-job: fluid-ctr - spec: - imagePullSecrets: -- name: default-secret - volumes: - - hostPath: - path: /home/work/ - type: "" - name: seqdata - containers: - - image: volcanosh/edlctr:v1 - command: - - paddle_k8s - - start_fluid - imagePullPolicy: IfNotPresent - name: trainer - volumeMounts: - - mountPath: /mnt/seqdata - name: seqdata - resources: - limits: - cpu: 10 - memory: 30Gi - ephemeral-storage: 10Gi - requests: - cpu: 1 - memory: 100M - ephemeral-storage: 10Gi - env: -- name: GLOG_v - value: "0" - - name: GLOG_logtostderr - value: "1" -- name: TOPOLOGY -- name: TRAINER_PACKAGE - value: /workspace - - name: CPU_NUM - value: "2" - - name: NAMESPACE - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.namespace - - name: POD_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: POD_NAME - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.name - - name: PADDLE_CURRENT_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: PADDLE_JOB_NAME - value: fluid-ctr - - name: PADDLE_IS_LOCAL - value: "0" - - name: FLAGS_rpc_deadline -" value: " - - name: PADDLE_PORT - value: "30236" -- name: PADDLE_PSERVERS_NUM - value: "2" - - name: PADDLE_TRAINERS_NUM - value: "2" -- name: PADDLE_TRAINING_ROLE - value: TRAINER - - name: TRAINING_ROLE - value: TRAINER - - name: LD_LIBRARY_PATH - value: /usr/local/lib:/usr/local/nvidia/lib64:/usr/local/rdma/lib64:/usr/lib64/mlnx_ofed/valgrind - - name: ENTRY - value: cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1 - restartPolicy: OnFailure -``` - - -在集群终端中执行以下指令在default namespace下创建volcano job -``` -root@volcano-paddlepaddle:~# kubectl apply -f ctr-volcano.yaml - -job.batch.volcano.sh/ctr-volcano create -``` - -检查pods的状态，无论是pserver任务还是trainer任务都被下发到集群中，并开始运行。如果当前集群下的空闲资源，不能满足pserver任务和trainer任务的资源述求，任何任务都不会被创建。 - -``` -root@volcano-paddlepaddle:~# kubectl get pods | grep ctr-volcano - - -ctr-volcano-pserver-0 1/1 Running 0 16s -ctr-volcano-pserver-1 1/1 Running 0 16s -ctr-volcano-trainer-0 1/1 Running 0 16s -ctr-volcano-trainer-1 1/1 Running 0 16 -``` - -选择一个PServer任务查看日志，看到PServer在监听端口，并对外提供服务 - -``` -root@volcano-paddlepaddle:~# kubectl logs ctr-volcano-pserver-0 - - -+ case "$1" in -+ start_fluid_process -+ pserver_label=paddle-job-pserver=fluid-ctr -+ trainer_label=paddle-job=fluid-ctr -+ hostname=ctr-volcano-pserver-0 -+ task_index= -+ '[' PSERVER == TRAINER ']' -+ '[' PSERVER == PSERVER ']' -+ stdbuf -oL python /root/k8s_tools.py wait_pods_running paddle-job-pserver=fluid-ctr 2 -label selector: paddle-job-pserver=fluid-ctr, desired: 2 -current cnt: 0 sleep for 5 seconds... -+ '[' PSERVER == TRAINER ']' -+ '[' PSERVER == WORKER ']' -++ python /root/k8s_tools.py fetch_endpoints paddle-job-pserver=fluid-ctr 30236 -+ export PADDLE_PSERVERS=172.20.0.148:30236,172.20.1.134:30237 -+ PADDLE_PSERVERS=172.20.0.148:30236,172.20.1.134:30237 -++ python /root/k8s_tools.py fetch_ips paddle-job=fluid-ctr -+ export PADDLE_TRAINER_IPS=172.20.0.147,172.20.1.133 -+ PADDLE_TRAINER_IPS=172.20.0.147,172.20.1.133 -+ '[' PSERVER == TRAINER ']' -+ '[' PSERVER == WORKER ']' -++ python /root/k8s_tools.py fetch_id paddle-job-pserver=fluid-ctr -+ task_index=0 -+ export PADDLE_TRAINER_ID=0 -+ PADDLE_TRAINER_ID=0 -+ export PADDLE_PSERVER_ID=0 -+ PADDLE_PSERVER_ID=0 -+ stdbuf -oL sh -c 'cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1' -2019-09-03 09:57:55,619 - INFO - run dist training -2019-09-03 09:57:55,708 - INFO - run pserver -get_pserver_program() is deprecated, call get_pserver_programs() to get pserver main and startup in a single call. -I0903 09:57:55.860916 41 grpc_server.cc:435] Server listening on 172.20.0.148:30236 selected port: -``` - - -选择一个Trainer任务查看日志，看到计算任务已经开始执行 -root@volcano-paddlepaddle:~# kubectl logs ctr-volcano-trainer-0 - -``` -root@volcano-paddlepaddle:~# kubectl logs ctr-volcano-trainer-0 - - -+ case "$1" in -+ start_fluid_process -+ pserver_label=paddle-job-pserver=fluid-ctr -+ trainer_label=paddle-job=fluid-ctr -+ hostname=ctr-volcano-trainer-0 -+ task_index= -+ '[' TRAINER == TRAINER ']' -+ stdbuf -oL python /root/k8s_tools.py wait_pods_running paddle-job-pserver=fluid-ctr 2 -label selector: paddle-job-pserver=fluid-ctr, desired: 2 -current cnt: 0 sleep for 5 seconds... -+ '[' TRAINER == TRAINER ']' -+ stdbuf -oL python /root/k8s_tools.py wait_pods_running paddle-job=fluid-ctr 2 -label selector: paddle-job=fluid-ctr, desired: 2 -++ python /root/k8s_tools.py fetch_endpoints paddle-job-pserver=fluid-ctr 30236 -+ export PADDLE_PSERVERS=172.20.0.148:30236,172.20.1.134:30237 -+ PADDLE_PSERVERS=172.20.0.148:30236,172.20.1.134:30237 -++ python /root/k8s_tools.py fetch_ips paddle-job=fluid-ctr -+ export PADDLE_TRAINER_IPS=172.20.0.147,172.20.1.133 -+ PADDLE_TRAINER_IPS=172.20.0.147,172.20.1.133 -+ '[' TRAINER == TRAINER ']' -+ check_failed_cnt 1 -+ max_failed=1 -++ python /root/k8s_tools.py count_pods_by_phase paddle-job=fluid-ctr Failed -+ failed_count=0 -+ '[' 0 -gt 1 ']' -++ python /root/k8s_tools.py fetch_id paddle-job=fluid-ctr -+ task_index=0 -+ export PADDLE_TRAINER_ID=0 -+ PADDLE_TRAINER_ID=0 -+ export PADDLE_PSERVER_ID=0 -+ PADDLE_PSERVER_ID=0 -+ stdbuf -oL sh -c 'cd /workspace/ctr && python train.py --is_local 0 --cloud_train 1' -2019-09-03 09:57:56,712 - INFO - run dist training -2019-09-03 09:57:56,773 - INFO - download the training materials - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed -100 433M 100 433M 0 0 96.2M 0 0:00:04 0:00:04 --:--:-- 96.2M -2019-09-03 09:58:27,648 - INFO - run trainer -2019-09-03 09:58:27,732 - WARNING - -I0903 09:58:27.734141 25 parallel_executor.cc:329] The number of CPUPlace, which is used in ParallelExecutor, is 2. And the Program will be copied 2 copies -I0903 09:58:27.937546 25 rpc_client.h:101] init rpc client with trainer_id 0 -2019-09-03 09:58:37,957 - INFO - TRAIN --> pass: 0 batch: 0 loss: 0.670620727539 auc: 0.510430537062, batch_auc: 0.510764985415 -2019-09-03 09:58:38,264 - INFO - TRAIN --> pass: 0 batch: 1 loss: 0.641319274902 auc: 0.503955813399, batch_auc: 0.503955813399 -2019-09-03 09:58:38,585 - INFO - TRAIN --> pass: 0 batch: 2 loss: 0.617138793945 auc: 0.50334993182, batch_auc: 0.50334993182 -2019-09-03 09:58:38,873 - INFO - TRAIN --> pass: 0 batch: 3 loss: 0.598490356445 auc: 0.507263818365, batch_auc: 0.507263818365 -2019-09-03 09:58:39,182 - INFO - TRAIN --> pass: 0 batch: 4 loss: 0.573976501465 auc: 0.510442316749, batch_auc: 0.51044231674 -``` - -等待大概70分钟，查看计算任务日志，发现任务已经安全退出 -``` -root@volcano-paddlepaddle:~# kubectl get pod | grep ctr-volcano - - -ctr-volcano-trainer-0 0/1 Completed 0 77m -ctr-volcano-trainer-1 0/1 Completed 0 77 -``` - -与此同时，在训练结束之后，我们可能需要训练出来的模型用于别处。在yaml文件当中，我们规定了该任务volcanosh/edlctr:v1镜像，该镜像的工作目录在/workspace/ctr下，在train.py当中有定义，会在每1000个batch或是每一轮pass（跑完一遍训练集）的时候，调用save_inference_model接口来保存模型。保存的模型在/workspace/ctr/models文件夹下。那么如何在任务结束后获取模型呢？我们建议以下几种方式。 - -在yaml文件当中trainer部分的spec当中定义volume，通过docker的volume映射容器路径和宿主机路径的机制，将/workspace/ctr/models文件夹映射到宿主机的文件夹中。接下来通过kubectl describe pod ctr-volcano-trainer-0，可以得知我们的模型所在的节点，接下来ssh登陆到对应的节点上，到宿主机被映射到路径下，就可以获取到训练出来到模型了。 - -如果需要更加灵活的，自动化的模型配送流程，可以在K8S集群上建立File Server和分布式文件系统，例如GlusterFS。将ctr-volcano-trainer-0容器内部的/workspace/ctr/models文件夹映射到GlusterFS的PVC（Persistent Volume Claim）上。通过ftp的wget/curl操作命令就可以实现模型的获取和配送。 - - - -综上，使用Volcano平台执行PaddlePaddle框架计算任务，可以实现计算任务的批量创建，任务的自动化管理，实现计算任务的自我管理。相较于普通的Replicaset+Job的模式，使用Volcano平台可以提升并行计算的管理效率。 - -作者 -董大祥，@guru4elephant, PaddlePaddle Architect, Principal Architect, Baidu - -王嘉炜，@wangjiawei04, PaddlePaddle Engineer, Senior Engineer, Baidu - -于佃海，@raindrops2sea, PaddlePaddle Architect, Distinguished Architect, Baidu - -张经辉, @sivanzcw, Volcano Contributor, Cloud Native software engineer, Huawei - -马达, @k82cn, Kubernetes Maintainer, SIG-Scheduling Co-Leader, Volcano Lead, Huawei - - - -参考文献 - -1.PaddlePaddle官网 - -https://www.paddlepaddle.org.cn - -2.Paddle on Spark - -https://github.com/hohdiy/paddle_on_spark/blob/master/doc/paddle_on_spark.md - -3.Run Deep Learning with PaddlePaddle on Kubernetes - -https://kubernetes.io/blog/2017/02/run-deep-learning-with-paddlepaddle-on-kubernetes/ - -4.Volcano官网 - -https://volcano.sh - -5.Volcao社区 - -https://github.com/volcano-sh/volcano - -6.百度CTR Demo - -https://www.paddlepaddle.org.cn/documentation/docs/zh/1.5/user_guides/howto/training/deploy_ctr_on_baidu_cloud_cn.html - -7.CTR-volcano 配置文件 - -https://github.com/volcano-sh/volcano/blob/master/example/integrations/paddlepaddle/ctr-paddlepaddle-on-volcano.yam - - - - - diff --git a/content/zh/blog/pengcheng.md b/content/zh/blog/pengcheng.md deleted file mode 100644 index 075cbfb0..00000000 --- a/content/zh/blog/pengcheng.md +++ /dev/null @@ -1,272 +0,0 @@ -+++ -title = "鹏城实验室启智章鱼教你彻底摆脱Kubernetes集群资源抢占难题" -description = "Volcano在科研领域的落地案例" -subtitle = "" - -date = 2021-06-01 -lastmod = 2021-08-26 -datemonth = "Aug" -dateyear = "2021" -dateday = 26 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["李俊茂/鹏城实验室人工智能中心系统架构师"] - -tags = ["Practice"] -summary = "Volcano在科研领域的落地案例" - -# Add menu entry to sidebar. -linktitle = "鹏城实验室启智章鱼教你彻底摆脱Kubernetes集群资源抢占难题" -[menu.posts] -parent = "tutorials" -weight = 10 -+++ - ->本文2021年6月1日首发于容器魔方微信公众号，原文链接[鹏城实验室启智章鱼教你彻底摆脱Kubernetes集群资源抢占难题](https://mp.weixin.qq.com/s/h4T7KbAiQZTKepYcTcgdlA) - -## 启智章鱼平台介绍 - -启智章鱼（OPENI-OCTOPUS）是一个集群管理和资源调度系统，由鹏城实验室、北京大学 、中国科学技术大学进行开发和维护。 - -- 完全开源：遵守Open-Intelligence许可 - -- 利用Kubernetes部署、管理和调度 - -- 支持在集群运行AI任务作业，支持GPU，NPU，FPGA，华为升腾芯片，寒武纪MLU等硬件 - -- 适用于AI的高性能网络，支持IB网络 - -- 提供监控分析工具，包括网络，平台和AI作业的监控分析 - -- 支持主流的深度学习框架 - -- 采用微服务结构 - -{{
}} - - -最底层为硬件层，可接入多种不同类型的异构硬件，包括CPU、GPU、NPU、FPGA等。这需要对不同的硬件进行适配，以便于上一层的Kubernetes服务识别和统一管理。 - - - -往上一层为平台的中间层，左边是管理节点的功能，一个是kubernetes本身的功能，包括编排规划，控制管理，调度方面我们使用的是Volcano。然后通过api-server和我们开发集成的服务通信。 - - - -我们开发的rest-server模块负责系统的核心功能，并集成了Grafana,普罗米修斯等监控工具，ES日志工具，Jupyterlab代理，模型仓库等功能。 - - - -右边是计算节点的能力，计算节点上有镜像工厂、运维监控、作业监控、kubebox客户端、用户登进容器的Jupyterlab客户端。 - - - -最上层是系统所提供的服务，对用户而言它拥有数据引擎，模型仓库，项目中心的能力，还能通过异地互联的方式，为异地用户提供集群服务。 - - - -## 业务场景与挑战 - -{{
}} - -启智章鱼平台在设立之初直到目前主要是为实验室内外的科研团队服务。它们包含交通、医疗、金融等方面的模型开发，模型训练或者模型推理方面的需求。如一些车辆追踪算法，医疗影像识别，辅助诊断算法，金融领域的量化算法。这些各个领域的算法会用到一些深度学习的算法，对计算资源的要求比较高。 - - - -平台要解决传统平台的模型孤岛、数据孤岛、算力孤岛问题。 - -- 在模型层面，提供多态异构的模型引擎，支持常见的开源计算框架，并提供各种常见计算框架的模型转换的能力。 - -- 在数据层，提供多源异质数据引擎，支持异构数据融合，并且提供半自动数据标注的能力。 - -- 在资源层，提供AI分布式计算引擎，一方面是对异构硬件的统一表示，另外的重点是对作业的调度。 - - -__业务场景__ - -- 业务以面向实验室内外的科研AI任务为主，包括智慧交通、医疗、金融等领域的算法训练与推理 - -- 高端异构硬件资源优势，150P+级算力集群和10PB级高速存储 - -- 支持快速和灵活部署，系统运行可靠稳定，方便外部团队使用 - - -__挑战__ - -- 缺少一个高性能计算平台，满足复杂场景的业务诉求 - -- 高效利用异构硬件资源，支持灵活调度策略，需解决资源抢占难题，避免关键任务资源饿死现象 - -- 系统架构需可扩展，服务高可用 - - -## Why Volcano? - -针对这些挑战，我们在选型时主要考虑有以下几个方面： - -1）首先自研还是开源。在调研了目前社区已有的开源项目之后，发现它们能够满足我们的基本诉求和对复杂业务场景的需求。 - - - -2）基于减少开发量的考虑，选择拿来主义。我们对比了常见的开源资源调度器。首先是K8S默认调度器，该调度器对批量调度并不友好，不能满足我们的要求。Yarn调度器是基于Hadoop的，而我们现在的架构已经转型为基于k8s，也不符合我们的要求。另外是kube-batch和volcano，kube-batch是volcano的前身，且volcano对深度学习支持比较友好，对各种常见计算框架的任务类型都有支持，另外它有一套插件化的调度策略实现机制，方便实现新的插件来开发我们的特定调度策略。 - - - -因此我们选择集成volcano，使用volcano的能力来应对启智章鱼面对的业务场景挑战。__具体来说，volcano对我们有以下价值：__ - -- 完善的架构和生态，社区反馈很及时，开发者比较多，发展很快 - -- 上文提到的插件机制，方便自定义新的调度策略 - -- Volcano支持任务队列的机制，这对我们很重要。我们基于任务队列，能够将集群进行逻辑分组，方便对用户进行项目制管理。一方面能够对不同项目配置不同限额的计算资源，另一方面能将不同类型的作业分到不同的任务队列中进行管理。对作业和计算资源的管理更加精细化。 - -- 可以使用Volcano的binpack等插件组成多种多样的调度策略，以binpack插件为例，其使用的装箱算法能够解决资源碎片的问题，能够很好的提高集群的整体使用率。 - - -## 基于Volcano的二次开发 - -Volcano虽然功能很强大，但为了能够进一步应对复杂的业务场景，我们也做了二次开发，新增了一些新的能力。 - - - -我们做的第一个二次开发是资源状态的统计与管理功能。这里的资源不单指集群计算资源，也指用户提交作业后，k8s服务中生成的Job、Task、Pod等资源。 - -{{
}} - -我们需要对整个资源的状态信息，状态转换都有很好的掌握，并且可以自定义资源状态转换的条件与回调事件，可以在业务逻辑层订阅自定义的Event与相对应的Policy. - - - -考虑以下的一种业务场景。有一种使用集成学习算法的训练任务，一般可以使用分布式的训练方式。它有一个结合模块，还有若干个体学习器，每个个体学习器使用一种算法来训练任务，结合模块结合各个个体学习器的结果，来输出最终结果。如果结合模块算出了最终的结果，整个训练任务就可以结束。在基于k8s的实现中，使用集成学习的训练作业整体是一个Job，结合模块和各个个体学习器都是这个Job下面的一个Task，Task下面可以再创建一个或者多个副本Pod。我们的场景需求是只要结合模块这个Task运行成功，就让整个Job退出，而不必等到所有的Task都成功再退出。那我们就需要针对这种场景，可以在调度器自定义一种Job退出的策略，并且可以在业务层面去使用。集成学习这个场景需要一种特定的策略，其他场景也需要相应的策略。这就要求我们去二次开发volcano调度器。 - -{{
}} - -上图的交互图显示了启智章鱼，k8s，volcano之间关于作业状态的信息传递。 - -首先，volcano和启智章鱼都会一直监听k8s所有作业的状态；然后，用户提交了作业到k8s之后，volcano会根据监听到的k8s作业启动的pod状态去更新Job的状态；最后，启智章鱼监听到这些Job的状态改变，会去处理这些状态改变事件。 - - - -基于这个交互图，我们发现 __问题的关键在于volcano如何去更新Job在k8s中的状态。__ - - - -__我们做了以下几点二次开发来实现目标：__ - -1）实现Job，Task，Replica的三级别状态机 - -- 资源状态统计与回显更加详细 - -- 更加细粒度的Job生命周期管理 - - - -2）实现自定义Event和Policy，能够解决类似集成学习应用场景的问题，自定义一个Event叫做MainTaskEvent, 在确定的Task运行结束时在调度器发布这个事件来结束整个Job - - - -3）实现生命周期回调钩子，可以在三个级别的状态机的任一状态转移事件中都加入回调钩子，以此支撑多样化的业务场景需求。例如我们的计费功能实现，就是基于Job的开始事件和结束事件来统计Job的运行时长。 - - -__Privilege Action__ - -问题 - -- 任务队列资源饿死现象，大作业一直等待 - -- 紧急作业、关键作业需要优先调度 - -- 用户作业可能在线开发，不可随意终止 - - - -Volcano已有能力 - -- 同队列中不同优先级Job的抢占，无法区分Job所来自的租户身份 - -- 以Pod为单位进行驱逐 - -- 立即抢占 - - - -需求 - -- 同队列中作业来自不同租户，不同租户拥有不同优先级和抢占权限 - -- 以Job为单位进行驱逐 - -- 延时抢占 - -{{
}} - -上图为延时抢占插件的流程图。最左边是插件在调度器的运行逻辑。中间是k8s服务，最右边是启智章鱼的核心模块。 - - - -具体的实现是在volcano中去找出具体需要被抢占的作业，这些作业所占用的计算资源要足够满足当前等待的优先作业。然后更新k8s中这些作业的状态，启智章鱼的核心模块一监听到这些状态改变，就会启动定时器来准备驱逐这些任务。如果定期器倒计时还没结束，抢占任务被取消或者抢占任务已经因为其他作业释放资源，满足了运行条件而不需要继续抢占，这个定时器也会被取消。 - - - -__具体看下我们做了什么开发：__ - -{{
}} - -1）为每个作业增加了一个布尔属性，叫Preemt，表示这个作业是否为一个抢占作业。 - -- 队列内抢占作业只能抢占同个队列中优先级较低的作业。 - - - -2）以Job为单位进行驱逐，而不是以Pod为单位驱逐。 - -- 驱逐待选的Pod按照所属JobID排序，可减少受影响的Job数量 - -- 由调度器通知启智章鱼，在业务层对Job进行停止 - - - -3）延时抢占 - -- 为Job状态机增加Privileged和WillEvicted状态 - -- 进入Privileged和WillEvicted状态的Job不可被其他Job抢占 - -- 如果抢占或被抢占的Job状态发生变化，另一方的状态也要相应改变 - -{{
}} - - -## 收益 - -__能力增强__ - -- 满足分布式大规模训练任务的需求 - -- 支持多种AI计算框架 - -- 插件化调度器，支持自定义开发，适应复杂业务场景 - -- 多队列调度，实现硬件资源的分组与组间动态资源分配 - - - -__性能优化__ - -- 硬件资源利用率大幅提高，达到90%以上 - -- 作业平均调度时延大幅降低，由使用Yarn调度器的平均任务等待时间60秒降到现在的10秒 - -- 系统稳定增强，集群节点资源利用率平衡度好，减少了节点间因资源使用差异度引起的运维工作 - -{{
}} - -上图为系统在Grafana显示情况： - -- 管理120+个节点，总GPU卡数为1.1K张，GPU在超负荷情况下使用率可达90%以上 - -- 各节点间资源利用率平衡性较好，差异度低于20% - -- 2019年上线以来，运行作业数为12w+ \ No newline at end of file diff --git a/content/zh/blog/ruitian.md b/content/zh/blog/ruitian.md deleted file mode 100644 index 409f697c..00000000 --- a/content/zh/blog/ruitian.md +++ /dev/null @@ -1,381 +0,0 @@ -+++ -title = "基于Volcano的锐天离线高性能计算最佳实践" -description = "深入介绍Volcano在金融领域的应用实践案例" -subtitle = "" - -date = 2021-01-05 -lastmod = 2021-08-23 -datemonth = "Dec" -dateyear = "2020" -dateday = 23 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["程运之/锐天投资技术总监", "徐征宇/锐天投资研发工程师"] - -tags = ["Practice"] -summary = "深入介绍Volcano在金融领域的应用实践案例" - -# Add menu entry to sidebar. -linktitle = "基于Volcano的锐天离线高性能计算最佳实践" -[menu.posts] -parent = "tutorials" -weight = 5 -+++ - ->本文2021年1月5日首发于容器魔方微信公众号，原文链接[基于Volcano的锐天离线高性能计算最佳实践](https://mp.weixin.qq.com/s/FDYExtj93lCrXmiFRozBPA) - -Volcano是一个基于Kubernetes的云原生批量计算平台，也是CNCF的首个批量计算项目。 - -Volcano 主要用于AI、大数据、基因、渲染等诸多高性能计算场景，对主流通用计算框架均有很好的支持。它提供高性能计算任务调度，异构设备管理，任务运行时管理等能力，目前在很多领域都已落地应用。__本篇文章以锐天投资基于Volcano进行的离线高性能计算最佳实践为线索，深度解读Volcano在金融领域技术架构中的定制与应用。__ - - -## Ruitian离线计算集群简介 - -锐天投资是一家私募量化基金公司，公司致力于通过多元的程序化交易策略，帮助客户实现资金的增值。锐天的离线计算集群是一个面向策略开发、研究大数据处理任务的集群。集群通过对海量数据进行分析研究，得出优秀的量化模型，用于股票及期货交易。 - - - -在锐天成立初期，我们使用业界成熟的 Yarn 调度批处理作业，并使用 Ceph 存储海量数据。随着公司规模的扩大，策略人员使用的环境不断变化，对于不同研究环境的需求促使我们开始调研容器技术来实现多环境研究的问题。 - - - -随着近几年 Kubernetes 的成熟稳定，在计算集群中使用容器技术已经几乎等同于使用Kubernetes。__但原生的Kubernetes调度器并未提供批处理任务的调度能力，即：__ - -- 支持在一个Job中运行多个Pod - -- 支持Job指定到不同的队列，用队列进行公平调度 - -- 支持一定程度的Gang Scheduling - -- 支持指定在多少个pod成功的情况下，任务就被算作成功 - -- 支持DRF算法 - - - -在调研期间，我们发现了Volcano 项目。在满足我们需求的同时，__它提供了丰富的调度策略与Job控制能力。__ 其简洁的结构成为了我们决定从Yarn迁移到Kubernetes的最后一块拼图。 - - - -## 迁移至Volcano - -由于集群用户主要为策略研究人员，他们对Kubernetes缺乏了解。我们希望将Kubernetes的相关细节对外封装，于是开发了一个 Jobctl 作为提交工具，来生成Volcano的Job模板。 - -### Volcano任务模板 - -__方案1：一个作业多个任务__ - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: awesome-job -spec: - minAvailable: 1 - tasks: - - name: simulation1 - replicas: 1 - template: - spec: - restartPolicy: Never - containers: - - name: worker - image: rt-python:latest - resources: - requests: - cpu: 1 - memory: 1Gi - limits: - cpu: 1 - memory: 1Gi - args: - - bash - - -c - - |- - python run.py --pickle-file /data/simulation/1.pickle - - name: simulation2 - replicas: 1 - template: - spec: - restartPolicy: Never - containers: - - name: worker - image: rt-python:latest - resources: - requests: - cpu: 1 - memory: 1Gi - limits: - cpu: 1 - memory: 1Gi - args: - - bash - - -c - - |- - python run.py --pickle-file /data/simulation/2.pickle - - name: simulation3 - replicas: 1 - template: - spec: - restartPolicy: Never - containers: - - name: worker - image: rt-python:latest - resources: - requests: - cpu: 1 - memory: 1Gi - limits: - cpu: 1 - memory: 1Gi - args: - - bash - - -c - - |- - python run.py --pickle-file /data/simulation/3.pickle -``` - -这个方式可以针对不同的Pod设置不同的参数，并且可以对不同的pod使用不同的镜像，让它们各自负责不同的子任务类型，非常灵活。 - - - -由于在大部分情况下，不同的子任务通常负责不同的时间段，互相之间独立没有依赖，通过将 minAvaliable 设置为 1 可以让Pod全部参与调度的同时，Job处于Running状态，便于管理。但是在试运行期间，我们发现有的任务无法提交。排查后发现，有些研究人员的任务并发Pod数量巨大，超过5000。生成出的yaml的大小超过了1.5MiB，已经超出了etcd的默认最大请求大小，因此当任务数量过大时会导致无法提交作业。 - - - -考虑到集群Job、Pod数量众多，etcd的原有压力已经很大，我们没有简单的改变默认最大请求大小，而是换了一种思路。 - - - - - -__方案2：一个作业一个任务多个副本__ - - - -由于在大多数情况下，作业里的所有任务仅仅是参数不一致，因此可借助任务的多副本功能，每个任务根据自己副本的id加载对应的参数文件即可，这样就解决了etcd的请求大小限制。 - -``` -apiVersion: batch.volcano.sh/v1alpha1 -kind: Job -metadata: - name: awesome-job -spec: - minAvailable: 1 - tasks: - - name: simulation - replicas: 10 - template: - spec: - restartPolicy: Never - containers: - - name: worker - image: rt-python:latest - resources: - requests: - cpu: 1 - memory: 1Gi - limits: - cpu: 1 - memory: 1Gi - args: - - bash - - -c - - |- - python -u call_module_func.py --pickle-file /data/simulation/.pickle module.submodule magic_function -``` - -call_module_func.py 是一个引导脚本，通过 configmap 挂载进容器，负责以下工作： - -- 将 转换成 replicas 当前的 id 号（通过获取容器内的 host 名后缀编号获取，如 host名为 awesome-job-awesome-job-1 ，id 号即为 1） - -- 加载 pickle （通过 pvc 挂载进容器） 参数，传递给模块 module.submodule 里的函数 magic_function 运行。 - - -### Volcano定制 -__minSuccess特性：__ - -我们的大部分任务并不需要Gang Scheduler特性，但是我们需要在所有的子任务成功的情况下，才能够认为整个任务成功。 - - - -在这种情况下，minAvailable 这个参数就不能很好的符合我们的需求，于是我们增加了 minSuccess 参数，将判断任务成功的逻辑从minAvailable中解耦合。 - -``` -minSuccess := ps.job.Job.Spec.MinSuccess -if minSuccess == 0 { - minSuccess = jobReplicas -} - -if status.Succeeded >= minSuccess { - status.State.Phase = vcbatch.Completed - return true -} - -if status.Succeeded+status.Failed == jobReplicas { - if status.Failed != 0 { - status.State.Phase = vcbatch.Failed - } else { - status.State.Phase = vcbatch.Completed - } - return true -} -``` - -__autoMemoryScale特性:__ - - - -使用Volcano提交任务需要指定申请的cpu和memory，而我们的大部分策略人员无法对自己的程序有很好的内存估计。于是我们定制了 autoMemoryScale 功能，监控OOM事件。 - - - -如果程序是由于OOM退出的，它将自动扩充内存，重新调度，从而减少策略人员试错参与。 - - -``` -for i := 0; i< int(ts.Replicas); i++ { - podName := fmt.Sprintf(jobhelpers.PodNameFmt, job.Name, name, i) - if pod, found := pods[pdName]; found { - if len(pod.Status.ContainerStatuses) == 0 { - continue - } - - reason := pod.Status.ContainerStatuses[0].State.Terminated.Reason - - if reason == "OOMKilled" { - podToScaleUp = append(podToScaleUp, pod) - jobResources := ts.Template.Spec.Containers[0].Resources - podResources := pod.Spec.Containers[0].Resources - - jobReqMem, _ := jobResources.Requests[v1.ResourceMemory] - podReqMem, _ := podResources.Requests[v1.ResourceMemory] - - if podReqMem.Value() >= jobReqMem.Value() { - scaleUpResource(jobResources.Requests, job.Spec.ScaleUpJobResourceRate) - scaleUpResource(jobResources.Limits, job.Spec.ScaleUpJobResourceRate) - ts.Template.Spec.Containers[0].Resources = jobResources - job.Spec.Task[taskId] = ts - } - } - } - } -``` - -__nodeZone特性:__ - -在原先Yarn的集群中，我们通过分Partition可以强行预留一部分机器给紧急任务使用。同样的特性我们需要迁移到Kubernetes也具备。 - - - -开始我们的方案是创建一个单独的 daily 队列，且该队列的权重相对较小。此外我们划分了某些节点，专门给该队列运行，而且指定 daily 队列的任务需使用 nodelSelector 来选择这些节点。 - - - -但随后我们发现当集群负载高的情况下，daily 队列能划分到的资源无法完全使用。我们分析发现了如下原因： - - - -假如有三个队列: - -``` -name weight -Q1 45 -Q2 45 -daily 10 -``` - -集群整体资源100CPU，100Gi，划分了20CPU，20GiB给 daily 队列。当各个队列高负载的情况下，daily队列由于公平性只能获取10CPU和10Gi资源，即使使用了nodeSelector 也没有办法满足我们的需求。 - - - -我们的解决方案是让调度器支持 node-selector（即node-zone），调度器只需负责匹配的节点，且去除掉了daily队列。当某个调度器上的各个队列很满的情况下，并不会影响队列在另一空闲调度器上的资源公平性划分，即如果该空闲调度器仅运行有一个队列，该队列就可以申请使用全部的资源。 - - - -对于不同特性的任务，用户需要选择的是不同的 调度器 而非某个特殊的 队列，避免了由于公平性导致的资源使用不充足情形的发生。 - - - -于是在 Volcano 中我们实现了选择特定label机器的特性，由多个scheduler实例来负责对多个node节点域进行调度。 - -``` -sc.nodeInformer.Informer().AddEventHandlerWithResyncPeriod( - cache.FilteringResourceEventHandler { - FilterFunc: func(obj interface{}) bool { - switch v := obj.(type) { - case *v1.Node: - nodeZone := v.Labels["node-zone"] - return nodeZone == sc.nodeZone - default: - return false - } - }, - Handler: cache.ResourceEventHandlerFuncs { - AddFunc: sc.AddNode, - UpdateFunc: sc.UpdateNode, - DeleteFunc: sc.DeleteNode, - }, - }, - 0, -) -``` - -### Volcano 监控 -由于当前Volcano社区提供的监控更多是调度性能指标，我们根据自身需求采集了一些额外的指标，并编写了一个简单的export server提供给我们的监控系统。 - - - -- volcano 生成的 Pod 是不带有队列标签的，这就导致了队列资源的搜集变得困难，因此我们特意对每个 task 添加了队列的标签 - - - -- 输出队列 Capability - - - -- 输出 Job 的开始和完成时间 - - - -有了这些额外指标，就可在 Grafana上构建出集群整体资源、队列资源、节点资源、用户作业的进度等相关的监控信息，实时了解到集群的使用率，也方便问题的排查，尤其为何作业处于Pending状态（可能队列/集群资源申请满了，每个节点剩余资源均不够单个任务申请）。 - - - - - -{{
}} - -{{
}} - - -### WatchDog - - -对于Volcano资源相关的一些自动化任务，我们编写了一个名为 WatchDog 的组件，从外部进行自动化运维。 - - - -__队列 Capability 自动更新：__ Capability是每个队列的资源使用上限，对其维护是一件繁琐的事情，每次新增节点或移除节点均需对其作出调整，所以在WatchDog中，监听节点资源信息，根据队列的百分比配置，动态的更新其 Capability - - - -__任务状态自动报警：__ 当任务完成或失败时，通知用户，便于及时了解到任务运行的状态。 - - - -__任务资源利用率通知：__ 在任务状态报警中，通过从监控系统获取任务申请和实际使用资源量，告知用户，便于其调整申请资源，使集群资源利用率提升。 - -## 总结 - -Volcano 是我们迁移至 Kubernetes 过程中至关重要的组件，其简洁清晰的设计，让我们能非常方便的自己定制化调度行为。 - - - -__截至目前，Volcano已在锐天生产环境稳定运行半年以上，峰值可调度作业10万+/天。__ 我们也时刻关注着社区的动态，积极参与项目的开发与设计，同时希望能有更多开源作者能加入到 Volcano 社区的建设中来，让 Volcano 能更加灵活、高效、智能的处理各种复杂作业场景。 - - ->公司简介：上海锐天投资公司成立于2013年，是一家重视科学研究和技术积累的量化基金公司。公司拥有业内领先的策略研发和回测平台，数百台高性能服务器组成的自建集群。创始人在全球顶级对冲基金取得相当成绩后归国创业。截至2019年一季度，锐天已跻身国内量化交易领域第一梯队，管理规模逾百亿人民币，管理基金90多支。公司官网：https://www.ruitiancapital.com/ - diff --git a/content/zh/blog/ruitian2.md b/content/zh/blog/ruitian2.md deleted file mode 100644 index b42aa2ea..00000000 --- a/content/zh/blog/ruitian2.md +++ /dev/null @@ -1,289 +0,0 @@ -+++ -title = "锐天投资基于Volcano的大规模分布式离线计算平台的应用实践" -description = "Volcano在金融领域的落地案例分享" -subtitle = "" - -date = 2021-06-15 -lastmod = 2021-08-25 -datemonth = "Aug" -dateyear = "2021" -dateday = 25 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["程运之/锐天投资技术架构部研发总监"] - -tags = ["Practice"] -summary = "深入介绍Volcano在锐天投资落地实践过程中遇到的挑战和解决方案" - -# Add menu entry to sidebar. -linktitle = "锐天投资基于Volcano的大规模分布式离线计算平台的应用实践" -[menu.posts] -parent = "tutorials" -weight = 8 -+++ - ->本文2021年6月15日首发于容器魔方微信公众号，原文链接[锐天投资基于Volcano的大规模分布式离线计算平台的应用实践](https://mp.weixin.qq.com/s/dC4IDNG7FMGLigNJaj_Qug) - -## 业务场景与技术选型 - -__业务场景__ - -- 为策略人员提供研究以及开发的虚拟机 - -- AI训练与推理 - -- 数据ETL - -- 通用的分布式批处理任务 - -__Why Use Kubernetes？__ - -分布式批处理任务平台用于管理计算资源和存储资源。对计算资源而言，我们最终选用了Kubernetes来进行管理，主要原因有以下几点： - -- 通过 Container 隔离用户环境:我们公司大概有4~5组不同的用户，其使用的开发环境、开发策略都是不同的，这对于整体集群环境管理而言是非常大的挑战，使用Container后，所有的环境可以封装在Container，直接通过k8进行调度，来解决环境隔离问题。 - -- 通过 Device Plugin 支持 GPU 等异构设备 - -- ETCD 的中心式数据存储机制 - -- 活跃健全的技术生态 - -- Go 语言符合公司技术栈 - - -__Why Use CephFS__ - - -存储部分我们使用的是CephFS，它是ceph提供的分布式文件存储的一种接口形式，ceph本身提供三种存储接口：S3、快存储以及CephFS。我们使用CephFS主要有以下几点原因： - -- Posix Filesystem 权限与接口：对于我们所有人而言，最熟悉最自然的文件存储及获取方式是本地文件系统，而CephFS可提供稳定文件系统挂载。并且对我们这个行业而言，所有的用户多租户的情况下，每个用户他有自己的一个uid，每个用户他自己的数据都只能自己来访问。那么Posix Filesystem的一个权限机制，可以让用户无缝的把自己的文件权限迁移到sap上。 - -- 强一致性：在 a节点写入一个文件，在b节点可直接读取。 - -- 同时支持大规模小文件存取与大带宽 IO - -- 层级化硬件支持 - -- Kubernetes ReadWriteMany PV - -## Why Volcano - -__Why not default-scheduler__ - -Default-scheduler无法满足我们需求的原因有4点：缺少队列调度、缺少公平调度、缺少多租户隔离、缺少高级调度策略，如Gang-Scheduling。最重要的是公平调度和高级调度策略。所谓的公平角度，是指当有很多人提交了非常多的任务排队正在等待时，应该来运行谁的任务？当集群空出来或有资源时，到底应该运行谁的任务？要完成这一点，每个队列映射于一个团队，基于队列的公平角度，每一个namespace对应一个用户，基于用户的公平调度，这两点都是必不可少的。所以在这两点上，default-scheduler完全没有办法满足我们的需求。 - - - -因此我们求助了社区。kube-batch是社区下面的批处理调度器，它只是一个调度器，不提供除调度以外的任何解决方案，这也是我们没有使用kube-batch的最主要原因。要做一个批处理方案，不只需要调度器的功能，还需要对于环境及其他任务 CRD做一系列处理。没有及时处理的话很难作为平台来使用。 - - - -{{
}} - -__Why is Volcano__ - -- 支持公平调度 - -- 支持丰富的高级调度策略，如 Gang-Scheduling、Binpack 等 - -- 支持通过 SSH plugin 方式实现 Pod 互访 - -- 支持通过 ENV plugin 方式向 Pod 注入任务依赖，天然支持 Tensorflow Worker Sharding - -- 支持 SVC plugin 方式对外提供服务 - -有了这样一个调度平台，基本可以满足我们的整体需求。 - - -## 系统架构 - -__服务架构__ - -{{
}} - -- 基于 Ceph 提供高性能存储 - -- 基于 Kubernetes 管理多种异构硬件 - -- 通过 Loki + Grafana 同时作为用户面板与监控面板 - -- 中间件与应用层混部，充分使用集群资源 - -- 基于 Batch Job 形式，扩展多种业务场景 - - -__多租户__ - -{{
}} - -用户提交任务还会遇到多租户的问题。例如用户提交一个pod到集群，这个pod运行用户与Uid是什么？默认情况下，它的运行用户Uid是image制造者的Uid,相当于所有用户提交的pod的Uid变成同一个人,这是不行的，因为他们获取的数据和生成的数据互相之间是不能看的。 - - - -在这种情况下，我们的解决方案是通过K8s Namespace做整个用户所有资源的隔离，namespace对应于一个用户，通过已有的LDAP服务和OIDC与开发信息对接，给用户一个认证，通过RBAC进行用户资源的授权，授权用户使用以下的一个Pod Security Policy，Pod Security Policy是直接限制用户提交时，必须采用SecurityContext写上其uid与gid是什么？用户运行时整个的环境全部以此为准。 - - - -有了Pod Security Policy后，用户提交的pod必须以其uid和gid进行体现。但访问的数据都在Ceph上，也就是其gid和uid所能够访问的数据。以此来解决多租户的问题。 - - -__工作流__ - -{{
}} - -解决刚才所有的问题后，基础的工作流就出现了。本地渲染 Job Yaml 进行提交，用户所有的依赖数据同步 CephFS，并通过 PVC挂载 Pod，每个用户 Namespace 下，拥有自己目录的 PVC 权限，全部是通过IBS进行权限管控，以此来提交任务到整个集群运行。 - - -## Volcano 的深度定制 - -基础的提交框架平台我们选择了直接给用户提供库的形式，自研提交工具Jobctl，其开发完成后，自动支持两种使用方式，一种是命令行里面直接使用,另外一种是作为Python列表input到用户本身的notebook进行使用或直接体现到自己的Python脚本。Jobctl同时支持两种提交状态形式，一种是异步的，也就是说不停的向整个集群提交任务，任务提交完成后可直接退出；另一种是同步，job control提交完一个任务后，并且watch这个任务，当任务全部结束后，才返回给用户，这是返回给用户的工作流。 - - - -有了Jobctl后，还可以对用户隔离整个K8s复杂性,同时支持命令行提交和 Python Lib 集成,并且提供最基础的按 replicas 并行与按天并行。 - -{{
}} - -__OOM Auto Scale Up__ - -{{
}} - -首先做的第一个定制是OOM的时候对整个任务进行资源Scale Up，因为用户不能很好预估内存，OOM 后需要重复提交验证，因此我们通过直接定制 OOMKill Auto Scale Up ，通过修改Volcano Controller自动把OOMKill的pod所申请的资源按比例进行放大，放大后自动重复提交，并且提交完成后通知用户。按照这个功能让用户的整个任务在他自己不参与的情况下自动重复提交，得到合理内存申请值,这里其实使用了上文提到的Volcano policy event机制。 - -__MinSuccess__ - -{{
}} - -- minAvailable 个 Pod 结束，则任务结束 - -- 非 Gang 的任务难以灵活调度 - -{{
}} - -- minSuccess 个 Pod 结束，则任务结束 - -- 解耦 Gang 所需 Task 数量和完成 Job 所需任务数量 - - -__NodeZone__ - -{{
}} - -- 一个 Volcano 实例管理所有节点 - -- 无法解决 Noisy Neighbor - -- 无法做紧急资源预留 - -{{
}} - -- 多个 Volcano 实例，分别管理多个Zone - -- 对某些任务做物理隔离 - -__Volcano Namespace Quota__ - -{{
}} - -默认的k8s Quota并不能满足我们的需求，因为触发原生 Namespace Quota 时，Pod 直接 Fail，因此我们在Volcano级别做了Quota。 - -{{
}} - -- 触发 Volcano Namespace Quota 时，Pod 排队延迟创建 - -__Volcano 监控与报警__ - -{{
}} - -Volcano Exporter - -- 输出 Task 的队列标签 - -- 输出队列 Capability - -- 输出 Job 的开始完成时间 - - - -WatchDog 组件 - -- 注册 Informer，并收集 Metrics - -- 负责任务失败与使用率报警 - -- 自动更新队列的 Capability - - -__任务状态面板__ - -{{
}} - -上层来承载所有job的信息，并且会有一个状态表来表示任务完成情况，以此来大致判断任务的形式。下面的三点是CPU、memory和networks的资源使用情况。除了正坐标轴外，副坐标轴的一些竖线是指它浪费的集群资源。这些浪费的集群资源帮助用户实时判断任务运行状态的节点发生的浪费情况。所以我们需要通过这样的时间序列状态表，来提醒用户。 - - -__集群资源面板__ - -{{
}} - -通过graph的话，对整体队列资源、 CPU、memory以及所有队列资源的使用，用户需要看到所有节点的资源使用情况，因为有些非常大的消耗任务，比如一个任务有可能需要申请300或500的内存，但并不是所有的信息都支持单个节点内存使用的，所以只有总体的内存使用率是不够的，还需要有每个节点内存使用率。 - - -## 高并发场景下的挑战与解决方案 - -随着场景变多和平台用户的增加，我们发现了一些问题，集群规模单集群计算节点数量200 、存储总量1.5PB，读写带宽15GB/s、同时承载长时间任务（1week）与短时间任务（1min）、Pod 每日增长量10W ~ 30W，这是一个非常恐怖的规模。 - -__问题1：单个Job对象过大__ - -{{
}} - -问题： - -- 在有大量 Pod 时超过 ETCD Max Request Size (1.5MB) - -- 直接调整 Max Request Size，大量的 Object对 ETCD 造成冲击 - - - -解决： - -- 通过单个 Task 多 Replica 的形式提交任务 - -- Pod 内部通过 ENV plugin 插件提供的信息，以 Sharding 的形式读取参数 - - -__问题二：Out Of Cpu/Memory__ - -{{
}} - -问题： - -- 节点数少 + 大量短时任务不停调度 - -- Kubelet PLEG 压力大，Pod Binding 时间过长 - -{{
}} - - -问题： - -- 节点数少 + 大量短时任务不停调度 - -- Kubelet PLEG 压力大，Pod Binding 时间过长 - -- Volcano 默认 session 间隔时间为 1s，造成 Cache Snapshot不一致 - -- Out of CPU + Out of Memory - - - -解决： - -- 增加 Node Binding Task Number - -- 在 Session 开始创建 Snapshot 时，跳过 Binding Task Number > 0 的 Node - -{{
}} - -上文提到各种各样的深度定制，包括bug fix和修改，我们都已提交给了社区，并且我们有一位成员已经成为 Volcano 社区 Approver。非常荣幸能够参与社区共同的改进，也希望更多的开发者能够加入Volcano社区。 \ No newline at end of file diff --git a/content/zh/blog/xiaohongshu.md b/content/zh/blog/xiaohongshu.md deleted file mode 100644 index cf079f9c..00000000 --- a/content/zh/blog/xiaohongshu.md +++ /dev/null @@ -1,175 +0,0 @@ -+++ -title = "小红书基于Volcano的大规模离线与在线推荐模型训练实践" -description = "小红书使用高性能批量计算平台Volcano的落地实践过程" -subtitle = "" - -date = 2021-05-27 -lastmod = 2021-08-24 -datemonth = "Dec" -dateyear = "2020" -dateday = 24 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "posts" # Do not modify. -authors = ["郭一/小红书智能分发工程负责人"] - -tags = ["Practice"] -summary = "小红书使用高性能批量计算平台Volcano的落地实践过程" - -# Add menu entry to sidebar. -linktitle = "小红书基于Volcano的大规模离线与在线推荐模型训练实践" -[menu.posts] -parent = "tutorials" -weight = 6 -+++ - ->本文2021年5月27日首发于容器魔方微信公众号，原文链接[小红书基于Volcano的大规模离线与在线推荐模型训练实践](https://mp.weixin.qq.com/s/nZXZx78EQoHzRj1-LqMPQQ) - -## 小红书背景介绍 - -小红书是国内领先的生活分享社区，在女性用户当中比较受欢迎，现在也有越来越多的潮流男生加入了小红书社区，目前月活超过1亿。小红书是一个UGC社区，每天有大概几十万篇的笔记投稿，近百次的笔记曝光量。 - - - -首页的推荐功能是小红书的核心业务场景之一。在小红书刚创建的几年当中，推荐是完全人工精选，没有机器学习模型，也不是千人千面，每个人看到的内容都是差不多的。从2016年开始，我们进行千人千面的推荐探索。2018年上线了第一个基于SparkML和GBDT的推荐机器学习模型，大概只有几万个参数。从2018年底开始，我们加速模型的迭代，到2020年下半年，模型规模已经达到了几千亿参数的级别，我们也上线了在线学习，模型可小时级别更新，在今年4月份已达到了分钟级别，用户的行为可以在一两分钟之内被我们模型所抓获，然后产生更加适合用户的推荐，捕捉到用户更短期的兴趣爱好。 - - - -## 小红书搜索推荐广告场景下的大数据架构 - -{{
}} - - - -架构可以分为4个部分，左上角是客户端和在线服务与打点服务之间进行交互。用户在打开小红书APP之后，会请求一个在线服务，在线服务进行推荐，把当时推荐的笔记和请求的特征缓存，将推荐结果返回给客户端，用户在浏览推荐给他的笔记过程中，会产生一系列交互行为，交互行为会变成数据流，经过打点服务，落到原始打点数据流。 - - - -左下角主要是有两个任务归因和汇总，它们主要是实时的对用户行为做数据清洗和处理，生成标签数据流。标签数据流和测值数据流会做拼接，生成训练样本，同时也会生成我们大数据主要的三个产品，训练数据、OLAP数据库和离线Hive表。 - - - -右上角为在线和离线训练的部分，所谓在线训练会实时的对训练数据进行训练，产生模型的更新数据；离线训练会产生一个全量的模型更新到线上。 - - -__行为归因和标签__ - -{{
}} - - -用户行为的归因和标签任务分为两个部分：归因和标签。归因主要是将用户在打点里每个行为和过去发生的行为关联起来。比如我小红书上看了4篇笔记，做了一个点赞，每篇笔记在打点当中其实是发生在不同的地方，比如发现页上面的曝光就会打点到发现页上面，如果用户有一个点赞，这个点赞其实是发生在笔记详情页，它并不是点击进去之后发生的，那么这个点赞其实也可以在不同的渠道发生，用户可能从发现页看到了这篇笔记，点击进去以后进行了点赞，可以是搜索了一些关键词，从搜索结果里进行了这篇笔记的点赞，同样也会进入到笔记详情页面，也可能从博主的个人页发现了这篇笔记点进去。 - - - -打点本身其实并没有告诉我们在点赞之前发生了什么，在归因的过程中，我们会根据用户数据行为流，将点赞之前的用户历史浏览记录，从而判断用户点赞行为的原因，补充归因这一列，这个就是归因。另外，用户关注了一个博主，为什么关注这个博主，在关注博主之前，他看过博主的哪些笔记，其实这些也是通过归因任务将这些标签给打上去的。 - - - -标签计算和归因相反，是用户产生某个行为之后，他进行了哪些动作，是对之后发生行为的汇总。用户在发现页浏览了4篇笔记，对于每篇笔记我们都会有几个标签，浏览之后有没有发生点击或点赞，如果点击会进入笔记详情页，在页面上停留了多长时间？这些其实都是我们后续模型训练很重要的标签数据，还有平时生成的用户日常报表，也是依赖这份标签数据的。 - -__搜推广的实时大数据产品__ - -{{
}} - -产生标签数据时，大数据其实对于业务而言，主要提供了三个重要的大数据产品： - -- 模型训练数据:用来实时训练模型，给出更加精准与实时的用户最新的兴趣爱好 - -- 用来做即席的数据分析和离线仓库，这两个其实都是通过标签数据来做分析，差别是即时分析是有非常实时需求的，比如在系统中做了任何改动与策略上的变化，我是希望能立刻从多维度切分去观测这个变化起了什么作用。而离线数据仓库更多的是日报或者周报，或者过去半年某些指标产生了什么变化。 - - -__在线和离线模型训练__ - -{{
}} - -标签数据和特征数据拼接之后产生的训练数据，分别进行离线和在线训练。这里有个差别是离线训练和在线训练用的同一份数据源，拼接之后会放到kafaka,kafaka的数据提供给在线做消费，在线数据消费kafaka后，它其实输出的是一个模型的更新数据流，这里其实是最近上一批次产生的对模型造成变化被实时的发布到在线。离线其实会做批量天级别的训练，会发布一个全量的迁移参数模型，同时更回到线上。 - - - -## 离线批量到在线流式计算的演进？ - -__Offline batch only__ - - -{{
}} - -上图为最早的离线批量标签计算流程图，用户发现了点击行为后被收集下来，会落到ODS表，ODS表是原始日志的表，我们是通过Spark任务进行归因和标签计算，这完全是一个离线的批示任务。加工好之后的数据，会形成DW数仓的表，基于DW表我们产生了日报和实验报表的大数据产品。在批量环境下，这个其实是T+1生成的，每次实验的改动，一般到第二天或者第三天才能看完整的报表。 - -__Offline Batch + Online Streaming__ - -{{
}} - -随着开发人员不断变多，对业务的实施性要求也逐渐变高，我们就发现了实时的链路，实时的链路是完全基于Flink框架做的。在实时链路中，它是通过流式kafaka输入，将数据输出到kafaka，同时会落到OLAP数据库和实时实验分析。这里我们有遇到一个挑战，同样的Spark框架和Flink框架，它是两个不同的编程框架。比如要判断某一个广告用户的点击是否是有效点击，这个有效点击的逻辑会稍微复杂一点，因为在点击进去之后，要产生互动行为或至少停留三秒以上，才算有效点击。 - - - -如果有两条数据流，往往会在离线实现一个逻辑，同时也会在Flink框架里也实现这样一个逻辑,实际上是同一个逻辑实现两遍会产生很多问题，一是有两份工作量，又要在Spark里开发，又要在Flink里开发。还有一个更大的问题是两边开发之后，逻辑可能会不一样。 - - - -对于一些比较复杂的场景会存在差别，最终造成报表和申请不一致。还有一些场景，比如数仓离线提出需求，只在Spark任务里实现了这个逻辑，但在离线没有实现，如果需要查看还要再重新实现，这都会造成额外的工作量。因为我们进行升级更新，所有新的标签不在spark上做计算了，完全在实时这条线进行计算。但实时有可能会出中断的问题，在中断之后，可能会从最新的开始，老的数据可能会产生变动。但离线遇到这种问题就比较简单，将每小时数据重新运行，就会产生一份完整的数据。 - - - -我们其实要解决一个技术问题:如何让同样实时的Flink训练任务从离线的数仓表里,转成实时的流表，用同样的计算逻辑生成一份新的数据重回再写回这个表，就保证实时和离线的核心逻辑只在实时实现一份即可，解决了两次开发和我数据逻辑不一致的问题。 - -__Offline Training__ - -{{
}} - -上图为机器学习模型的训练流程。最早也是只有一个批量的数据计算，数据计算是特征数据和用户行为数据都放在离线的表里面，通过Spark任务去做拼接，拼接完之后形成训练任务，然后学习模型发布上线，整个过程可能每天进行一次。 - - -__Online + Offline Training__ - -{{
}} - -因为要更加快速的捕捉到用户实时兴趣点，同时对于有些新发布的笔记，要更快启动笔记做出判断，因此我们的模型需要做更加实时更新的需求。所以将其改造成通过Flink来做实时的模型训练，Flink生成数据之后，使用Volcano的调度系统进行实时模型和批量模型更新。 - -## 离线训练的优化和多云管理 - -{{
}} - -上图为小红书在机器学习和大数据的技术栈。小红书因为是一个比较新的公司，我们没有自建的机房，所有的服务器是买的云厂商的云服务，我们大多数业务都通过Kubernetes来管理的。 - - - -我们有两个很重要的平台：一个是内部称为Baichuan的流计算平台，主要用来管理上文提到的实时标签计算与在线学习的Flink任务。还有一个是机器学习的任务管理平台，内部称Firefly。我们是基于TensorFlow开发的一些模型训练，会运行在机器学习平台上，同时针对推荐搜索广告的大规模稀疏离散模型训练场景，也基于TensorFlow研究了自己的训练框架LarC，TensorFlow和LarC的框架模型是通过Firefly运行在机器学习平台上。 - - - -中间有非常关键的一层，如何将我们的任务调度到Kubernetes集群上，其实原生的Kubernetes对于这种场景存在一个比较大的问题，它是基于单pod来做调度的。但流计算和机器学习的任务，其实并不是单pod任务，是一组pod的job。因此我们开始有一个比较大的痛点，如果有两个job，每个job都有多个pod需要被调度，如果用原生的Kubernetes调度器，比如有10个资源，但是有两个任务，每个任务都需一个pod,但现在只有15个空闲核，每个任务比如说是10个核，加起来是不够的，但调度单个任务其实是可以。 - - - -如果用原生的 k8s去做调度，比如一个任务调度上去7个核，另外1个任务调度上去8个核，一共用了15个核，但每个任务其实不能调度完，所以每个任务都没有办法完成整个调度过程。因为它是基于单pod的，调度上去的时候不会把它撤下来，所以这个集群就会卡死。 - - - -基于这些痛点我们去做了一些调研，发现Volcano的前身是Kubernetes batch，可以完全解决我们的痛点。因此，我们也参与到Kubernetes batch社区当中，成为了Volcano的忠实用户。 - -__增强型Volcano调度：binpack->task-topology__ - -{{
}} - -原生的Volcano支持的调度是binpack的算法。在机器学习训练时，训练任务分两类：一类是worker，主要是做前向和反向的计算，是一个计算型的任务。ps主要的任务是存储参数，是一个内存型的服务。如果用原生开源版的 Volcano，它默认支持的调度算法会优化要减少碎片，所以要把尽量多的任务调到同一个node上，然后会把这个任务调成所有work和ps放在一个node上，这个ps放不下，所以只能放在另外一个node上。 - - - -这种场景下，会发生worker和某ps0在一台机器上，他们之间 io没有跨机器，就会比较快，存储量也比较大。但是它运行的速度比较慢，因为被一个慢的 ps拖累了。我们发现把任务变成一个平衡的状态，速度均衡之后，它的整体存储是有比较大提升的。从binpack到task-topology的优化，总共能提升大概10%-20%的任务训练的吞吐。 - - -__多云之间的数据流转__ - -{{
}} - -在线上的话，我们会把用户分发到不同的AZ上，推荐服务它的特征缓存会存在本地AZ,在用户打点之后，会根据用户请求将用户分发到不同的集群上，分别对不同的用户做标签系统的计算，最终我们会把所有标签系统计算又流转到为我们提供离线训练和服务的云厂商，去进行数据训练的拼接和生成，同时做分布式的模型训练，训练好的模型再被分发到不同的AZ做线上服务。 - - - -怎么在这个架构下面做迁移学习？小红书的用户会在首页做流量，这是产生沉淀大量数据的一个场景。它会训练出来一个千亿级别的模型，那如何让推荐生产出来的大模型让搜索广告都能使用？将推荐的模型训练完之后，会把我们训练的推荐的模型，同步到搜索训练的集群上，在搜索集群训练上用搜索的数据，去发现推荐的模型，最终产生搜索的模型同时被发布到线上。如此在小规模的数据训练场景下，能够拿到训练推荐大模型的特点，使得推荐的大模型能够赋能搜索这个场景。 - - - - diff --git a/content/zh/docs/.DS_Store b/content/zh/docs/.DS_Store deleted file mode 100644 index 5008ddfcf53c02e82d7eee2e57c38e5672ef89f6..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 6148 zcmeH~Jr2S!425mzP>H1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3 zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ zLs35+`xjp>T0 Volcano多集群调度仓库详见：[volcano-global](https://github.com/volcano-sh/volcano-global) - -### 重调度 -* 支持动态重调度，优化集群负载分布，提升系统稳定性 - -> Volcano重调度仓库详见：[descheduler](https://github.com/volcano-sh/descheduler) - -### 监控与可观测性 -* 完整的日志系统 -* 丰富的监控指标 -* 提供可视化的Dashboard，便于用户进行图形化界面操作 - -> Volcano Dashboard详见：[dashboard](https://github.com/volcano-sh/dashboard) -> -> Volcano指标详见: [metrics](https://github.com/volcano-sh/volcano/blob/master/docs/design/metrics.md) - -## 生态 -Volcano已经成为业界批量计算场景中的事实标准，并被广泛应用于以下高性能计算框架中： - -* [Spark](https://spark.apache.org/) -* [TensorFlow](https://tensorflow.google.cn/) -* [PyTorch](https://pytorch.org/) -* [Flink](https://flink.apache.org/) -* [Argo](https://argoproj.github.io/) -* [Ray](https://www.ray.io/) -* [MindSpore](https://www.mindspore.cn/) -* [PaddlePaddle](https://www.paddlepaddle.org.cn/) -* [OpenMPI](https://www.open-mpi.org/) -* [Horovod](https://horovod.readthedocs.io/) -* [mxnet](https://mxnet.apache.org/) -* [Kubeflow](https://www.kubeflow.org/) -* [Cromwell](https://cromwell.readthedocs.io/) - -此外，Volcano已被多个企业和组织广泛应用于AI和大数据领域。它通过强大的资源管理能力、高效的Job管理机制，以及丰富的调度策略（如Gang调度、异构设备调度、拓扑感知调度等），有效满足了分布式训练、数据分析等任务的复杂需求。同时，Volcano在提升调度性能的基础上，兼顾了任务调度的灵活性和可靠性，为企业打造高效的资源利用体系提供了有力支持。 - -## 未来展望 -Volcano将继续通过社区协作和技术创新，扩展其功能边界，成为高性能计算和云原生批量调度的引领者。 - diff --git a/content/zh/docs/actions.md b/content/zh/docs/actions.md deleted file mode 100644 index 64054c94..00000000 --- a/content/zh/docs/actions.md +++ /dev/null @@ -1,84 +0,0 @@ -+++ -title = "Actions" - -date = 2021-04-07 -lastmod = 2025-01-21 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "Actions" -[menu.docs] - parent = "scheduler" - weight = 2 -+++ - - - -### Enqueue - -#### 简介 - -Enqueue action筛选符合要求的作业进入待调度队列。当一个Job下的最小资源申请量不能得到满足时，即使为Job下的Pod执行调度动作，Pod也会因为gang约束没有达到而无法进行调度。只有当集群资源满足作业声明的最小资源需求时，Enqueue action才允许该作业入队，使得PodGroup的状态由Pending状态转换为Inqueue状态。这个状态转换是Pod创建的前提，只有PodGroup进入Inqueue状态后，vc-controller才会为该PodGroup创建Pod。这种机制确保了Pod只会在资源满足的情况下被创建，是调度器配置中必不可少的action。 - -#### 场景 - -Enqueue action是调度流程中的准备阶段，只有当集群资源满足作业调度的最小资源请求，作业状态才可由"pending"变为"enqueue"。这样在AI/MPI/HPC这样的集群资源可能不足的高负荷的场景下，Enqueue action能够防止集群下有大量不能调度的pod，提高了调度器的性能。 - -> 注意：enqueue action和preempt/reclaim action是互相冲突的，如果同时配置了enqueue action和preempt/reclaim action，且enqueue action判断作业无法入队，有可能导致无法生成Pending状态的Pod，从而无法触发preempt/reclaim action。 - - -### Allocate - -#### 简介 - -Allocate action是调度流程中的正常分配步骤，用于处理在待调度Pod列表中具有资源申请量的Pod调度，是调度过程必不可少的action。这个过程包括作业的predicate和prioritize。使用predicateFn预选，过滤掉不能分配作业的node；使用NodeOrderFn打分来找到最适合的分配节点。 - -Allocate action遵循commit机制，当一个Pod的调度请求得到满足后，最终并不一定会为该Pod执行绑定动作，这一步骤还取决于Pod所在Job的gang约束是否得到满足。只有Pod所在Job的gang约束得到满足，Pod才可以被调度，否则，Pod不能够被调度。 - -#### 场景 - -在集群混合业务场景中，Allocate的预选部分能够将特定的业务（AI、大数据、HPC、科学计算）按照所在namespace快速筛选、分类，对特定的业务进行快速、集中的调度。在Tensorflow、MPI等复杂计算场景中，单个作业中会有多个任务，Allocate action会遍历job下的多个task分配优选，为每个task找到最合适的node。 - -### Backfill - -#### 简介 - -Backfill action是调度流程中处理BestEffort Pod（即没有指定资源申请量的Pod）的调度步骤。与Allocate action类似，Backfill也会遍历所有节点寻找合适的调度位置，主要区别在于它处理的是没有明确资源申请量的Pod。 - -#### 场景 - -在集群中，除了需要明确资源申请的工作负载外，还存在一些对资源需求不明确的工作负载。这些工作负载通常以BestEffort的方式运行，Backfill action负责为这类 Pod寻找合适的调度位置。 - - -### Preempt - -#### 简介 - -Preempt action是调度流程中的抢占步骤，用于处理高优先级调度问题。Preempt用于同一个Queue中job之间的抢占，或同一Job下Task之间的抢占。 - -#### 场景 - -- Queue内job抢占：一个公司中多个部门共用一个集群，每个部门可以映射成一个Queue，不同部门之间的资源不能互相抢占，这种机制能够很好的保证部门资源的隔离性。多业务类型混合场景中，基于Queue的机制满足了一类业务对于某一类资源的集中诉求，也能够兼顾集群的弹性。例如，AI业务组成的queue对集群GPU占比90%，其余图像类处理的业务组成的queue占集群GPU10%。前者占用了集群绝大部分GPU资源但是依然有一小部分资源可以处理其余类型的业务。 -- Job内task抢占：同一Job下通常可以有多个task，例如复杂的AI应用场景中，tf-job内部需要设置一个ps和多个worker，Preempt action就支持这种场景下多个worker之间的抢占。 - -### Reclaim - -#### 简介 - -Reclaim action是调度流程中的**跨队列**资源回收步骤。与Preempt不同，Reclaim专门处理不同Queue之间的资源回收。当某个Queue中的作业需要资源且该Queue未超用时，可以从其他可回收队列中回收资源。 - -#### 场景 - -- 跨队列资源回收：在多部门共用集群的场景下，当高优先级部门（如在线业务部门）的Queue资源不足时，可以从其他可回收的部门Queue（如离线计算部门）回收资源。例如，在线业务Queue可以从离线业务Queue回收资源，但离线业务Queue之间不能互相回收资源。 - -- 资源利用率优化：通过跨队列资源回收机制，集群可以在保证高优先级业务SLA的同时，提高整体资源利用率。当高优先级Queue资源不足时，可以从低优先级Queue回收资源，确保关键业务的资源需求。 - -> 注意： -> -> 1. Reclaim在执行时会检查多个条件：目标Queue是否可回收（Reclaimable）、任务是否可被回收（Preemptable）、资源回收后是否满足作业运行需求等，从而确保资源回收的合理性。 -> 2. 要使Queue中的作业可以被其他Queue回收资源，需要在Queue的spec中将reclaimable字段设置为true。 - - diff --git a/content/zh/docs/architecture.md b/content/zh/docs/architecture.md deleted file mode 100644 index a5e7aa4f..00000000 --- a/content/zh/docs/architecture.md +++ /dev/null @@ -1,42 +0,0 @@ -+++ -title = "架构" - - -date = 2019-01-28 -lastmod = 2020-09-03 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "架构" -[menu.docs] - parent = "home" - weight = 2 -+++ - -## 架构概览 - - -{{
}} - -Volcano与Kubernetes天然兼容，并为高性能计算而生。它遵循Kubernetes的设计理念和风格。 - -{{
}} - -Volcano由scheduler、controllermanager、admission和vcctl组成: - -- Scheduler -Volcano scheduler通过一系列的action和plugin调度Job，并为它找到一个最适合的节点。与Kubernetes default-scheduler相比，Volcano与众不同的 -地方是它支持针对Job的多种调度算法。 - -- Controllermanager -Volcano controllermanager管理CRD资源的生命周期。它主要由**Queue ControllerManager**、 **PodGroupControllerManager**、 **VCJob -ControllerManager**构成。 - -- Admission -Volcano admission负责对CRD API资源进行校验。 - -- Vcctl -Volcano vcctl是Volcano的命令行客户端工具。 diff --git a/content/zh/docs/cli.md b/content/zh/docs/cli.md deleted file mode 100644 index a9645174..00000000 --- a/content/zh/docs/cli.md +++ /dev/null @@ -1,69 +0,0 @@ -+++ -title = "CLI" - - -date = 2019-01-28 -lastmod = 2020-09-04 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "命令行" -[menu.docs] - parent = "cli" - weight = 1 -+++ - -### 简介 -Volcano提供了命令行工具用于管理资源。 -## 配置 - -1. 您可以自己从 github 上克隆代码并在项目的根目录下执行以下命令制作最新的可执行文件： -```shell -# make vcctl -``` -2. 将可执行文件拷贝到$PATH下以便您能在任何地方执行它。 - -## 命令行列表 -### 列举所有的Job - -```shell -# vcctl job list -Name Creation Phase JobType Replicas Min Pending Running Succeeded Failed Unknown RetryCount -job-1 2020-09-01 Running Batch 1 1 0 1 0 0 0 0 -``` - -### 删除指定的Job - -```shell -# vcctl delete job --name job-1 --namespaces default -delete job job-1 successfully -``` - -### 中止一个Job - -```html -# vcctl job suspend --name job-1 --namespace default -``` - -### 消费一个Job (与"vcctl job suspend"相反) - -```html -# vcctl job resume --name job-1 --namespace default -``` - -### 运行一个Job - -```shell -# vcctl job run --name job-1 --namespace default -``` - -## 说明事项 -如需获取更多命令行详情请按如下操作: - -```html -# vcctl -h -# vcctl [command] -h -``` diff --git a/content/zh/docs/colocation.md b/content/zh/docs/colocation.md deleted file mode 100644 index d6ce6abf..00000000 --- a/content/zh/docs/colocation.md +++ /dev/null @@ -1,624 +0,0 @@ -+++ -title = "云原生混部" - -date = 2025-01-20 -lastmod = 2025-01-20 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -toc_depth = 5 -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -[menu.docs] - parent = "features" - weight = 3 -+++ - -## 背景 - -随着云原生技术的快速发展，越来越多的业务已逐渐迁移到Kubernetes，使用云原生化的方式进行开发维护，极大地简化了应用程序的部署、编排和运维，Kubernetes已逐渐成为云原生时代的“操作系统”。但在另一方面，应用云原生技术之后，数据中心的资源使用率仍然较低，为了提升资源利用率同时保障高优先级业务的SLO，Volcano推出了云原生混部解决方案，从应用层到内核提供端到端的资源隔离与共享机制，最大化提升资源利用率。 - -云原生混部是指通过云原生的方式将在线业务和离线业务部署在同一个集群。由于在线业务运行具有明显的波峰波谷特征，因此当在线业务运行在波谷时，离线业务可以利用这部分空闲的资源，当在线业务到达波峰时，通过在线作业优先级控制等手段压制离线作业的运行，保障在线作业的资源使用，从而提升集群的整体资源利用率，同时保障在线业务SLO。 - -典型的在线、离线业务具有如下特征: - -| | **在线业务** | **离线业务** | -| -------- | -------------------------- | -------------------------------- | -| 典型应用 | 微服务、搜索、推荐、广告等 | 大数据批处理、AI训练、视频转码等 | -| 时延 | 敏感 | 不敏感 | -| SLO | 高 | 低 | -| 负载模型 | 分时性 | 持续占用资源 | -| 错误容忍 | 容忍度低，对可用性要求高 | 允许失败重试 | -| 运行时间 | 稳定持续运行 | 任务型、运行时间短 | - -## 优势 - -业界有较多的公司和用户对在离线混部技术进行了不同程度的探索与实践，为在离线混部提供了积极有益的设计与实践，但也存在一些不足之处，比如不能做到和Kubernetes完全解耦，超卖资源计算方式粗糙，在离线作业使用方式不一致、用户体验不友好等问题。 - -基于以上考虑，Volcano针对在离线混部技术进行了进一步的增强与优化，对比业界的在离线混部我们具有如下的优势： - -- Volcano Scheduler天然支持离线作业调度与管理。 -- 对Kubernetes无侵入式修改。 -- 超卖资源实时动态计算，更好地平衡资源利用与与业务QoS需求。 -- OS层面的隔离与QoS保障。 - -## 架构 - -整个云原生混部架构主要包括Volcano Scheduler、Volcano SLO Agent、Enhanced OS几部分。 - -- Volcano Scheduler：负责在离线作业的统一调度，提供队列、组、作业优先级、公平调度、资源预留等多种抽象，统一满足微服务、大数据、AI等业务调度需求。 -- Volcano SLO Agent：集群内的每个节点都会部署一个Volcano SLO Agent，动态实时计算每个节点已经分配但未使用的资源，将这部分资源进行超卖，供离线作业进行使用。同时对节点QoS进行保障，在检测到节点出现CPU/Memory压力时，对离线作业进行驱逐，保障在线业务的优先级。 -- Enhanced OS：Volcano SLO Agent在应用层进行节点级别的QoS保证，为了进行更加精细化和强制性的隔离，内核层面也需要区分QoS类型，在CPU/Memory/Network/L3 cache等层面进行隔离，内核暴露了一系列的cgroup接口，Volcano SLO Agent可以为在线和离线业务设置不同的cgroup，做到内核层面的精细隔离，实现在线作业对离线作业的压制。 - -
{{
}} -架构 -
- -## 功能 - -### 基于QoS的混部模型 - -将在线和离线作业混合部署在一个集群之后，由于离线作业通常是CPU或者IO密集型任务，因此会对在线作业造成干扰，导致在线业务QoS受损，为了尽可能降低离线业务对在线业务的干扰，需要对在线和离线业务进行QoS分级管控，通过对在线和离线作业进行标识来定义QoS模型，进而在运行态优先保障在线业务QoS，降低离线作业对在线的干扰。 - -根据在线和离线作业的分类和运行特点，Volcano对在线和离线作业做了模型抽象，定义了不同的QoS等级，不同类型的业务可以设置不同的QoS等级，同时在内核层面映射到CPU和Memory等级，更高的等级将会获得更高的资源使用权和抢占优先级。在调度时会区分不同QoS等级对应的作业类型，执行丰富的调度策略，同时通过Volcano SLO Agent调用内核接口为在离线作业设置不同的QoS优先级。QoS模型定义如下： - -| Qos等级 | 典型应用场景 | CPU优先级 | Memory优先级 | -| :---------------------------: | :---------------------------------: | :-------: | :----------: | -| LC(Latency Critical) | 时延敏感极高的核心在线业务，独占CPU | 独占 | 0 | -| HLS(Highly Latency Sensitive) | 时延敏感极高的在线业务 | 2 | 0 | -| LS(Latency Sensitive) | 时延敏感型的近线业务 | 1 | 0 | -| BE(Best Effort) | 离线的AI、大数据业务，可容忍驱逐 | -1 | -1 | - -用户可以通过设置作业对应Pod的annotation来表示不同的作业类型，比如设置volcano.sh/qos-level="LS"，表示Pod为时延敏感型的近线作业，设置volcano.sh/qos-level="BE"，表示Pod为离线作业。 - -### 在离线作业统一调度 - -将在线和离线作业同时部署在一个集群时，使用多个调度器分别调度不同类型的作业时，每个调度器都可以看到全局的资源视图，多个调度器在进行节点资源计算和绑定时，就极有可能存在并发资源更新冲突，为了避免这一问题，需要用统一的调度器调度在线和离线作业。 - -Volcano作为业界首个云原生批量计算项目，天然支持AI、Big Data等作业的调度和管理，并且支持多租户的队列管理和公平调度，统一支持几乎所有主流的计算框架，包括Ray 、Kubeflow、Spark、Flink、Pytorch、Tensorflow、MPI、Horovod、MindSpore、PaddlePaddle、MXNet、Argo等。并且集成了K8s默认的调度算法，支持批处理作业和微服务的统一调度，根据作业的QoS模型进行优先级调度。因此支持在线和离线作业的统一调度。 - -### 动态资源超卖 - -Kubernetes现有的资源调度模型基于Pod的requests进行计算，而用户在设置资源requests时往往具有盲目性，requests值设置较大而实际使用量很少，导致资源浪费。同时对于在线作业而言，其运行规律具有明显的波峰波谷特征，因此在业务运行低谷时期，非常适合将未充分使用的资源进行二次超卖，给离线作业使用，从而提升集群的资源利用率。 - -Volcano SLO Agent实时计算Pod已经申请但未使用的资源，将这部分资源动态超卖给离线作业使用，提高Pod部署密度，提升资源利用率。 - -
{{
}} -动态资源超卖原理图 -
- -由于超卖资源的增加，改变了节点原有的资源可用量，且超卖资源单独给离线作业使用，因此对于超卖资源的计算、上报方式和使用方式会有不同的方案选择。为了更好地与Kubernetes解耦，以及支持用户自定义设置超卖资源的表现形式，Volcano提供了native、extend等超卖资源计算和上报模式，native的模式会上报超卖资源至节点的allocatable字段，这样一来在线和离线作业的使用方式是一致的，提升了用户体验 ，而extend模式支持将超卖资源以扩展方式上报至节点，做到和Kubernetes的解耦，用户可以根据实际需求灵活选择超卖资源的上报和使用方式。 - -### QoS保障 - -将在线和离线作业混合部署后，由于离线作业和在线作业会发生资源争用，离线作业会对在线业务造成干扰，因此在资源利用率提升的同时还需要保障在线作业的QoS，避免离线业务对在线业务的干扰。 - -在离线作业通常会使用多种不同维度的资源，因此需要对各个维度的资源设置资源隔离措施，Volcano会通过内核态接口设置CPU、Memory、Network等维度的资源隔离，当在离线作业发生资源争用时，压制离线作业的资源使用，优先保障在线作业QoS。 - -- **CPU:** OS层面提供了5级CPU QoS等级，数值从-2到2，QoS等级越高则代表可以获得更多的CPU时间片并有更高的抢占优先级。通过设置cpu子系统的cgroup cpu.qos_level可以为不同业务设置不用的CPU QoS。 - -- **Memory:** Memory隔离体现在系统发生OOM时离线作业会被有限OOM Kill掉，通过设置memory子系统的cgroup memory.qos_level可以为不同业务设置不同的Memory QoS。 - -- **Network:** 网络隔离实现了对在线作业的出口网络带宽保障，它基于整机的带宽大小，并通过cgroup + tc + ebpf技术，实现在线作业对离线作业的出口网络带宽压制。 - -
{{
}} -网络隔离技术方案 -
- -上图为网络隔离的技术方案，通过ebpf将限速程序注入到kernel，实现对报文转发的控制，从而达到限速的目的。cgroup ebpf可以为在离线业务的的报文设置不同的标签以区分在线和离线业务流量，tc ebpf可以设置三个水位线：在线业务水位线，离线业务高水位线和离线业务低水位线。当在线业务流量超过水位线时，限制离线作水位带宽，离线业务带宽使用上限为离线业务低水位线，避让在线流量；当在线业务流量低于水位线时，放开对离线作业的带宽限制，离线业务带宽使用上限为离线业务高水位线，提高资源利用率，同时可以根据离线流量带宽计算报文的发送时间（EDT），实现离线作业流量限速。 - -
{{
}} -在离线作业带宽限制示意图 -
- -### CPU Burst - -若Pod中容器设置了CPU Limit值，则该容器CPU使用将会被限制在Limit值以内，形成对CPU的限流。频繁的CPU限流会影响业务性能，增大业务长尾响应时延，对于时延敏感型业务的影响尤为明显。 - -Volcano agent的CPU Burst能力提供了一种可以短暂突破CPU Limit值的弹性限流机制，以降低业务长尾响应时间。其原理是业务在每个CPU调度周期内使用的CPU配额有剩余时，系统对这些CPU配额进行累计，在后续的调度周期内如果需要突破CPU Limit时，使用之前累计的CPU配额，以达到突破CPU Limit的效果。 - -当未开启CPU Burst时，容器可以使用的CPU配额会被限制在Limit以内，无法实现Burst。如下图所示： - -
{{
}} -
- -开启CPU Burst后，容器使用的CPU配额可以突破Limit限制，实现Burst。如下图所示： - -
{{
}} -
- -通过Volcano agent提供的CPU Burst能力，可以避免高优业务在关键时刻被限流，保障时延敏感型业务的稳定性。 - -## 使用指导 - -### 安装Volcano agent - -#### 通过 Helm 安装 - -```shell -helm repo add volcano-sh https://volcano-sh.github.io/helm-charts - -helm repo update - -helm install volcano volcano-sh/volcano -n volcano-system --create-namespace --set custom.colocation_enable=true -``` - -#### 通过 Yaml 安装 - -请按照该[文档](https://github.com/volcano-sh/volcano?tab=readme-ov-file#quick-start-guide)安装Volcano，然后通过以下命令安装Volcano agent。 - -```shell -kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-agent-development.yaml -``` - -并检查Volcano所有组件已成功运行。 - -```shell -kubectl get po -n volcano-system -NAME READY STATUS RESTARTS AGE -volcano-admission-76bd985b56-fnpjg 1/1 Running 0 3d -volcano-admission-init-wmxc7 0/1 Completed 0 3d -volcano-agent-q85jn 1/1 Running 0 3d -volcano-controllers-7655bb499f-gpg9l 1/1 Running 0 3d -volcano-scheduler-6bf4759c45-c666z 1/1 Running 0 3d -``` - -通过设置标签的方式，在节点级别打开混部和超卖开关。 - -```shell -kubectl label node $node volcano.sh/oversubscription=true # replace $node with real node name in your kubernetes cluster. - -kubectl label node $node volcano.sh/colocation=true # replace $node with real node name in your kubernetes cluster. -``` - -### CPU Burst示例 - -该示例将演示如何使用 CPU Burst 及它带来的收益。 - -#### 开启CPU Burst - -部署一个Deploment并暴露类型为 Cluster IP 的service，Pod 的注解 `volcano.sh/enable-quota-burst: "true"` 表示为 Pod 开启CPU Burst功能。 - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: nginx - namespace: default -spec: - replicas: 1 - selector: - matchLabels: - app: nginx - template: - metadata: - labels: - app: nginx - annotations: - volcano.sh/enable-quota-burst: "true" # pod enabled cpu burst - spec: - containers: - - name: container-1 - image: nginx:latest - resources: - limits: - cpu: "2" - requests: - cpu: "1" ---- -apiVersion: v1 -kind: Service -metadata: - name: nginx - namespace: default - labels: - app: nginx -spec: - selector: - app: nginx - ports: - - name: http - targetPort: 80 - port: 80 - protocol: TCP - type: ClusterIP -``` - -#### 进行压力测试 - -使用`stress`工具为nginx Pod加压。 - -```bash -wrk -H "Accept-Encoding: deflate, gzip" -t 2 -c 8 -d 120 --latency --timeout 2s http://$(kubectl get svc nginx -o jsonpath='{.spec.clusterIP}') -``` - -#### 检查CPU限流情况 - -检查 Pod 容器的 CPU 限流状态，我们可以看到 `nr_bursts` 和 `burst_time` 不为 0，而 `nr_throttled` 和 `throttled_time` 是一个较小的值，这表明 Pod 已经使用了突发的 CPU 配额。 - -```bash -cat /sys/fs/cgroup/cpu/kubepods/burstable/podd2988e14-83bc-4d3d-931a-59f8a3174396/cpu.stat # replace nginx pod uid in your kubernetes cluster. -nr_periods 1210 -nr_throttled 9 -throttled_time 193613865 -nr_bursts 448 -burst_time 6543701690 -``` - -如果我们设置 Pod 的注解 `volcano.sh/enable-quota-burst=false`（禁用 Pod 的 CPU Burst）并进行另一次压力测试，`nr_throttled` 和 `throttled_time` 将会是一个相对较大的值，这表明 Pod 的 CPU 被严格限制；而 `nr_bursts` 和 `burst_time` 为 0，表明 Pod 的 CPU Burst 没有发生。 - -```bash -cat /sys/fs/cgroup/cpu/kubepods/burstable/podeeb542c6-b667-4da4-9ac9-86ced4e93fbb/cpu.stat #replace nginx pod uid in your kubernetes cluster. -nr_periods 1210 -nr_throttled 488 -throttled_time 10125826283 -nr_bursts 0 -burst_time 0 -``` - -#### 说明 - -CPU Burst 依赖于 Linux 内核提供的功能，该特性仅在主机使用的 Linux 内核版本 >= 5.14 以及某些 Linux 发行版（如 OpenEuler 22.03 SP2 或更高版本）上生效。 - -### 动态资源超卖示例 - -本示例将演示节点上的动态资源能力，并展示节点在面临资源压力时的压制和驱逐机制。节点的配置为 8 核 CPU 和 16GB 内存。 - -#### 检查节点超卖资源 - -节点的超卖资源是通过节点的可分配资源（`Allocatable`）减去实际资源使用量得到的。超卖资源包括 CPU 和内存，分别由 `kubernetes.io/batch-cpu` 和 `kubernetes.io/batch-memory` 表示，并作为扩展资源上报到节点的 `Allocatable` 字段中。在线任务使用原生资源（`cpu` 和 `memory`），而离线任务使用超卖资源（`kubernetes.io/batch-cpu` 和 `kubernetes.io/batch-memory`），这样可以提高 Pod 的部署密度和资源利用率。 - -```bash -kubectl describe node $node # 将$node替换为群集中的真实节点 -Allocatable: - cpu: 8 - ephemeral-storage: 33042054704 - hugepages-1Gi: 0 - hugepages-2Mi: 0 - kubernetes.io/batch-cpu: 7937 # CPU超卖资源，单位为毫核cpu（1核CPU=1000毫核CPU） - kubernetes.io/batch-memory: 14327175770 # emory超卖资源, 单位为字节 - memory: 15754924Ki - pods: 110 -``` - -#### 部署在线和离线作业 - -在线作业通过设置注解 `volcano.sh/qos-level: "LC"`、`volcano.sh/qos-level: "HLS"` 或 `volcano.sh/qos-level: "LS"` 来标识。离线作业通过设置注解 `volcano.sh/qos-level: "BE"` 来标识，它只能使用超分配资源（`kubernetes.io/batch-cpu` 和 `kubernetes.io/batch-memory`）。我们使用一个包含 `stress` 工具的镜像来模拟在线作业的业务压力上升。如果你无法访问该镜像，也可以替换为其他包含 `stress` 工具的镜像。 - -```yaml -# 在线作业 -apiVersion: apps/v1 -kind: Deployment -metadata: - name: online-demo - namespace: default -spec: - replicas: 1 - selector: - matchLabels: - app: online-demo - template: - metadata: - labels: - app: online-demo - annotations: - volcano.sh/qos-level: "HLS" # 标识在线作业 - spec: - containers: - - name: container-1 - image: polinux/stress - imagePullPolicy: IfNotPresent - command: ["stress", "--cpu", "7"] # 执行stress压测 - resources: - requests: - cpu: 2 ---- -# 离线作业 -apiVersion: apps/v1 -kind: Deployment -metadata: - name: offline-demo - namespace: default -spec: - replicas: 1 - selector: - matchLabels: - app: offline-demo - template: - metadata: - labels: - app: offline-demo - annotations: - volcano.sh/qos-level: "BE" # 标识离线作业 - spec: - containers: - - name: container-1 - image: nginx:latest - resources: - requests: - kubernetes.io/batch-cpu: 4000 # 4核CPU - kubernetes.io/batch-memory: 10737418240 # 10Gi内存 -``` - -#### 确保在线作业和离线作业成功运行 - -```bash -kubectl get po -NAME READY STATUS RESTARTS AGE -offline-demo-f59758bb-vlbp7 1/1 Running 0 6s -online-demo-9f9bbdb58-fljzs 1/1 Running 0 6s -``` - -#### 节点压力下的驱逐机制 - -当节点出现压力，资源资源利用率达到设定的阈值时，会触发驱逐机制。在线作业的 QoS（服务质量）由 **Volcano Agent** 和 **主机操作系统** 共同保障。Volcano Agent 会实时检测节点的资源利用率，当节点资源利用率超过阈值时，会驱逐离线作业。对于 CPU 资源，默认阈值是 **80%**。我们通过为在线作业施加 **7 核 CPU 压力** 来模拟资源压力，大约 1 分钟后，可以通过事件日志观察到离线作业被驱逐。 - -```bash -kubectl get event | grep Evicted -69s Warning Evicted pod/offline-demo-785cff7f58-gwqwc Evict offline pod due to cpu resource pressure -``` - -当节点资源压力上升时，我们可以观察到超卖资源（`kubernetes.io/batch-cpu` 和 `kubernetes.io/batch-memory`）会减少。这是因为超卖资源是通过节点的可分配资源减去实际资源使用量计算得出的。当在线或者离线作业的资源使用量增加时，节点的可用资源减少，从而导致超卖资源也随之减少。 - -```bash -kubectl describe node $node # 将$node替换为群集中的真实节点 -Allocatable: - cpu: 8 - ephemeral-storage: 33042054704 - hugepages-1Gi: 0 - hugepages-2Mi: 0 - kubernetes.io/batch-cpu: 978 # CPU超卖资源减少 - kubernetes.io/batch-memory: 14310391443 - memory: 15754924Ki - pods: 110 -``` - -当驱逐发生时，**Volcano Agent** 会为当前节点添加一个驱逐污点（Eviction Taint），以避免新的工作负载继续调度到该节点，从而避免给已经处于压力下的节点增加额外的负担。我们可以观察到，新创建的离线作业 Pod 会因为该驱逐污点而处于 `Pending` 状态。 - -```bash -kubectl get po -NAME READY STATUS RESTARTS AGE -offline-demo-f59758bb-kwb54 0/1 Pending 0 58s -online-demo-9f9bbdb58-54fnx 1/1 Running 0 2m1s - -kubectl describe po offline-demo-f59758bb-kwb54 -Events: - Type Reason Age From Message - ---- ------ ---- ---- ------- - Warning FailedScheduling 69s default-scheduler 0/1 nodes are available: 1 node(s) had taint {volcano.sh/offline-job-evicting: }, that the pod didn't tolerate. -``` - -如果停止在线作业以释放节点的资源压力，**Volcano Agent** 会检测到节点资源利用率下降，并自动移除驱逐污点（Eviction Taint）。一旦污点被移除，新的 Pod 就可以正常调度到该节点上。 - -#### 说明 - -**Volcano Agent** 为在线作业和离线作业定义了一个 QoS（服务质量）资源模型，并为在线作业提供了应用级别的保障机制（例如在节点资源压力下驱逐离线作业）。同时，CPU 和内存的隔离与抑制由**主机内核**（Host Kernel）在操作系统级别提供保障。需要注意的是，目前 Volcano Agent 仅适配 **openEuler 22.03 SP2** 及更高版本，因此使用该功能时请确保使用正确的操作系统类型和版本。 - -### 出口网络带宽保障示例 - -在出口网络带宽隔离机制中，离线作业的带宽使用会受到限制，尤其是在在线作业需要更多带宽时。为了实现更精细的带宽控制，通常会定义三个水位线参数（Watermark Parameters），用于动态调整离线作业的带宽分配。 - - - - - - - - - - - - - - - - - - - - - - - - - - -
水位线说明默认值
onlineBandwidthWatermarkPercent在线带宽水位线值与节点基础带宽的比值：
onlineBandwidthWatermark 值 = 节点基础带宽 * onlineBandwidthWatermarkPercent / 100		   80   
offlineHighBandwidthPercent离线高带宽水位线值与节点基础带宽的比值：
offlineHighBandwidth 值 = 节点基础带宽 * offlineHighBandwidthPercent / 100
它表示当在线工作负载的带宽使用比例低于 onlineBandwidthWatermarkPercent 时，离线工作负载可使用的带宽上限。
例如：节点基础带宽 = 100Mbps，onlineBandwidthWatermarkPercent = 80，offlineHighBandwidthPercent = 40，当在线工作负载使用的带宽小于 100Mbps * 0.8 = 80Mbps 时，离线工作负载最多可以使用 100Mbps * 0.4 = 40Mbps 的带宽。		   40   
offlineLowBandwidthPercent离线低带宽水位线值与节点基础带宽的比值：
offlineLowBandwidth 值 = 节点基础带宽 * offlineLowBandwidthPercent / 100
它表示当在线工作负载的带宽使用比例高于 onlineBandwidthWatermarkPercent 时，离线工作负载可使用的带宽上限。
例如：节点带宽 = 100Mbps，onlineBandwidthWatermarkPercent = 80，offlineLowBandwidthPercent = 10，当在线工作负载使用的带宽大于 100Mbps * 0.8 = 80Mbps 时，离线工作负载最多可以使用 100Mbps * 0.1 = 10Mbps 的带宽。		   10   
- - - - -#### 设置节点网络带宽 - -本示例将演示在线作业如何抑制离线作业的整个网络带宽。我们将使用 `iperf` 工具来模拟在线作业和离线作业的网络入口带宽流量。 - -在所有节点上添加注解 `volcano.sh/network-bandwidth-rate`，以指定网络带宽速率。示例中设置的值为 `1000Mbps`，请根据实际环境设置合适的值，并替换 `$node` 为实际节点名称。 - -```bash -kubectl annotate node $node_name volcano.sh/network-bandwidth-rate=1000 -``` - -#### 部署在线和离线作业 - -部署一个在线作业和离线作业deployment，请将 `$node_ip` 替换为在你的环境中Pod可以访问到的节点 IP。同时，请在 `$node_ip` 节点上使用命令 `iperf -s` 启动 `iperf` 服务器，以确保 Pod 可以访问 `iperf` 服务器。 - -```yaml -# 在线作业 -apiVersion: apps/v1 -kind: Deployment -metadata: - name: online-iperf - namespace: default -spec: - replicas: 1 - selector: - matchLabels: - app: online-iperf - template: - metadata: - labels: - app: online-iperf - annotations: - volcano.sh/qos-level: "HLS" # 标识在线作业 - spec: - containers: - - name: container-1 - image: volcanosh/iperf - command: - - /bin/sh - - -c - - | - iperf -c $node_ip -i 1 -t 30 -f mb # 模拟消耗带宽 - echo finished... - sleep 1000000 ---- -# 离线作业 -apiVersion: apps/v1 -kind: Deployment -metadata: - name: offline-iperf - namespace: default -spec: - replicas: 1 - selector: - matchLabels: - app: offline-iperf - template: - metadata: - labels: - app: offline-iperf - annotations: - volcano.sh/qos-level: "BE" # 标识离线作业 - spec: - containers: - - name: container-1 - image: volcanosh/iperf - command: - - /bin/sh - - -c - - | - iperf -c $node_ip -i 1 -t 30 -f mb # 模拟消耗带宽 - echo finished... - sleep 1000000 -``` - -#### 查看日志 - -查看在线和离线作业日志: - -在线作业日志: - -```bash -Connecting to host 192.168.2.30, port 5201 -[ 5] local 192.168.2.115 port 58492 connected to 192.168.2.30 port 5201 -[ ID] Interval Transfer Bandwidth -[ 5] 0.00-1.00 sec 118 MBytes 990 Mbits/sec -[ 5] 1.00-2.00 sec 106 MBytes 889 Mbits/sec -[ 5] 2.00-3.00 sec 107 MBytes 897 Mbits/sec -[ 5] 3.00-4.00 sec 107 MBytes 903 Mbits/sec -[ 5] 4.00-5.00 sec 107 MBytes 899 Mbits/sec -[ 5] 5.00-6.00 sec 107 MBytes 902 Mbits/sec -[ 5] 6.00-7.00 sec 705 MBytes 884 Mbits/sec -... -``` - -离线作业日志: - -```bash -Connecting to host 192.168.2.30, port 5201 -[ 5] local 192.168.2.115 port 44362 connected to 192.168.2.30 port 5201 -[ ID] Interval Transfer Bandwidth -[ 5] 0.00-1.00 sec 8 MBytes 70 Mbits/sec -[ 5] 1.00-2.00 sec 12 MBytes 102 Mbits/sec -[ 5] 2.00-3.00 sec 11 MBytes 98 Mbits/sec -[ 5] 3.00-4.00 sec 11 MBytes 99 Mbits/sec -[ 5] 4.00-5.00 sec 11 MBytes 99 Mbits/sec -[ 5] 5.00-6.00 sec 11 MBytes 97 Mbits/sec -[ 5] 6.00-7.00 sec 11 MBytes 98 Mbits/sec -... -``` - -可以看到，当在线作业使用的带宽资源超过整个节点的 `onlineBandwidthWatermarkPercent`即默认值80时，离线作业只能使用大约 10% 的带宽，表明在线作业使用超过水位线的网络带宽时，离线作业的网络带宽使用被压制在较低值。 - -### 高级设置 - -#### 功能开关 - -混部（Colocation）功能在节点上有一个统一的开关。如果节点具有标签 `volcano.sh/oversubscription=true` 或 `volcano.sh/colocation=true`，则表示混部功能已启用。你可以移除这两个标签以禁用所有混部功能。当节点具有这些标签时，所有混部功能才会生效。 - -- 如果你只想使用在线作业和离线作业的混部功能，而不启用资源超卖（Resource Oversubscription），只需设置节点标签 `volcano.sh/colocation="true"`。 -- 如果你想同时使用混部功能和资源超卖功能，则应设置节点标签 `volcano.sh/oversubscription=true`。 - -默认情况下，`volcano-system` 命名空间中的 `volcano-agent-configuration` ConfigMap 保存了 Volcano Agent 的所有配置。 - -每个混部功能（CPU Burst / 动态资源超卖 / 出口网络带宽保障）都有一个独立的开关。你可以通过修改 `volcano-system` 命名空间中的 `volcano-agent-configuration` ConfigMap 来启用或禁用每个功能。 - -- `enable` 字段的值为 `true` 表示启用 CPU Burst 功能，`false` 表示禁用该功能。 - - ```json - "cpuBurstConfig":{ - "enable": true - } - ``` - -- `enable` 字段的值为 `true` 表示启用动态资源超卖功能，`false` 表示禁用该功能。 - - ```json - "overSubscriptionConfig":{ - "enable": true, - } - ``` - - - -- `enable` 字段的值为 `true` 表示启用出口网络带宽保障功能，`false` 表示禁用该功能。 - - ```json - "networkQosConfig":{ - "enable": true, - } - ``` - -#### CPU Burst - -启用了 CPU Burst 功能的 Pod 中的容器，其 CPU 使用量最多可以突增到容器的 CPU 限制值（`cpu limit`）。如果多个 Pod 同时使用突增的 CPU 资源，可能会发生 CPU 争用，从而影响 CPU 的 CFS（完全公平调度器）调度。 - -你可以通过设置 Pod 的注解 `volcano.sh/quota-burst-time` 来指定自定义的突增配额。例如： - -如果一个容器的 CPU 限制为 4 核，Volcano Agent 默认会将容器的 CGroup `cpu.cfs_quota_us` 值设置为 `400000`（CFS 的基本周期为 `100000`，因此 4 核 CPU 对应 `4 * 100000 = 400000`）。这意味着容器在某一时刻最多可以额外使用 4 核 CPU。如果你设置 `volcano.sh/quota-burst-time=200000`，则表示容器在某一时刻最多只能额外使用 2 核 CPU。 - -#### 动态资源超卖 - -默认情况下，超卖资源的计算和离线工作负载的驱逐仅考虑节点上 Pod 的资源使用情况。如果您希望考虑节点本身的资源利用率，应设置 volcano agent 的 `--include-system-usage=true` 标志。 - -为了避免对节点造成过大压力，volcano agent 设置了一个超卖比例来确定空闲资源的超卖比例。您可以通过设置 `--oversubscription-ratio` 标志来更改此参数，默认值为 60，表示 60% 的空闲资源将被超卖。如果您设置 `--oversubscription-ratio=100`，则表示所有空闲资源都将被超卖。 - -当节点有压力时，volcano agent 会驱逐离线工作负载。驱逐阈值可以通过 configMap `volcano-agent-configuration` 进行配置。`"evictingCPUHighWatermark":80` 表示当节点的 CPU 利用率在一段时间内超过 80% 时，将触发离线作业驱逐，并且在驱逐期间当前节点无法调度新的 Pod。`"evictingCPULowWatermark":30` 表示当节点的 CPU 利用率低于 30% 时，节点将恢复调度。`evictingMemoryHighWatermark` 和 `evictingMemoryLowWatermark` 的含义相同，但针对内存资源。 - -```json -"evictingConfig":{ - "evictingCPUHighWatermark": 80, - "evictingMemoryHighWatermark": 60, - "evictingCPULowWatermark": 30, - "evictingMemoryLowWatermark": 30 -} -``` - -#### 出口网络带宽保障 - -你可以通过修改 configMap `volcano-agent-configuration` 来调整在线和离线带宽的水位线（watermark）。`qosCheckInterval` 表示 volcano agent 监控带宽水位线的时间间隔，请谨慎修改此值。 - -```json -"networkQosConfig":{ - "enable": true, - "onlineBandwidthWatermarkPercent": 80, - "offlineHighBandwidthPercent":40, - "offlineLowBandwidthPercent": 10, - "qosCheckInterval": 10000000 - } -``` - -#### 自定义开发超卖策略 - -Volcano agent默认使用[extend]([volcano/pkg/agent/oversubscription/policy at master · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/tree/master/pkg/agent/oversubscription/policy))的方式上报和使用超卖资源，也就是将超卖资源作为一种扩展资源类型上报到节点，如果想自定义超卖资源的上报和使用，如上报为原生的cpu和memory资源，以及暂停和恢复调度的行为，请实现[policy Interface](https://github.com/volcano-sh/volcano/blob/4dea29b334877058786615ac1ed79143601dc600/pkg/agent/oversubscription/policy/policy.go#L48)实现自定义的超卖策略开发，并设置Volcano agent的启动参数`oversubscription-policy`为相应的策略。 diff --git a/content/zh/docs/contribution.md b/content/zh/docs/contribution.md deleted file mode 100644 index 4405725d..00000000 --- a/content/zh/docs/contribution.md +++ /dev/null @@ -1,166 +0,0 @@ -+++ -title = "贡献" - - -date = 2019-01-28 -lastmod = 2020-09-04 - -draft = false # Is this a draft? true/false -toc = true # Show table of contents? true/false -type = "docs" # Do not modify. - -# Add menu entry to sidebar. -linktitle = "向Volcano做贡献" -[menu.docs] - parent = "contribution" - weight = 1 -+++ - -## 欢迎 - -欢迎来到 Volcano! - -- [上手必读](#上手必读) - - [代码管理](#代码管理) - - [参与讨论](#参与讨论) - - [社区要求](#社区要求) -- [开始](#开始) -- [您的第一个贡献](#您的第一个贡献) - - [寻找您感兴趣的领域开展工作](#寻找您感兴趣的领域开展工作) - - [寻找一个话题](#寻找一个话题) - - [围绕一个 Issue 开展工作](#围绕一个-issue-开展工作) - - [提交 Issue](#提交-issue) -- [贡献者工作流程](#贡献者工作流程) - - [创建 Pull Requests](#创建-pull-requests) - - [代码检视](#代码检视) - - [commit信息的格式](#commit-信息的格式) - - [测试](#测试) - -## 上手必读 - -### 代码管理 - -请务必关注和阅读我们的[代码管理](https://github.com/volcano-sh/website/blob/master/CODE_OF_CONDUCT.md) - -### 参与讨论 - -为了更好的和volcano社区的开发者们进行交流讨论，欢迎通过如下方式订阅volcano频道。 - -- 注册账号并访问 `slack.cncf.io/` 加入cncf工作区 -- 通过添加频道搜索`volcano`，参与社区讨论。 - -### 社区要求 - -Volcano 是一个社区驱动的开源项目，致力于打造健康、友好和富有成效的环境。 -社区的目标是开发 Volcano 系统，该系统有助于在 Kubernetes 上运行高性能的工作负载，如 AI、ML、深度学习应用程序。要建立这样一个系统，需要有共同目标的社 -区贡献者的支持。 - -- 请查看 [社区角色](https://github.com/volcano-sh/volcano/blob/master/community-membership.md) 了解社区角色的具体情况。随着贡献度的提高，您的角色会逐渐升级。 - -## 开始 - -- 查看[安装文档](../installation) 了解更多关于编译和部署的细节。 - -## 您的第一个贡献 - -我们将会帮助您在不同的领域做出贡献，如处理 issue、开发特性、修复关键 bug、检视您的代码并合入。如果您对开发流程还有疑问，请查看[Slack Channel](https://cloud-native.slack.com/archives/C011GJDQS0N)( 注册[点击这里](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk))， -也可以加入我们的[mailing list](https://groups.google.com/forum/#!forum/volcano-sh) 。 - -### 寻找您感兴趣的领域开展工作 - -我们会一直需要您的帮助，如文档更新、上报 bug 或是编写代码。 -试试寻找那些没有遵循最佳编码实践、需要代码重构或是缺少测试用例的地方吧。 -下面的样例指导您如何开始。 - -#### 寻找一个话题 - -在 Volcano 组织下有[多个仓库](https://github.com/volcano-sh/)。 -每个仓库都有针对新人友好的第一个 issue 实践。 -举个例子，[Volcano-Issues](https://github.com/volcano-sh/volcano) 中具有 [help wanted](https://github.com/volcano-sh/volcano/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) -和 [good first issue](https://github.com/volcano-sh/volcano/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) -标签的 issue 一般不需要您对 Volcano 有非常多的了解即可开始工作。 -我们会帮助新的贡献者针对这类 issue 开展工作。 -另一个开始贡献的好办法是寻找一个待完善的文档，比如一个缺少超链接或超链接不可用的文档。请查看[Contributing](#contributing) 下方的工作流程。 - -#### 围绕一个 Issue 开展工作 - -当您有意愿针对某个 issue 开展工作时，您可以将这个 issue 分配给自己。在 issue 下回复`/assign` 或是 `/assign @yourself`即可。机器人将会把这个 issue -分配给您，您的昵称将会出现在被分配者列表中。 - -#### 提交 Issue - -我们不仅鼓励大家贡献代码，也非常欢迎大家提交 issue。 -issue 需要被提交到 Volcano 的对应子仓库中。 - -举个例子：在 Volcano 中开启一个 issue[Volcano](https://github.com/volcano-sh/volcano/issues) 。 - -开启一个 issue 时，请遵循默认的提交指南。 - -## 贡献者工作流程 - -非常欢迎您提问题或是提交 pull request。 - -这里有一份贡献者工作流程的大纲供您参考： - -- 基于已有分支创建一个自定义分支，通常是基于 master 分支 -- 做一些修改后创建 commit -- 请确保 commit 信息符合正确的格式 -- 将本地个人自定义分支的改动提交到远程个人库 -- 提交的 PR 必须得到至少 2 个社区维护者的 approval 才能合入 - -### 创建 Pull Requests - -Pull requests 经常被简称为“PR”。 -Volcano 遵循标准的[github pull request](https://help.github.com/articles/about-pull-requests/) 流程。 - -在上述流程中，volcano 机器人会对您的 PR 加入结构化标签。 - -Volcano 机器人也会给出一些有用的命令建议，这些命令将会在你的 PR 中运行以提示检视代码。 -可以在注释中输入这些`/command`选项来触发自动标记和通知。 - -### 代码检视 - -为了使您的 PR 更容易的被检视，您需要做到以下几点： - -- 遵循[好代码编写指南](https://github.com/golang/go/wiki/CodeReviewComments) 。 -- 撰写[合规的 commit messages](https://chris.beams.io/posts/git-commit/) -- 将较大的修改分解成若干个逻辑独立的小的修改，这样会使得这些修改更容易被理解，统一起来可以解决一个较大的问题。 -- 给 PR 打上标签以关联到合适的检视人：阅读机器人发给你的指引你完成 PR 流程的提示信息有助于做到这一点。 - -### commit 信息的格式 - -我们对提交消息遵循一个粗略的约定，旨在回答两个问题：做了哪些修改，为什么做这些修改。 -主题中应该描述做了什么，commit 中应该描述为什么做这些。 - -```shell -scripts: add test codes for metamanager - -this add some unit test codes to improve code coverage for metamanager - -Fixes #12 -``` - -也可以采用以下更加正式的格式: - -```shell -: - - - -