data lost after upgrade of craft 4 -> 5

[TK1288]

Describe the bug

we had craft 4 (4.16.19) with supertable plugin v.3.0.16 (schemaVersion: 3.0.0)
several supertable fields, mostly used inside NEO fields. (but also directly used in entry types)

after upgrade to craft 5 (5.9.5) supertable v.4.0.5 schemaVersion: 4.0.0, the structure of supertable fields is empty and also content is gone.

is there any way to fix this? or any solution?

Steps to reproduce

1. in craft 4 create supertable field
2. upgrade craft 4 -> 5

Craft CMS version

5.9.5

Plugin version

4.0.5

Multi-site?

No

Additional context

We have multi-language site

[engram-design]

Out of interest, do you have any Super Table fields outside of Neo, and do they exhibit the same behaviour? Or similarly, any used within Matrix?

I'll have to spin up some new tests for Neo, but last time I checked that was all working.

[TK1288]

Yes we have many supertable fields outside of NEO as well. all of them are gone.
none used within matrix.

So I did upgrade again today, and here is the commands I run:

composer remove carlcs/craft-assetmetadata
composer clear-cache
rm composer.lock
update composer.json content from craft4 admin (prepare composer)
composer require craftcms/cms:^5.0
composer require verbb/super-table:^4.0
composer require spicyweb/craft-neo:^5.0
composer update
composer require carlcs/craft-assetmetadata (did not work but I will add it later)
php craft migrate/all
php craft utils/fix-field-layout-uids

[at this point, craft was upgraded, and supertable field content and stucture was there]

change schema versions manually (as several of them were giving errors) - i attached which ones I changed manually.

and then last command:
php craft project-config/apply
this resulted into error and removed all supertable fields.

[engram-design]

Interesting, thanks for the detailed steps. As you mention, something seems pretty off about the project config handling there, particularly the need to manually change schema versions.

But it could be the fact that you might've rolled back the database on a previous upgrade attempt, but not reverted the YAML changes?

I'll spin up some tests to see if I can replicate.

[TK1288]

Maybe there can be a tool to migrate supertable fields to matrix in craft4?

[engram-design]

Yep, there already is, as part of the regular plugin migration (so it runs at migrate/all), but not as a separate tool.

[TK1288]

Yes there is for craft 5 which obviously doesn't work.. I meant for craft 4 (before upgrading)

[engram-design]

Ah, sorry I see what you mean. Unfortunately, that's probably not going to work and cause more headaches. Mostly because Matrix in Craft 4 just won't be able to handle the (newly converted to Matrix - if we had a tool) Super Table fields.

Still looking into this one.

[TK1288]

I tried to recreate similar matrix fields in craft5, but got errors, as the handles that I had in craft4 for supertable - were reserved words. This made me thinking - maybe this is the reason supertables could not be migrated to matrix fields when upgrading to craft5.

So I went back to craft4 site. changed handles. changed code correspondingly. then upgraded again to craft5.
and it worked! all was converted fine.

Conclusion: when upgrading to craft5, there is no error given when supertable field's handles are reserved words. Would be good to give alert, or maybe it could rename them to keep it? Anyhow, please add this note to the documentation, maybe it will be useful for others :)

Here are the handles that I changed. (the ones that look to me that could cause issues. for sure "key" and "value" was causing issues. others I did not check, as rolling back to craft5 was very time consuming, so I just changed all handles that looked "suspicious" to me.

here are the handles that I changed: (some of them were causing looking structure and data for super tables)

en
es
fr
ar
ch
ru
label
key
val
text
tags