Should we consider removing the import preview option?

[pavish]

We have a preview option when importing data. When we first started discussing this feature, I was very much in support of it for the following reasons:

• I assumed the data will be shown directly from the uploaded file instead of the db, making it less resource intensive for the backend.
• I expected this to provide a quick overview to the user before the import actually happens into the table which might take considerable time.
• From the user's perspective, this preview data never enters the db. It is just displayed and then saved only when we complete the import.
• The user will be able to choose the first row as the column name, change column names, select which columns to import and choose column type, before "actually" importing the data into the db. We also automatically load type suggestions.

However, later during implementation, our entire backend logic depended on importing the data into a table, in the db. Every user operation translated to a database action, making it not very different from performing these actions on any other table. We went ahead with the feature nevertheless.

---

I am now questioning the need for this feature in the first place. Each of the above reasons is mostly void.

I assumed the data will be shown directly from the uploaded file instead of the db, making it less resource intensive for the backend.
I expected this to provide a quick overview to the user before the import actually happens into the table which might take considerable time.

These were the major reasons for including this feature in the first place. But these are not the cases anymore, since the whole data is stored in a table anyway.

From the user's perspective, this preview data never enters the db. It is just displayed and then imported only when we complete the import.

This isn't true, and we might confuse a technical user who doesn't expect the data to be in the db.

The user will be able to choose the first row as the column name, change column names, change table name, select which columns to import and choose column type, before "actually" importing the data into the db. We also automatically load type suggestions.

• We can retain the options to select whether the first row is a column, type in table name after the file uploads. We could present it on the same page instead of the next.
• When the table is uploaded, we can do the type interference and selection on the backend directly, instead of needing the suggestions endpoint.
• All of the other operations can be done on a table directly.

---

I still am attached to this feature because it took us all considerable work to implement. I remember working more than a week on getting the file upload flow ready on the frontend. But now I'm not sure if the preview adds any value to the user at all.

• From the user's perspective:

  1. It introduces an additional step in the workflow, which is mostly unnecessary.
  2. It may seem to the user like a confirmation step. But in reality it's all in the db and that might confuse the user.
  3. The options provided are going to be very similar to the actual table displayed in the end. We have a ticket to change the column header to select the mathesar type and db & display options. What additional benefit does the preview table offer compared to the actual table?
  4. The user can easily perform the same actions on the actual table.

• From a development perspective:

  1. We have to perfect this feature (at the very least bug-free) for the alpha, since this is the very first step into using Mathesar.
  2. We still have a lot of work here.
     • Changing the column header to be able to select Mathesar types
     • Selecting db options and display options.
     • I don't see what benefit this will provide to the user since the same can be done on the table later.

---

Overall,

• This feature offers very little to the user
• Still contains a lot of development work
• Is not any less resource intensive for the backend
• It's a non-frequent user operation

Should we consider removing it?

[mathemancer]

For me, the main piece of this step that needs to be kept is a confirmation from the user that we haven't mangled their data when importing it and inferring the type. For example, depending how they write their dates, and what locale the DB is in, it's possible the days and months would be switched.

One way to avoid this would be to import every column as a string, and get them to either

• manually change the types as appropriate, verifying each column one at a time, or
• Add some "infer a type for this column" flow where they can try to infer a type for each column one at a time, or
• Add some "infer types of all columns in this table" flow where they can do the table-wide inference at a later step in the flow.

I'm not sure this is better. The plus side is that the initial import is quicker, and we don't need to write to the DB for the preview. The downside is that then they need to do more work to get their columns properly typed.

[pavish]

Ah, I completely overlooked this issue. Okay, I think this is a very strong argument to keep the import preview feature.

[pavish]

Perhaps if we do the import directly without preview, we don't automatically infer Date fields, instead we infer them as Text. I assume auto inferring the other types should be mostly fine.

Instead we show a hint bubble on the table after import, near the column header, asking the user whether they'd like to change the type as DATE. We have the type_suggestion endpoint which we can use for it.

The downside is that then they need to do more work to get their columns properly typed.

They'll still have to do this during preview right? Considering that we may have to show both database and display options here, what is the difference in doing it during preview vs on the actual table (except the cases like the one with dates)?

[mathemancer]

The date example was only an example. I think we need to double-check with the user for any type where we've made decisions during import about interpreting the input data strings. Even BOOLEAN could have some ambiguity. The only exceptions are where we simply read in the input string as TEXT.

I do see a valid flow where we just import everything as text and let them handle it after the fact.

[kgodey]

As @mathemancer pointed out, the primary purpose of the import preview is to help the user get their columns set up with the correct data type without:

• us making assumptions that break their data (i.e. the result of us guessing types without seeking confirmation)
• the user having to make a lot of choices manually to get their data set up correctly (i.e. the result of importing all data as text)

I agree that our current import preview UX isn't optimal and perhaps we can make it a special state of a table (and reuse most of our table UX). We already have a design issue to update our import preview UX (#934) and we can handle that when we get there. I've linked to this discussion in this issue description.