Data Import
Read CSV, TSV, XLSX and the clipboard. The wizard guesses types across locales, remembers a column mapping, validates through each column's schema, and stages the rows in the grid itself so they can be fixed in place before they are committed.
The Whole Flow
Edit the CSV below if you like, then load it. The wizard parses it, guesses which file column feeds which grid column, and stages the rows in the grid. Three of the four rows are deliberately wrong.
<script lang="ts">
import {
createDataGrid, DataGrid, dataImport, Grid
} from '@sv5ui/datagrid';
const grid = createDataGrid<Order>({
columns, data: orders, getRowId,
features: [
editing(), // so a staged cell can be fixed where it sits
dataImport<Order>({
onCommit: (rows) => { orders = [...orders, ...rows]; }
})
]
});
</script>
<Grid.ImportWizard {grid} />
<DataGrid {grid} />
<!-- onCommit is the only required option. It is handed the rows the reader
accepted, and what you do with them is yours: append them, POST them,
reconcile them against what is already there. -->notanumber is still in the quantity cell and maybe is still in the boolean one, so opening the cell starts the editor on exactly what the
file said. That is what makes fixing it in place possible, and it is why the rows are
staged in the grid rather than in a modal. notanumber renders nothing. The red ring says which cell is wrong and the summary above says what
kind of wrong, but the offending text only appears once the editor opens. Scanning a hundred
staged rows means opening the marked cells one at a time. Committing to a Server
The wizard answers one question: can this cell be read as the type its column declares. That is not the same as whether the row is allowed, and nothing in the browser can answer the second one. The catalogue lives on the server, so the SKU that does not exist is only found there.
This CSV parses cleanly and all five rows pass review. Commit it and the server keeps one: of the other four, one SKU is not in the catalogue, one was ordered last week, one asks for a quantity of none, and one repeats a line above it. Only the accepted row is added to the grid, and each refusal comes back with a reason attached.
No data
dataImport<Order>({
newRow: (index) => ({ id: `import-${index}` }),
// onCommit is the seam. The wizard has said the cells can be read; the
// server says whether the rows are allowed, and only it can.
onCommit: async (imported) => {
const response = await fetch('/api/orders', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ rows: imported })
});
const { verdicts } = await response.json();
// Add the rows it took, and keep the reasons for the ones it did not.
const allowed = new Set(verdicts.filter((v) => v.accepted).map((v) => v.id));
grid.data = [...grid.data, ...imported.filter((row) => allowed.has(row.id))];
rejected = verdicts.filter((v) => !v.accepted);
}
});
// The two checks answer different questions, and the wizard cannot do the
// second one. "7" is a readable quantity whatever the catalogue says; DG-999
// parses as text and is still not a product. A row that passed review can
// come back refused, so the commit is not the end of the flow.Three Steps
// Three steps, and the state says which one it is on.
//
// idle nothing loaded. The drop zone is showing
// mapping the file is parsed. Which file column feeds which grid column
// review the rows are staged in the grid: what is wrong, what to keep
const wizard = getDataImport(grid)!;
await wizard.take(file); // a File from a drop or a picker
await wizard.takeText(csv, 'csv'); // a string you already hold
await wizard.takeBytes(bytes, 'Sheet2'); // an XLSX workbook as bytes
wizard.step; // 'idle' | 'mapping' | 'review'
await wizard.review(); // mapping -> review
await wizard.commit(); // review -> onCommit, then back to idle
wizard.back();
wizard.cancel();ImportStep
| Step | What it means |
|---|---|
idle | Nothing loaded. The drop zone is showing |
mapping | The file is parsed. Which file column feeds which grid column |
review | The rows are staged in the grid. What is wrong, and what to keep |
The Mapping Is Guessed From Header Text
Not from column position. The sample above has headers unit price and ordered on, and they land in the price and orderedOn columns, so a file whose columns are in another order still arrives correctly. Reorder the header
line above and load it again to see it hold.
wizard.headers;
// ['sku', 'quantity', 'unit price', 'ordered on', 'active']
wizard.mappings;
// [{ columnId: 'sku', sourceIndex: 0 },
// { columnId: 'quantity', sourceIndex: 1 },
// { columnId: 'price', sourceIndex: 2 },
// { columnId: 'orderedOn', sourceIndex: 3 },
// { columnId: 'active', sourceIndex: 4 }]
wizard.matchedCount; // 5
wizard.preview; // the first rows, as text
wizard.setMapping('price', 2);
wizard.setMapping('notes', null);
wizard.setHasHeader(false);
dataImport({ onCommit, rememberAs: 'orders-import' });
// "unit price" found price and "ordered on" found orderedOn: the guess
// reads header text, so a file in another column order still lands.Review, and Fixing in Place
review() coerces every cell against its column's type and stages the rows in the grid, where the data
will actually live, rather than in a modal that looks nothing like it. Then commit everything
or only what is clean.
// review() coerces every cell against its column's type and stages the rows
// IN THE GRID, so the reader sees them where the data will live, rather
// than in a modal that looks nothing like it.
wizard.staged; // the rows, coerced
wizard.problems; // { rowIndex, columnId, kind, message }[]
wizard.badRows; // Set of row indices with at least one problem
wizard.issueCounts; // [{ message, count }] for a summary
wizard.validCount; // how many rows are clean
wizard.sourceCount; // how many the file held
// A cell that will not coerce KEEPS ITS ORIGINAL TEXT rather than being
// nulled, which is what makes fixing it in place possible: the reader sees
// exactly what the file said, in the cell, and edits it there.
// Then keep everything, or only what is clean:
await wizard.commit();
await wizard.commit({ validOnly: true });What a Cell Can Fail On
wizard.problems;
// [{ rowIndex: 1, columnId: 'quantity',
// kind: 'number', message: 'Quantity is not a number' },
// { rowIndex: 2, columnId: 'orderedOn',
// kind: 'date', message: 'Ordered on is not a date' },
// { rowIndex: 3, columnId: 'active',
// kind: 'boolean', message: 'Active is not a yes or a no' }]
dataImport({
onCommit,
messageFor: (kind, column) =>
kind === 'number' ? `${column.header} must be a quantity` : undefined,
dedupe: { columns: ['sku'], against: 'both' }
});
// The other three kinds are invalid (the column's schema refused it),
// duplicate (repeats a row in the file) and existing (repeats one in the
// grid). Duplicates are opt-in: what counts as the same row is yours.ProblemKind
Each problem carries the row index, the column id, the kind, and a message worded from the column's own header.
| Kind | When |
|---|---|
number | The column is numeric and the text is not |
date | The column is a date and the text is not one |
boolean | The column is a boolean and the text is neither a yes nor a no |
invalid | The column's own schema refused it |
duplicate | The row repeats another row in the same file |
existing | The row repeats one already in the grid |
Guards
An XLSX is unzipped and parsed in the page, so a file large enough to take the tab down has to be refused before it is read rather than after.
dataImport({
onCommit,
accept: ['csv', 'tsv', 'xlsx', 'clipboard'], // all four by default
// Both already have a default: 200,000 rows and 32 MB. Lower them rather
// than assuming a file arrives unguarded.
maxRows: 50_000,
maxBytes: 20 * 1024 * 1024,
// Where a comma is the decimal mark rather than the thousands separator.
decimal: ',',
// Otherwise the reader picks from wizard.sheets.
sheet: 'Orders',
// Defaults for the fields the file does not carry.
newRow: (index) => ({ status: 'pending', createdAt: new Date() })
});Options
| Option | Description |
|---|---|
onCommit | Handed the rows the reader accepted. The only required option |
accept | 'csv', 'tsv', 'xlsx', 'clipboard' |
rememberAs | Key in localStorage the column mapping is remembered under. The default is built from the columns, not from the grid, so two grids with the same columns share one remembered mapping and changing the columns starts a fresh one. Pass a string to name it yourself, or null to remember nothing |
newRow | Defaults for the fields the file does not carry, an id among them |
messageFor | Reword a problem. Return undefined to keep the default |
dedupe | Which columns make a row the same. against is 'file', 'grid' or 'both', and defaults to 'both' |
maxRows / maxBytes | Refuse a file before parsing it, because an XLSX is unzipped in the page. Both are already set, so lower them rather than assuming there is no guard |
decimal | Where a comma is the decimal mark rather than the thousands separator |
sheet | Which sheet of a workbook, rather than asking the reader |
Driving It From Code
import { getDataImport } from '@sv5ui/datagrid';
const wizard = getDataImport(grid)!;
wizard.busy; // a parse or a review is running
wizard.error; // what went wrong, as a string, or null
wizard.fileName;
wizard.sheets; // the sheet names an XLSX held
wizard.accepts; // the formats this wizard takes
wizard.canReview;
wizard.isReviewing;
wizard.isStaged(rowId); // whether a row came from this import