Conversation
JulienVig
left a comment
JulienVig
left a comment
There was a problem hiding this comment.
Missing things required before merging (for me to handle):
- Use metadata statistics during model validation
- Current titanic sample csv is rejected by webapp because of missing data
- Standardization is done locally. This can lead to different weight scales between clients. Should the standardization statistics be averaged by the server?
- Unit tests for new tabular preprocessing functions
- Validate CSV right away when connecting it
- update docs/examples/custom_task.ts
| for (const col of columns){ | ||
| const values = rows.map((row)=> { | ||
| const rawValue = extractColumn(row, col); | ||
| return convertToNumber(rawValue !== "" ? rawValue : "0"); |
There was a problem hiding this comment.
Note to self: prompt users to choose missing data imputation method
| } | ||
|
|
||
| // If the column is numerical column, apply standardization | ||
| const value = convertToNumber(raw !== "" ? raw : "0"); |
There was a problem hiding this comment.
default missing data imputation
| tabularStandardization: stats, | ||
| }; | ||
|
|
||
| const preprocessed = processing.preprocess( |
There was a problem hiding this comment.
this.preprocessOnce is ignored here
| let preprocessed = processing.preprocess(this.#task, dataset); | ||
| if (validationSplit === 0){ | ||
| if (this.#task.dataType === "tabular"){ | ||
| const rows = await arrayFromAsync(dataset as Dataset<DataFormat.Raw["tabular"]>); |
There was a problem hiding this comment.
Here we read throught the whole dataset. Therefore this.preprocessOnce is ignored=false for tabular dataset. This is fine but we might as well take advantage of this and keep the dataset cached rather than discard it and read it again during training
There was a problem hiding this comment.
extractToNumbers is not used anymore
| // TODO totally unsafe casting | ||
| rawModel as tf.io.ModelArtifacts, | ||
| // metadata for tabular task standardization | ||
| rawMetadata as ModelMetadata, |
There was a problem hiding this comment.
Implement more checks before casting. msgpack potentially returns null instead of undefined when the field is missing
2 participants
Initial implementation of tabular data preprocessing
Closes #1062, fixes #1063
Changes
TODO