Skip to content
This repository was archived by the owner on Sep 28, 2022. It is now read-only.
This repository was archived by the owner on Sep 28, 2022. It is now read-only.

GenericParser should be resilient to bad fields #53

Description

@jaffee

The generic parser currently returns an error any time it fails to parse a field's value, and stops processing for that record. In reality, there are lots of innocuous reasons why a certain field might fail to parse, and it doesn't indicate that the entire record is suspect.

We should have stat/log options for notifying and counting which fields are failing, why, and how often, but we should make a best effort to parse any record and return some data.

I don't think we need to go crazy with configurability, some count stats that use the field path and encapsulate what the error is (e.g. null value, unsupported type, etc.). pdk/ingest.go has an example of a simple stats interface, that we should probably extend throughout the codebase so that it can be configured at the top level.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions