Updated mapping and respective files with download script in NCSES_Employed_College_Grads_Import

[smarthg-gi]

Import name : NCSES_Employed_College_Grads_Import
Test job run: ncses-employed-college-grads-import-smarthg-20260908-060246
validation_output.csv: LINK
differ_summary.json: LINK

Note: Validation reflects 1 deletion and multiple modifications. All of these are justified and expected due to updates in the PVmap. These will be fixed with updates in the latest_version.txt once approved.

Summary

This PR refactors and fixes data mapping and download for the NCSES_Employed_College_Grads_Import.

Key Changes:

1. Automated download script (download.py): Automatically finds and downloads the latest survey file from the NSF website based on the table number, and cleans up year column headers (e.g. changing 2023a to 2023).
2. Fixed mapping errors (pv_map.csv): Fixed mapping where overall group totals were mistakenly assigned to specific jobs. For example, the total count of college-educated women (~15.1M) was previously recorded as female engineers instead of the actual count of 154K.
3. Expanded data coverage: Mapped all available occupations, demographic groups, and survey years (2003–2023), increasing total observations from 68 to 1,571.
4. Added validation rules (validation_config.json): Added validation checks, confirming all changes match the source survey.
5. Documentation & configs: Added a clear README.md and other test files, metadata and manifest file

[gemini-code-assist]

Code Review

This pull request refactors and documents the NCSES Employed College Graduates import pipeline. Key changes include adding a dynamic scraper and header normalizer script (download.py), updating the mapping configuration (pv_map.csv) with future years and header directives, introducing a validation configuration (validation_config.json), and significantly expanding the documentation in README.md. Feedback is provided regarding manifest.json, where running download.py directly without the python3 interpreter or a path prefix will cause execution failures in Unix/Linux environments.

[smarthg-gi]

Import Code Review: PR #2203 & Bug b/535027917

PR: datacommonsorg/data#2203 (NCSES_Employed_College_Grads_Import_mapping_fix)
Bug: b/535027917
Import: NCSES_Employed_College_Grads_Import (statvar_imports/us_nces/nces_employed_college_grads/)
Full Review Document: https://paste.googleplex.com/5971036290547712

---

Key Findings & Required Actions

[P0] Validation Threshold Blocks Production Auto-Refresh

• File: statvar_imports/us_nces/nces_employed_college_grads/validation_config.json:4-11
• Issue: validation_config.json specifies "validator": "DELETED_RECORDS_PERCENT", "params": {"threshold": 0.1}. In production GCS (gs://datcom-prod-imports/.../latest_version.txt -> 2026_09_04T00_03_19_110059_07_00), the baseline dataset generated 68 observations across 14 StatVars. PR Updated mapping and respective files with download script in NCSES_Employed_College_Grads_Import #2203 corrects legacy data corruption, purging 54 corrupted legacy observation records (e.g., Female_SOCEngineersOccupation = 15,111,000). This results in a calculated deletion rate of:
  $$\frac{54}{68} \times 100 = 79.41%$$
  Because $79.41% > 0.1%$, Validator.validate_deleted_records_percent() triggers ValidationStatus.FAILED.
• Impact: On the next scheduled Cloud Batch execution (0 07 4,18 * *), runner.py will fail with exit code 1, aborting the auto-refresh pipeline immediately.
• Action: For this one-time transition to clear historical corruption, either:
  1. Add "enabled": false to check_deleted_records_percent (matching standard practice in PR FAO_Currency_statvar - Added goldens #2107) until the corrected dataset establishes the new baseline in GCS; OR
  2. Relax threshold to 85.0% with documented rationale in the PR description per guidelines.

[P1] Missing Mandatory node_mcf in manifest.json

• File: statvar_imports/us_nces/nces_employed_college_grads/manifest.json:18-23
• Issue: manifest.json specifies template_mcf and cleaned_csv under import_inputs, but omits node_mcf. Core guidelines strictly require:

  "node_mcf (under import_inputs in manifest.json) is mandatory and must use a wildcard pattern (e.g., *.mcf)."

• Impact: In manifest.json, stat_var_processor.py is invoked with --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf. Any newly generated or unmapped StatisticalVariables in future survey cycles are written to output/nces_college_stat_vars.mcf. Omitting node_mcf means the Data Commons import loader will never upload or ingest these schema definitions into the knowledge graph, resulting in unresolvable observation nodes in production.
• Action: Add "node_mcf": "output/*.mcf" to import_inputs[0] in manifest.json.

[P1] Missing Mandatory Freshness / Date Validation

• File: statvar_imports/us_nces/nces_employed_college_grads/validation_config.json:1-13
• Issue: validation_config.json contains no date freshness rules. Guidelines mandate explicit date validation to guarantee freshness and date consistency across refreshes. Because 11 StatVars in the 2023 survey release are statistically suppressed (D, S, *) and lack 2023 observations, a blanket MAX_DATE_CONSISTENT fails, requiring scoped SQL_VALIDATOR rules.
• Action: Add scoped SQL_VALIDATOR rules in validation_config.json:

  {
      "rule_id": "check_dataset_max_date_freshness",
      "description": "Verifies that the dataset includes observations from at least the 2023 survey cycle",
      "validator": "SQL_VALIDATOR",
      "params": {
          "query": "SELECT MAX(CAST(MaxDate AS INTEGER)) AS MaxSurveyYear FROM stats",
          "condition": "MaxSurveyYear >= 2023"
      }
  },
  {
      "rule_id": "check_national_totals_max_date",
      "description": "Ensures benchmark demographic totals reach the latest survey year (2023)",
      "validator": "SQL_VALIDATOR",
      "params": {
          "query": "SELECT StatVar, CAST(MaxDate AS INTEGER) AS MaxYear FROM stats WHERE StatVar IN ('Count_Person_EducationalAttainmentBachelorsDegreeOrHigher_Employed', 'Count_Person_EducationalAttainmentBachelorsDegreeOrHigher_Employed_Female', 'Count_Person_EducationalAttainmentBachelorsDegreeOrHigher_Employed_Male')",
          "condition": "MaxYear >= 2023"
      }
  }

[P1] Unit Test Suite Omitted from PR Commit

• File: statvar_imports/us_nces/nces_employed_college_grads/download_test.py
• Issue: A complete 6-test suite (download_test.py, 158 lines) testing download.py (covering header regex, dynamic URL discovery, network retry mocks, and atomic workbook cleaning) exists locally, but was omitted from PR commit cf2716b.
• Action: Stage and commit download_test.py to the PR branch.

[P2] Test Fixtures & Housekeeping

• File: statvar_imports/us_nces/nces_employed_college_grads/test_data/ncses_input.xlsx: Truncated at row 70 of data (Both sexes), cutting off Female (row 77+) and Male (row 149+). Consequently, the primary bug reported in b/535027917 is completely unexercised by test fixtures.
  • Action: Add rows 77–85 (Female total and occupation sub-rows) and rows 149–157 (Male total and occupation sub-rows) to test_data/ncses_input.xlsx, and regenerate test_data/ncses_output.csv.
• File: statvar_imports/us_nces/nces_employed_college_grads/test_data/nces_input.xlsx: Obsolete 20.8 KB legacy binary file was deleted locally but remains committed in git commit cf2716b.
  • Action: Run git rm statvar_imports/us_nces/nces_employed_college_grads/test_data/nces_input.xlsx and commit the deletion.
• File: statvar_imports/us_nces/nces_employed_college_grads/RUNBOOK.md: 288-line troubleshooting guide exists locally but is untracked. Stage and commit RUNBOOK.md to preserve operational context.

[P3] Code Hygiene & Consistency

• Punctuation in pv_map: pv_map.csv:34 defines "Biological, agricultural, and other life scientists" with punctuation. Guidelines recommend normalized keys without commas or including variants without commas.
• TMCF Dataset Prefix Discrepancy: test_data/ncses_output.tmcf defines Node: E:ncses_output->E0, whereas manifest.json specifies --output_path=output/nces_college (Node: E:nces_college->E0).
• Private Helper Import: download.py:42,218 imports private _retry_method from util/download_util_script.py. Use download_util.request_url() or public retry helpers.
• Permissions: download.py is mode 100644 and lacks #!/usr/bin/env python3. Add shebang and chmod +x download.py.
• Manifest source_files: Add "validation_config.json" to source_files in manifest.json.
• README Paths: Document cd statvar_imports/us_nces/nces_employed_college_grads and library dependencies (openpyxl, requests).

---

Positive Highlights

• Direct fix for b/535027917: Adding #Header,"gender,race,ethnicity" on gender group totals and #Header,"race,ethnicity" on race/ethnicity totals in pv_map.csv:23-33 cleanly separates group totals from occupational breakdowns, populates demographic context for sub-rows, eliminates duplicate observation collisions, and restores valid ingestion of all 209 canonical StatVars across 1,571 observations.
• Dynamic Scrape: download.py:195-241 dynamically discovers the latest survey publication on the NSCG portal, verified live to resolve nsf25322-tab006-002.xlsx.
• Atomic Cleaning: clean_year_headers() in download.py:243-307 normalizes footnote markers in header rows 1–4, writes atomically to cleaned_*.xlsx, and preserves raw file timestamps and HTTP cache integrity.
• Lightweight Test Fixtures: Test fixture footprint pruned from 1,382 rows down to 122 rows (9.2 KB input, 17.6 KB output), strictly adhering to guidelines and 100% byte-for-byte verified with stat_var_processor.py.

[smarthg-gi]

Import Code Review: PR #2203 & Bug b/535027917

PR: datacommonsorg/data#2203 (NCSES_Employed_College_Grads_Import_mapping_fix) Bug: b/535027917 Import: NCSES_Employed_College_Grads_Import (statvar_imports/us_nces/nces_employed_college_grads/) Full Review Document: https://paste.googleplex.com/5971036290547712

Key Findings & Required Actions

[P0] Validation Threshold Blocks Production Auto-Refresh

• File: statvar_imports/us_nces/nces_employed_college_grads/validation_config.json:4-11

• Issue: validation_config.json specifies "validator": "DELETED_RECORDS_PERCENT", "params": {"threshold": 0.1}. In production GCS (gs://datcom-prod-imports/.../latest_version.txt -> 2026_09_04T00_03_19_110059_07_00), the baseline dataset generated 68 observations across 14 StatVars. PR Updated mapping and respective files with download script #2203 corrects legacy data corruption, purging 54 corrupted legacy observation records (e.g., Female_SOCEngineersOccupation = 15,111,000). This results in a calculated deletion rate of:
  79.41
  , Validator.validate_deleted_records_percent() triggers ValidationStatus.FAILED.

• Impact: On the next scheduled Cloud Batch execution (0 07 4,18 * *), runner.py will fail with exit code 1, aborting the auto-refresh pipeline immediately.

• Action: For this one-time transition to clear historical corruption, either:

  1. Add "enabled": false to check_deleted_records_percent (matching standard practice in PR FAO_Currency_statvar - Added goldens #2107) until the corrected dataset establishes the new baseline in GCS; OR
  2. Relax threshold to 85.0% with documented rationale in the PR description per guidelines.

[P1] Missing Mandatory node_mcf in manifest.json

• File: statvar_imports/us_nces/nces_employed_college_grads/manifest.json:18-23
• Issue: manifest.json specifies template_mcf and cleaned_csv under import_inputs, but omits node_mcf. Core guidelines strictly require:

  "node_mcf (under import_inputs in manifest.json) is mandatory and must use a wildcard pattern (e.g., *.mcf)."

• Impact: In manifest.json, stat_var_processor.py is invoked with --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf. Any newly generated or unmapped StatisticalVariables in future survey cycles are written to output/nces_college_stat_vars.mcf. Omitting node_mcf means the Data Commons import loader will never upload or ingest these schema definitions into the knowledge graph, resulting in unresolvable observation nodes in production.
• Action: Add "node_mcf": "output/*.mcf" to import_inputs[0] in manifest.json.

[P1] Missing Mandatory Freshness / Date Validation

• File: statvar_imports/us_nces/nces_employed_college_grads/validation_config.json:1-13
• Issue: validation_config.json contains no date freshness rules. Guidelines mandate explicit date validation to guarantee freshness and date consistency across refreshes. Because 11 StatVars in the 2023 survey release are statistically suppressed (D, S, *) and lack 2023 observations, a blanket MAX_DATE_CONSISTENT fails, requiring scoped SQL_VALIDATOR rules.
• Action: Add scoped SQL_VALIDATOR rules in validation_config.json:

  {
      "rule_id": "check_dataset_max_date_freshness",
      "description": "Verifies that the dataset includes observations from at least the 2023 survey cycle",
      "validator": "SQL_VALIDATOR",
      "params": {
          "query": "SELECT MAX(CAST(MaxDate AS INTEGER)) AS MaxSurveyYear FROM stats",
          "condition": "MaxSurveyYear >= 2023"
      }
  },
  {
      "rule_id": "check_national_totals_max_date",
      "description": "Ensures benchmark demographic totals reach the latest survey year (2023)",
      "validator": "SQL_VALIDATOR",
      "params": {
          "query": "SELECT StatVar, CAST(MaxDate AS INTEGER) AS MaxYear FROM stats WHERE StatVar IN ('Count_Person_EducationalAttainmentBachelorsDegreeOrHigher_Employed', 'Count_Person_EducationalAttainmentBachelorsDegreeOrHigher_Employed_Female', 'Count_Person_EducationalAttainmentBachelorsDegreeOrHigher_Employed_Male')",
          "condition": "MaxYear >= 2023"
      }
  }

[P1] Unit Test Suite Omitted from PR Commit

• File: statvar_imports/us_nces/nces_employed_college_grads/download_test.py
• Issue: A complete 6-test suite (download_test.py, 158 lines) testing download.py (covering header regex, dynamic URL discovery, network retry mocks, and atomic workbook cleaning) exists locally, but was omitted from PR commit cf2716b.
• Action: Stage and commit download_test.py to the PR branch.

[P2] Test Fixtures & Housekeeping

• File: statvar_imports/us_nces/nces_employed_college_grads/test_data/ncses_input.xlsx: Truncated at row 70 of data (Both sexes), cutting off Female (row 77+) and Male (row 149+). Consequently, the primary bug reported in b/535027917 is completely unexercised by test fixtures.

  • Action: Add rows 77–85 (Female total and occupation sub-rows) and rows 149–157 (Male total and occupation sub-rows) to test_data/ncses_input.xlsx, and regenerate test_data/ncses_output.csv.

• File: statvar_imports/us_nces/nces_employed_college_grads/test_data/nces_input.xlsx: Obsolete 20.8 KB legacy binary file was deleted locally but remains committed in git commit cf2716b.

  • Action: Run git rm statvar_imports/us_nces/nces_employed_college_grads/test_data/nces_input.xlsx and commit the deletion.

• File: statvar_imports/us_nces/nces_employed_college_grads/RUNBOOK.md: 288-line troubleshooting guide exists locally but is untracked. Stage and commit RUNBOOK.md to preserve operational context.

[P3] Code Hygiene & Consistency

• Punctuation in pv_map: pv_map.csv:34 defines "Biological, agricultural, and other life scientists" with punctuation. Guidelines recommend normalized keys without commas or including variants without commas.
• TMCF Dataset Prefix Discrepancy: test_data/ncses_output.tmcf defines Node: E:ncses_output->E0, whereas manifest.json specifies --output_path=output/nces_college (Node: E:nces_college->E0).
• Private Helper Import: download.py:42,218 imports private _retry_method from util/download_util_script.py. Use download_util.request_url() or public retry helpers.
• Permissions: download.py is mode 100644 and lacks #!/usr/bin/env python3. Add shebang and chmod +x download.py.
• Manifest source_files: Add "validation_config.json" to source_files in manifest.json.
• README Paths: Document cd statvar_imports/us_nces/nces_employed_college_grads and library dependencies (openpyxl, requests).

Positive Highlights

• Direct fix for b/535027917: Adding #Header,"gender,race,ethnicity" on gender group totals and #Header,"race,ethnicity" on race/ethnicity totals in pv_map.csv:23-33 cleanly separates group totals from occupational breakdowns, populates demographic context for sub-rows, eliminates duplicate observation collisions, and restores valid ingestion of all 209 canonical StatVars across 1,571 observations.
• Dynamic Scrape: download.py:195-241 dynamically discovers the latest survey publication on the NSCG portal, verified live to resolve nsf25322-tab006-002.xlsx.
• Atomic Cleaning: clean_year_headers() in download.py:243-307 normalizes footnote markers in header rows 1–4, writes atomically to cleaned_*.xlsx, and preserves raw file timestamps and HTTP cache integrity.
• Lightweight Test Fixtures: Test fixture footprint pruned from 1,382 rows down to 122 rows (9.2 KB input, 17.6 KB output), strictly adhering to guidelines and 100% byte-for-byte verified with stat_var_processor.py.

[P0] Validation Threshold Blocks Production Auto-Refresh
A single deletion is the cause of validation failure and this deletion is justified, since existing pvmap was incorrectly mapping SV to suppressed Data, when this mapping was corrected, the SV correctly pointed to suppressed Data (null value), hence causing deletion. This will not be the case in future. Hence increasing the deletion threshold or removing it is not correct.

[P1] Missing Mandatory node_mcf in manifest.json
Added node_mcf in manifest.json

[P1] Missing Mandatory Freshness / Date Validation
MAX_DATE_CONSISTENT fails because source suppresses Data for small demographic groups across the years, so max date will always stay inconsistent.
When tried with SQL validator, freshness (CURRENT_DATE - N) cannot be used since this is a biennial survey with a ~3-year publication lag (2023 data released in 2026; 2025 expected ~2028). A calendar-based date rule returns incorrect response.
Example: If we use Year - 3, in 2027, the check expects 2024 data. Valid 2023 data will falsely FAIL simply because source data has not been published for 2025 yet.
If we use Year - 5, in 2028, the check threshold is 2023. It will falsely PASS even if the pipeline got stuck and failed to update to the new 2025 release.

[P1] Unit Test Suite Omitted from PR Commit
The test_download script was strictly a temporary debugging tool used during development to verify regex patterns locally. It was not meant to be added to the PR. The execution of download.py has been verified live. Therefore, download_test.py is not being included in this PR.

[P2] Test Fixtures & Housekeeping
Due to the test_data file size limits, the input files is limited to 70 contiguous rows. The data follows a strict hierarchical structure with multiple parent demographic sections. Trimming or splicing non-contiguous rows cuts off the necessary parent headers and section dividers, which will produce invalid test output.
Runbooks are not generally added to the PR

[P3] Code Hygiene & Consistency
The first column of pv_map serves as the exact lookup key against the raw cell values in the input file. In the source data, the cell contains the exact text "Biological, agricultural, and other life scientists" (including the commas). Because this table contains multiple other occupation categories sharing overlapping phrasing, keeping the full, exact string in pv_map.csv ensures unambiguous key matching and avoids data corruption.
The test_data and output/ are completely different directories and test_data is not run in the prod runs.

Removed _retry_method and updated resolve_url() to use the public download_util.request_url(). Added #!/usr/bin/env python3 to the top of download.py

[saanikaaa]

I see deletion in test run. https://screenshot-v2.corp.google.com/5tstqqt2ioqeo
This will again led to failure. Pls look into it

[saanikaaa]

Pls add import name in PR title

[smarthg-gi]

Pls add import name in PR title

Added

[smarthg-gi]

I see deletion in test run. https://screenshot-v2.corp.google.com/5tstqqt2ioqeo This will again led to failure. Pls look into it

The deletion is expected due to PVmap changes. Please refer to the Deletion report

[hareesh-ms]

Take a look at the review comments addded.

[smarthg-gi]

Take a look at the review comments addded.

No new comments are visible. Please point to the comments

[vish-cs]

I am wondering why we have to add a default deletion threshold. If we know that the deletion is expected in this case, we should simply allow the ingestion for this particular run. I would like to know if there is a different guidance share with the team to address deletions.

[smarthg-gi]

I am wondering why we have to add a default deletion threshold. If we know that the deletion is expected in this case, we should simply allow the ingestion for this particular run. I would like to know if there is a different guidance share with the team to address deletions.

Hi Vishal, as a standard practice, we are adding a deletion threshold to all the imports irrespective of the deletions. Here the deletion is expected and the import will still fail in the validation even if we remove the deletion threshold. Please suggest if there is a different understanding