Ecosyste.ms: Timeline
Browse the timeline of events for every public repo on GitHub. Data updated hourly from GH Archive.
rymarczy pushed 1 commit to main mbta/dmap-import
- FIX: Staging failing on 2d6f7140c642 migration (#39) Staging is failing on this migration because ODS tables do not ... 769cee8
rymarczy closed a pull request on mbta/dmap-import
FIX: Staging failing on 2d6f7140c642 migration
Staging is failing on this migration because ODS tables do not exist. This was already successfully deployed to PROD.
rymarczy opened a pull request on mbta/dmap-import
FIX: Staging failing on 2d6f7140c642 migration
Staging is failing on this migration because ODS tables do not exist. This was already successfully deployed to PROD.
rymarczy created a branch on mbta/dmap-import
fix-staging-migration-fail - Import DMAP data into a PostgreSQL database
rymarczy closed a pull request on mbta/dmap-import
FEAT: Create VIEWS for WC700 Comp A&B Reports
This change creates views for the WC700 Comp A&B Reports. SQL for these reports was provided by Cubic. Reports are grouped by primarily (settlement_day_key, operating_day_key & payment_type_key).
rymarczy opened a pull request on mbta/dmap-import
FEAT: Create VIEWS for WC700 Comp A&B Reports
This change creates views for the WC700 Comp A&B Reports. SQL for these reports was provided by Cubic. Reports are grouped by primarily (settlement_day_key, operating_day_key & payment_type_key).
rymarczy created a branch on mbta/dmap-import
feat-wc700-comp-a-b - Import DMAP data into a PostgreSQL database
grejdi-mbta created a review comment on a pull request on mbta/dmap-import
Makes sense. Thanks for the extra explanation.
rymarczy pushed 1 commit to main mbta/dmap-import
- FIX: Drop Tables for migration / Dynamic JOIN operation (#37) - Explicitly drop these tables when deleting the statu... 77abc0c
rymarczy closed a pull request on mbta/dmap-import
FIX: Drop Tables for migration / Dynamic JOIN operation
Two changes in this PR: 1. Explicitly drop these tables when deleting the status files to force full table reset. 2. Dynamically select join operation (`=` or `IS NOT DISTINCT FROM`) in update...
rymarczy created a review comment on a pull request on mbta/dmap-import
No, we're doing all the existing operations in polars already. We limit the "csv" size of each batch to 60mb, so the memory footprint for any batch is always less than 1gb.
grejdi-mbta created a review comment on a pull request on mbta/dmap-import
Any risk here with a file with many records here? Polars is pretty fast so I'm assuming not.
rymarczy created a review comment on a pull request on mbta/dmap-import
This is all un-changed just got indented into the `for` loop
rymarczy created a review comment on a pull request on mbta/dmap-import
I moved this `for` loop into the `cdc_update` function for clarity.
rymarczy created a review comment on a pull request on mbta/dmap-import
This is the function that produces the dynamic JOIN operands.
rymarczy opened a pull request on mbta/dmap-import
FIX: Also Drop Tables for migration
I thought these tables get automatically dropped if we delete the status file, but that is not the case. Explicitly drop these tables when deleting the status files to force full table reset.
rymarczy created a branch on mbta/dmap-import
hotfix-drop-null-tables - Import DMAP data into a PostgreSQL database