Ecosyste.ms: Timeline
Browse the timeline of events for every public repo on GitHub. Data updated hourly from GH Archive.
rymarczy pushed 1 commit to main mbta/dmap-import
- FIX: CUBIC DMAP API Authorization Timeout (#30) This change fixes the situation where DMAP API file links timeout af... 8237ce6
rymarczy closed a pull request on mbta/dmap-import
FIX: CUBIC DMAP API Authorization Timeout
This change fixes the situation where DMAP API file links timeout after about an hour. If file link timeout is detected, via a HTTP 403 Code during the file link download, API results are refre...
rymarczy opened a pull request on mbta/dmap-import
FIX: CUBIC DMAP API Authorization Timeout
This change fixes the situation where DMAP API file links timeout after about an hour. If file link timeout is detected, via a HTTP 403 Code during the file link download, API results are refre...
rymarczy pushed 1 commit to main mbta/dmap-import
- FIX: Load _history tables from temp table (#29) This change modifies the loading of _history table. Rather than l... a02fdea
rymarczy created a review comment on a pull request on mbta/dmap-import
Added the fix for the S3 Permission error we were seeing as well
rymarczy pushed 1 commit to rr-load-history-from-temp-table mbta/dmap-import
- fix error prefix 4e021ef
rymarczy created a review comment on a pull request on mbta/dmap-import
It's sort of a "temp" table. It gets created by the `dmap_import` application when it starts to load a QLIK table and then gets dropped at the end. So DB users would only ever see it while the ETL ...
grejdi-mbta created a review comment on a pull request on mbta/dmap-import
If it is just a table to load stuff, maybe truncating at the end as well? I want it to be obvious that it shouldn't be used by data analysts.
grejdi-mbta created a review comment on a pull request on mbta/dmap-import
update comment here. q: is `_load` a new type of table at this point? we need to communicate to folks who are using the database what it is, so they can ignore if need be.
rymarczy opened a pull request on mbta/dmap-import
FIX: Load _history tables from temp table
This change modifies the loading of `_history` table. Rather than load a CSV directly into the `_histroy` table, we will first load records into the `_load` table and then INSERT into the `_hist...
rymarczy closed a pull request on mbta/dmap-import
FEAT: Decrease run-time duration of ODS-QLIK Process
This change alters the way ods-qlik data is loaded into the dmap-import database. Previously the ods-qlik data loading steps where: 1. load individual cdc csv.gz files into `_history` table 2....
grejdi-mbta created a review on a pull request on mbta/dmap-import
Please make sure to test the error bucket addition.
rymarczy pushed 1 commit to rr-change-ods-qlik-update-process mbta/dmap-import
- add ERROR bucket back 04e6005
grejdi-mbta created a review comment on a pull request on mbta/dmap-import
Up to you. But I think we should have it in there. The glue pipeline is unnecessarily strict so we shouldn't constrain this pipeline because of it.
rymarczy created a review comment on a pull request on mbta/dmap-import
I dropped it when I was trying to simplify things, I can add it back.
rymarczy created a review comment on a pull request on mbta/dmap-import
In local testing it didn't look like this would be an issue. Limiting the size of the hash folders to ~60MB will also limit the memory usage of the dataframe that gets created from the csv files. W...
rymarczy created a review comment on a pull request on mbta/dmap-import
Correct, using `TemporaryDirectory()` as `with` block removes the whole folder when exiting the block.