marianobrc closed a pull request on PADAS/cdip

October 22, 2024 8:53pm

GUNDI-3561: Partition activity logs

### What does this PR do? - Ports some partitioning utils from core - Implements a partitioner class to partition activity logs: - Logs are partitioned first by type (list partition for eve...

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 8:38pm

• Remove feature branch from pipeline 0c1514b

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 8:05pm

• Remove psql logging disable step d9db276

View on GitHub

marianobrc pushed 3 commits to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 7:49pm

• Fix issue increasing offset in batch copy 3758095
• Change partitioning interval to one week a6b274a
• Migrate data since 3 weeks ago 82f6e02

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 4:33pm

• Disable psql logging while migrating events 0129b70

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 2:56pm

• Fix in max-batches behaviour 8a81cf9

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 2:45pm

• Add missing quotes in function call 41dc09f

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 2:30pm

• Only migrate events of the last month bab03c5

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 11:18am

• Support setting a maximum number of batches to migrate 5dbf7f8

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 10:51am

• Improvement in batch migration functions 0e539be

View on GitHub

marianobrc pushed 2 commits to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 10:36am

• Fix sql issues saving offset in logging table 01f8c3d
• Improvement in batch migration functions b9e872d

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 10:30am

• Fix sql issues saving offset in logging table ff73826

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 10:21am

• Fix sql issues in store function insert_activity_log_batch d803027

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 4:25am

• Extra logging in batch data migration ee53453

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 4:17am

• Fix sql issues in store function insert_activity_log_batch 08c3ba5

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 3:58am

• Refactor data migration using stored functions fe333ff

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 2:53am

• Fix sql issue initializing logging table 74ba311

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 2:45am

• Fix sql issue with logging table e8279e5

View on GitHub

marianobrc pushed 4 commits to gundi-3561-partition-logs PADAS/cdip

October 22, 2024 2:38am

• Migrate data changes and events in separate steps and track last offset f87a480
• Add savepoints per batch and change exit condition c4cd0c7
• Add exception handling on batch queries 4f8abfb
• log offset in batch processing 187b00e

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 21, 2024 8:26pm

• Copy all the data change logs in data migration 912b4ff

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 21, 2024 7:05pm

• Add more memory to dev and stage api pods 434d7dc

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 21, 2024 6:28pm

• Migrate onyl events of the last month 1baadd2

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 21, 2024 6:21pm

• Migrate only events of the last month f8b1b57

View on GitHub

marianobrc pushed 2 commits to gundi-3561-partition-logs PADAS/cdip

October 21, 2024 4:30pm

• Split data migration queries for data changes and events 7a68d2f
• Rename variable 1a3be65

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 21, 2024 3:36pm

• Add extra logging for the data migration step 54f04aa

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 21, 2024 3:19pm

• Fix var name in data migration logging 80490c0

View on GitHub

marianobrc pushed 1 commit to gundi-3561-partition-logs PADAS/cdip

October 21, 2024 3:09pm

• Add extra logging for the data migration step d5d15d5

View on GitHub

davidovichmarcos created a review on a pull request on PADAS/cdip

October 21, 2024 11:53am

lgtm

View on GitHub

marianobrc pushed 4 commits to gundi-3561-partition-logs PADAS/cdip

October 21, 2024 11:49am

• Support filtering connections by source ebf2126
• Merge pull request #326 from PADAS/filter-connections-by-source Support filtering connections by source 0357100
• change connection url 160a29e
• Merge branch 'main' into gundi-3561-partition-logs 5d31b4a

View on GitHub

daniefdz pushed 1 commit to main PADAS/cdip

October 18, 2024 10:58pm

• change connection url 160a29e

View on GitHub