Commit Graph

92 Commits

Author SHA1 Message Date
Mostafa Abdelraouf
87a771aecc Log error messages for network failures (#289)
We are seeing some Error reading message code from socket error messages, we want to get more context so this PR logs the actual error reported.
2023-01-19 05:18:08 -06:00
dependabot[bot]
99a3b9896d chore(deps): bump activerecord from 7.0.3.1 to 7.0.4.1 in /tests/ruby (#287)
Bumps [activerecord](https://github.com/rails/rails) from 7.0.3.1 to 7.0.4.1.
- [Release notes](https://github.com/rails/rails/releases)
- [Changelog](https://github.com/rails/rails/blob/v7.0.4.1/activerecord/CHANGELOG.md)
- [Commits](https://github.com/rails/rails/compare/v7.0.3.1...v7.0.4.1)

---
updated-dependencies:
- dependency-name: activerecord
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-18 16:56:16 -08:00
Mostafa Abdelraouf
7894bba59b Introduce least-outstanding-connections load balancing (#282)
Least outstanding connections load balancing can improve the load distribution between instances but for Pgcat it may also improve handling slow replicas that don't go completely down. With LoC, traffic will quickly move away from the slow replica without waiting for the replica to be banned.

If all replicas slow down equally (due to a bad query that is hitting all replicas), the algorithm will degenerate to Random Load Balancing (which is what we had in Pgcat until today).

This may also allow Pgcat to accommodate pools with differently-sized replicas.
2023-01-17 06:52:18 -06:00
zainkabani
19f635881a Don't send discard all when state is changed in transaction (#186)
* Don't send discard all when state is changed in transaction

* Remove unnecessary clone

* spelling

* Move transaction check to SET command

* Add test for set command in transaction

* type

* Update comments

* Update comments

* use moves instead of clones for initial message

* don't make message mutable

* Update unwrap

* but i'm not a wrapper

* Add set local test

* change continue
2022-10-13 19:33:12 -07:00
Mostafa Abdelraouf
3d33ccf4b0 Fix maxwait metric (#183)
Max wait was being reported as 0 after #159

This PR fixes that and adds test
2022-10-05 21:41:09 -05:00
Mostafa Abdelraouf
af064ef447 Set client state to idle after error (#179)
* Set client state to idle after error

* fmt

* spelling

* clean up
2022-09-24 09:09:15 -07:00
Mostafa Abdelraouf
f7a951745c Report Query times (#166)
* Report avg and total query timing

* Report query times

* fmt
2022-09-15 02:21:45 -04:00
Mostafa Abdelraouf
4ae1bc8d32 Add SHOW CLIENTS / SHOW SERVERS + Stats refactor and tests (#159)
* wip

* Main Thread Panic when swarmed with clients

* fix

* fix

* 1024

* fix

* remove test

* Add SHOW CLIENTS

* revert

* fmt

* Refactor + tests

* fmt

* add test

* Add SHOW SERVERS + Make PR unreviewable

* prometheus

* add state to clients and servers

* fmt

* Add application_name to server stats

* Add tests for waiting clients

* Docs

* remove comment

* comments

* typo

* cleanup

* CI
2022-09-14 11:20:41 -04:00
Mostafa Abdelraouf
9514b3b2d1 Clean connection state up after protocol named prepared statement (#163)
* Clean connection state up after protocol named prepared statement

* Avoid cloning + add test

* fmt
2022-09-07 20:37:17 -07:00
Lev Kokotov
6d41640ea9 Send signal even if process is gone (#162)
* Send signal even if process is gone

* hmm

* hmm
2022-09-07 09:22:52 -07:00
Mostafa Abdelraouf
23a642f4a4 Send DISCARD ALL even if client is not in transaction (#152)
* Send DISCARD ALL even if client is not in transaction

* fmt

* Added tests + avoided sending extra discard all

* Adds set name logic to beginning of handle client

* fmt

* refactor dead code handling

* Refactor reading command tag

* remove unnecessary trim

* Removing debugging statement

* typo

* typo{

* documentation

* edit text

* un-unwrap

* run ci

* run ci

Co-authored-by: Zain Kabani <zain.kabani@instacart.com>
2022-09-01 20:06:55 -07:00
Mostafa Abdelraouf
65b69b46d2 Allow running integration tests with coverage locally (#151) 2022-08-30 10:43:45 -07:00
Mostafa Abdelraouf
d48c04a7fb Ruby integration tests (#147)
* Ruby integration tests

* forgot a file

* refactor

* refactoring

* more refactoring

* remove config helper

* try multiple databases

* fix

* more databases

* Use pg stats

* ports

* speed

* Fix tests

* preload library

* comment
2022-08-30 09:14:53 -07:00
Lev Kokotov
9d84d6f131 Graceful shutdown and refactor (#144)
* Graceful shutdown and refactor

* ok

* _Graceful_ shutdown

* Remove hardcoded setting

* clean up

* end

* timeout

* hmm

* hmm!

* bash

* bash

* hmm

* maybe maybe

* Adds tests and move non-admin connection rejection to startup (#145)

* Move error response

* Adds tests and removes unused variable

* Adds debug log

Co-authored-by: zainkabani <77307340+zainkabani@users.noreply.github.com>
2022-08-25 06:40:56 -07:00
Mostafa Abdelraouf
c054ff068d Avoid sending Z packet in the middle of extended protocol packet sequence if we fail to get connection from pool (#137)
* Failing test

* maybe

* try fail

* try

* add message

* pool size

* correct user

* more

* debug

* try fix

* see stdout

* stick?

* fix configs

* modify

* types

* m

* maybe

* make tests idempotent

* hopefully fails

* Add client fix

* revert pgcat.toml change

* Fix tests
2022-08-23 11:02:23 -07:00
zainkabani
65c32ad9fb Validates pgcat is closed after shutdown python tests (#116)
* Validates pgcat is closed after shutdown python tests

* Fix pgrep logic

* Moves sigterm step to after cleanup to decouple

* Replace subprocess with os.system for running pgcat
2022-08-09 14:09:53 -07:00
Mostafa Abdelraouf
7592339092 Prevent clients from sticking to old pools after config update (#113)
* Re-acquire pool at the beginning of Protocol loop

* Fix query router + add tests for recycling behavior
2022-08-09 12:18:27 -07:00
zainkabani
3719c22322 Implementing graceful shutdown (#105)
* Initial commit for graceful shutdown

* fmt

* Add .vscode to gitignore

* Updates shutdown logic to use channels

* fmt

* fmt

* Adds shutdown timeout

* Fmt and updates tomls

* Updates readme

* fmt and updates log levels

* Update python tests to test shutdown

* merge changes

* Rename listener rx and update bash to be in line with master

* Update python test bash script ordering

* Adds error response message before shutdown

* Add details on shutdown event loop

* Fixes response length for error

* Adds handler for sigterm

* Uses ready for query function and fixes number of bytes

* fmt
2022-08-08 16:01:24 -07:00
Mostafa Abdelraouf
106ebee71c Fix local dev (#112)
* Fix Dev env

* Update tests/sharding/query_routing_setup.sql

* Update tests/sharding/query_routing_setup.sql

* bring pgcat.toml on ci and local dev to parity

* more parity

* pool names

* pool names

* less diff

* fix tests

* fmt

* add other user to setup

Co-authored-by: Lev Kokotov <levkk@users.noreply.github.com>
2022-08-08 13:15:48 -07:00
Mostafa Abdelraouf
5ac85eaadd Fix Python tests and remove CircleCI-specific path (#106)
* Remove CircleCI-specific path in tests

* ..?

* Fix testsP

* Fix python test

* remove pip

* Maybe fail?

* return code?

* no &

* Fix tests
2022-08-02 15:52:22 -07:00
Mostafa Abdelraouf
1b648ca00e Send proper server parameters to clients using admin db (#103)
* Send proper server parameters to clients using admin db

* clean up

* fix python test

* build

* Add python

* missing &

* debug ls

* fix tests

* fix tests

* fix

* Fix warning

* Address comments
2022-07-31 19:52:23 -07:00
Mostafa Abdelraouf
2ae4b438e3 Add support for multi-database / multi-user pools (#96)
* Add support for multi-database / multi-user pools

* Nothing

* cargo fmt

* CI

* remove test users

* rename pool

* Update tests to use admin user/pass

* more fixes

* Revert bad change

* Use PGDATABASE env var

* send server info in case of admin
2022-07-27 19:47:55 -07:00
dependabot[bot]
eff8e3e229 Bump activerecord from 7.0.2.2 to 7.0.3.1 in /tests/ruby (#94)
Bumps [activerecord](https://github.com/rails/rails) from 7.0.2.2 to 7.0.3.1.
- [Release notes](https://github.com/rails/rails/releases)
- [Changelog](https://github.com/rails/rails/blob/v7.0.3.1/activerecord/CHANGELOG.md)
- [Commits](https://github.com/rails/rails/compare/v7.0.2.2...v7.0.3.1)

---
updated-dependencies:
- dependency-name: activerecord
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-12 13:24:41 -07:00
Lev Kokotov
b93303eb83 Live reloading entire config and bug fixes (#84)
* Support reloading the entire config (including sharding logic) without restart.

* Fix bug incorrectly handing error reporting when the shard is set incorrectly via SET SHARD TO command.
selected wrong shard and the connection keep reporting fatal #80.

* Fix total_received and avg_recv admin database statistics.

* Enabling the query parser by default.

* More tests.
2022-06-24 14:52:38 -07:00
Lev Kokotov
37e3a86881 Pass application_name to server (#73)
* Pass application_name to server

* fmt
2022-06-03 00:15:50 -07:00
Lev Kokotov
ccbca66e7a Poorly behaved client fix (#65)
* Poorly behaved client fix

* yes officer

* fix tests

* no useless rescue

* Looks ok
2022-05-09 09:09:22 -07:00
Lev Kokotov
35828a0a8c Per-shard statistics (#57)
* per shard stats

* aight

* cleaner

* fix show lists

* comments

* more friendly

* case-insensitive

* test all shards

* ok

* HUH?
2022-03-04 17:04:27 -08:00
Lev Kokotov
303fec063b Ruby (#30)
* cop

* log
2022-02-20 23:33:04 -08:00
Lev Kokotov
a556ec1c43 More query router commands; settings last until changed again; docs (#25)
* readme

* touch up docs

* stuff

* refactor query router

* remove unused

* less verbose

* docs

* no link

* method rename
2022-02-19 08:57:24 -08:00
Lev Kokotov
bbacb9cf01 Explicit shard selection; Rails tests (#24)
* Explicit shard selection; Rails tests

* try running ruby tests

* try without lockfile

* aha

* ok
2022-02-18 09:43:07 -08:00
Lev Kokotov
4c8a3987fe Refactor query routing into its own module (#22)
* Refactor query routing into its own module

* commments; tests; dead code

* error message

* safer startup

* hm

* dont have to be public

* wow

* fix ci

* ok

* nl

* no more silent errors
2022-02-16 22:52:11 -08:00
Lev Kokotov
c1476d29da config tests 2022-02-10 09:07:10 -08:00
Lev Kokotov
daf120aeac more tests 2022-02-10 08:35:25 -08:00
Lev Kokotov
fccfb40258 nl 2022-02-09 21:20:20 -08:00
Lev Kokotov
a9b2a41a9b fixes to the banlist 2022-02-09 21:19:14 -08:00
Lev Kokotov
dfc05c3dca sharding readme 2022-02-08 17:15:35 -08:00
Lev Kokotov
9657256adf fixed health check; sharding setup and tests 2022-02-08 15:48:28 -08:00
Lev Kokotov
39a9dff26b pg sharding works 2022-02-08 11:10:28 -08:00
Lev Kokotov
f8d6bd8d89 rename 2022-02-06 11:22:09 -08:00
Lev Kokotov
05250d9374 fix 2022-02-03 18:13:36 -08:00
Lev Kokotov
6e83556867 Ruby tests 2022-02-03 18:08:51 -08:00
Lev Kokotov
83daaf92d1 Less dirty servers & fix python 2022-02-03 18:02:50 -08:00