Commit Graph

196 Commits

Author SHA1 Message Date
Kurtsley
1c73889fb9 Add initial Windows support, ref #298 (#301) 2023-01-28 15:51:05 -08:00
Lev Kokotov
24e79dcf05 Startup improvements & PAUSE/RESUME (#300)
* Dont require servers to be online to start pooler

* PAUSE/RESUME

* fix

* Refresh pool

* Fixes

* lint
2023-01-28 15:36:35 -08:00
Lev Kokotov
2e3eb2663e Fix formatting (#299) 2023-01-28 09:17:49 -08:00
zainkabani
a0e740d30f Refactors is_banned logic and forces health check on unban (#288)
* Refactors is_banned logic and forces healthcheck on unban

* typo

* Make is banned log debug

* addressing comments

* Comment
2023-01-19 17:36:48 -08:00
Jose Fernández
c58f9557ae Add more metrics to prometheus endpoint (#263)
This change:
- Adds server metrics to prometheus endpoint.
- Adds database metrics to prometheus endpoint.
- Adds pools metrics to prometheus endpoint.
- Change metrics name to have a prefix of (stats|pools|databases|servers).
2023-01-19 07:48:12 -08:00
zainkabani
ca8901910c Removes message cloning operation required for query router (#285)
* Removes message cloning operation required for query router

* fmt

* flakey?

* ?
2023-01-19 07:19:49 -08:00
Mostafa Abdelraouf
87a771aecc Log error messages for network failures (#289)
We are seeing some Error reading message code from socket error messages, we want to get more context so this PR logs the actual error reported.
2023-01-19 05:18:08 -06:00
zainkabani
85ac3ef9a5 Buffer client CopyData messages (#284)
Buffers CopyData messages and removes buffer clone for the sync message
2023-01-17 17:39:55 -08:00
Mostafa Abdelraouf
7894bba59b Introduce least-outstanding-connections load balancing (#282)
Least outstanding connections load balancing can improve the load distribution between instances but for Pgcat it may also improve handling slow replicas that don't go completely down. With LoC, traffic will quickly move away from the slow replica without waiting for the replica to be banned.

If all replicas slow down equally (due to a bad query that is hitting all replicas), the algorithm will degenerate to Random Load Balancing (which is what we had in Pgcat until today).

This may also allow Pgcat to accommodate pools with differently-sized replicas.
2023-01-17 06:52:18 -06:00
zainkabani
ab0bad6da0 Write messages directly onto message buffer instead of allocating on own buffer (#283)
* initial commit

* comment

* fmt
2023-01-16 20:22:06 -08:00
zainkabani
8720ed3826 Buffer copy data messages (#265)
* Buffer copy data messages

* Update comment
2022-12-21 06:57:53 -08:00
Jose Fernández
9e8ef566c6 Allow setting the number of runtime workers to be used. (#258)
This change adds a new configuration parameter called `worker_threads` that
allows setting the number of workers the Tokio Runtime will use. It defaults to
4 to maintain backward compatibility.

Given that the config file parse is done asynchronously, first, a transient runtime
is created for reading config, and once it has been parsed, the actual runtime
that will be used for PgCat execution is created.
2022-12-16 11:13:13 -08:00
Jose Fernández
99247f7c88 Allow setting idle_timeout for server connections. (#257)
In postgres, you can specify an `idle_session_timeout` which will close
sessions idling for that amount of time. If a session is closed because
of a timeout, PgCat will erroneously mark the server as unhealthy as the next
health check will return an error because the connection was drop, if no
health check is to be executed, it will simply fail trying to send the query
to the server for the same reason, the conn was drop.

Given that bb8 allows configuring an idle_timeout for pools, it would be
nice to allow setting this parameter in the config file, this way you can
set it to something shorter than the server one. Also, server pool will be kept
smaller in moments of less traffic. Actually, currently this value is set as its
default in bb8, which is 10 minutes.

This changes allows setting the parameter using the config file. It can be set both
globally and per pool. When creating the pool, if the pool don't have it defined, global
value is used.
2022-12-16 08:01:00 -08:00
zainkabani
c62b86f4e6 Adds details to errors and fixes error propagation bug (#239) 2022-11-17 09:24:39 -08:00
zainkabani
fcd2cae4e1 Move get_config in startup to admin branch to scope down usage (#238) 2022-11-17 09:22:12 -08:00
zainkabani
5145b20e02 Move ClientBadStartup error log to debug (#237) 2022-11-16 22:16:16 -08:00
zainkabani
fe0b012832 Adds configuration for logging connections and removes get_config from entrypoint (#236)
* Adds configuration for logging connections and removes get_config from entrypoint

* typo

* rename connection config var and add to toml files

* update config log

* fmt
2022-11-16 22:15:47 -08:00
zainkabani
0c96156dae Adds health check setting to pool and avoids get_config in hotpath (#235)
* Adds healthcheck settings to pool

* fmt

* Fix test
2022-11-16 18:51:15 -08:00
zainkabani
b7e70b885c Default to using username when database isn't present on startup (#234) 2022-11-16 18:49:04 -08:00
Cluas
dfa26ec6f8 chore: make clippy lint happy (#225)
* chore: make clippy happy

* chore: cargo fmt

* chore: cargo fmt
2022-11-09 10:04:31 -08:00
Pradeep Chhetri
63d4431046 Fix for warnings about avg_errors not implemented (#220) 2022-11-02 08:11:47 -07:00
Lev Kokotov
9fe8d5e76f Dont change shard unless you know (#195) 2022-10-26 00:14:08 -07:00
Lev Kokotov
0524787d31 Automatic sharding: part one of many (#194)
Starting automatic sharding
2022-10-25 11:47:41 -07:00
Lev Kokotov
dea952e4ca Re-enable query parser and parse multiple statements (#191)
* Re-enable query parser and parse multiple statements

* no diff
2022-10-23 16:59:51 -07:00
zainkabani
19f635881a Don't send discard all when state is changed in transaction (#186)
* Don't send discard all when state is changed in transaction

* Remove unnecessary clone

* spelling

* Move transaction check to SET command

* Add test for set command in transaction

* type

* Update comments

* Update comments

* use moves instead of clones for initial message

* don't make message mutable

* Update unwrap

* but i'm not a wrapper

* Add set local test

* change continue
2022-10-13 19:33:12 -07:00
Mostafa Abdelraouf
eceb7f092e Use Jemalloc (#189)
Jemalloc performs better than the standard allocator in various metrics (http://ithare.com/testing-memory-allocators-ptmalloc2-tcmalloc-hoard-jemalloc-while-trying-to-simulate-real-world-loads/).

This PR makes changes to use Jemalloc as the global allocator for Pgcat. Windows is not officially supported by Pgcat but it should still compile but without Jemalloc as the allocator.
2022-10-13 11:13:45 -05:00
Mostafa Abdelraouf
83fd639918 A bit faster get_pool (#187)
* A bit faster get_pool

* fmt
2022-10-08 08:16:04 -07:00
Mostafa Abdelraouf
3d33ccf4b0 Fix maxwait metric (#183)
Max wait was being reported as 0 after #159

This PR fixes that and adds test
2022-10-05 21:41:09 -05:00
Lev Kokotov
7987c5ffad Replace a few types with more developer-friendly names (#182)
* Replace a few types with more developer-friendly names

* UserPool -> PoolIdentifier
2022-10-01 10:25:59 -07:00
zainkabani
24f5eec3ea Change sharding config to enum and move validation of configs into public functions (#178)
Moves config validation to own functions to enable tools to use them
Moves sharding config to enum
Makes defaults public
Make connect_timeout on pool and option which is overwritten by general connect_timeout
2022-09-28 08:50:14 -05:00
Mostafa Abdelraouf
af064ef447 Set client state to idle after error (#179)
* Set client state to idle after error

* fmt

* spelling

* clean up
2022-09-24 09:09:15 -07:00
Lev Kokotov
19fd677891 Fix the pool fix (#176)
* Always listen to the compiler

* Its fine
2022-09-23 12:06:07 -07:00
Lev Kokotov
964a5e1708 Don't drop connections if DB hasn't changed (#175)
* Don't drop connections if DB hasn't changed

* Incoporate connect_timeout into the pool config

* use the field
2022-09-23 11:32:05 -07:00
Mostafa Abdelraouf
d126c7424d Log failed client logins (#173)
* Log failed client logins

* more logging

* remove clones

* remove
2022-09-23 09:08:38 -07:00
zainkabani
f72dac420b Add defaults for configs (#174)
* add statement timeout to readme

* Add defaults to various configs

* primary read enabled default to false
2022-09-22 23:00:46 -07:00
zainkabani
3a729bb75b Minor refactor for configs (#172)
* Changes shard struct to use vector of ServerConfig

* Adds to query router

* Change client disconnect with error message to warn instead of debug

* Add warning logs for clean up actions
2022-09-22 10:07:02 -07:00
zainkabani
85cc2f4147 Update to latest library versions (#170) 2022-09-21 13:48:33 -07:00
zainkabani
8c09ab6c20 Export pgcat objects in lib (#169)
* Export pgcat objects in lib

* fmt
2022-09-20 18:47:32 -07:00
Mostafa Abdelraouf
f7a951745c Report Query times (#166)
* Report avg and total query timing

* Report query times

* fmt
2022-09-15 02:21:45 -04:00
Mostafa Abdelraouf
4ae1bc8d32 Add SHOW CLIENTS / SHOW SERVERS + Stats refactor and tests (#159)
* wip

* Main Thread Panic when swarmed with clients

* fix

* fix

* 1024

* fix

* remove test

* Add SHOW CLIENTS

* revert

* fmt

* Refactor + tests

* fmt

* add test

* Add SHOW SERVERS + Make PR unreviewable

* prometheus

* add state to clients and servers

* fmt

* Add application_name to server stats

* Add tests for waiting clients

* Docs

* remove comment

* comments

* typo

* cleanup

* CI
2022-09-14 11:20:41 -04:00
Mostafa Abdelraouf
9514b3b2d1 Clean connection state up after protocol named prepared statement (#163)
* Clean connection state up after protocol named prepared statement

* Avoid cloning + add test

* fmt
2022-09-07 20:37:17 -07:00
Mostafa Abdelraouf
744ceada86 Better logging for failure to get connection from pool (#161) 2022-09-07 08:24:07 -07:00
Mostafa Abdelraouf
a5c8dd69b2 Avoid reporting ProtocolSyncError when admin session disconnects (#160)
* Avoid reporting ProtocolSyncError when admin session disconnects

* rebuild

* rebuild
2022-09-06 22:22:31 -07:00
zainkabani
6a9a4db648 Adds microsecond logging and also reformats duration to include milliseconds (#156)
* Adds microsecond logging and also reformats duration to include milliseconds

* fmt

* attempt to fix cd

* revert
2022-09-05 01:21:27 -07:00
Mostafa Abdelraouf
976b406468 Main Thread Panic when swarmed with clients (#158)
* Main Thread Panic when swarmed with clients

* fix

* fix

* 1024

* fix

* remove test

* Update src/client.rs

* Update src/main.rs

* Update src/client.rs

* Update src/main.rs

Co-authored-by: Lev Kokotov <levkk@users.noreply.github.com>
2022-09-05 01:21:06 -07:00
zain-kabani
417358c35d Patch graceful shutdown bug (#157)
* Fixes non-admin client counting error

* Add log when sigterm received and log number of active clients when shutdown timeout is reached
2022-09-05 01:02:49 -07:00
Mostafa Abdelraouf
23a642f4a4 Send DISCARD ALL even if client is not in transaction (#152)
* Send DISCARD ALL even if client is not in transaction

* fmt

* Added tests + avoided sending extra discard all

* Adds set name logic to beginning of handle client

* fmt

* refactor dead code handling

* Refactor reading command tag

* remove unnecessary trim

* Removing debugging statement

* typo

* typo{

* documentation

* edit text

* un-unwrap

* run ci

* run ci

Co-authored-by: Zain Kabani <zain.kabani@instacart.com>
2022-09-01 20:06:55 -07:00
Mostafa Abdelraouf
7f20dc3054 Better handling extended protocol messages in the event of busy pool (#155)
* Better handling for checkout errors during extended protocol messages

* Fix specs

* comment
2022-09-01 15:02:39 -07:00
Mostafa Abdelraouf
36339bd96f Log Address information in connection create/drop (#154)
* Log Address information in connection create/drop

* run ci
2022-09-01 11:16:22 -07:00
zainkabani
2628dec42e Move autoreloader to own tokio task (#148) 2022-08-29 00:08:44 -07:00