feat: lift the client's concurrency ceiling from ~15 peers to 1024
The client had never been driven at scale. A stress harness that mixes searches, downloads, uploads and browses against a local soulfind found four ceilings, three of which lose work silently rather than merely being slow. Peers were capped at one per core. ActorSystem ran each actor's whole lifetime as one job on a fixed-size ThreadPool, and actor loops block until stopped, so the pool acted as a semaphore: past the cap an actor never started, never connected and never errored, and uploads stopped working entirely. The pool had exactly one consumer and every job it received was non-terminating, so it is gone; actors get their own thread. TransferResponse raced the bookkeeping it depends on. It was sent straight back to the peer while UpdateDownloadTokens was still queued for the client loop — but the response is what invites the peer's file connection, and that connection is matched by token. Under load the client loop lost, the file connection matched nothing, and the download sat on Queued forever. The response now goes out after the token is recorded. Download tokens were md5(filename) truncated to 20 bits, and the store deletes every entry matching a token, so two downloads sharing one destroyed each other — which fetching the same filename from two peers does every time. They now come from a counter. Port 0 from GetPeerAddress means the server does not know where a user listens. It was cached and dialled, and the upload waiting on it was dropped. It is no longer cached or served against; the connect attempt stays, because its failure is what makes an unreachable peer fall back to server brokering. Mixed profile at 1024 peers: 512/512 downloads, 256/256 uploads, 400-850 MiB/s in ~7s, against 71/96 downloads at 7.9 MiB/s and a 60s timeout before.
This commit is contained in:
Michel de Graaf
@@ -3,7 +3,6 @@ pub mod logger;
|
||||
pub mod lock;
|
||||
pub mod md5;
|
||||
pub mod path;
|
||||
pub mod thread_pool;
|
||||
pub mod zlib;
|
||||
|
||||
// Re-export commonly used items
|
||||
|
||||
@@ -1,124 +0,0 @@
|
||||
use std::sync::{Arc, Mutex, mpsc};
|
||||
use std::thread;
|
||||
|
||||
use crate::utils::lock::MutexExt;
|
||||
|
||||
type Job = Box<dyn FnOnce() + Send + 'static>;
|
||||
|
||||
enum Message {
|
||||
NewJob(Job),
|
||||
Terminate,
|
||||
}
|
||||
|
||||
pub struct ThreadPool {
|
||||
workers: Vec<Worker>,
|
||||
sender: Option<mpsc::Sender<Message>>,
|
||||
}
|
||||
|
||||
impl ThreadPool {
|
||||
#[must_use]
|
||||
pub fn new(size: usize) -> Self {
|
||||
assert!(size > 0);
|
||||
|
||||
let (sender, receiver) = mpsc::channel();
|
||||
let receiver = Arc::new(Mutex::new(receiver));
|
||||
let mut workers = Vec::with_capacity(size);
|
||||
|
||||
for _ in 0..size {
|
||||
workers.push(Worker::new(Arc::clone(&receiver)));
|
||||
}
|
||||
|
||||
Self {
|
||||
workers,
|
||||
sender: Some(sender),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn execute<F>(&self, f: F)
|
||||
where
|
||||
F: FnOnce() + Send + 'static,
|
||||
{
|
||||
let job = Box::new(f);
|
||||
if let Some(ref sender) = self.sender {
|
||||
let _ = sender.send(Message::NewJob(job));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for ThreadPool {
|
||||
fn drop(&mut self) {
|
||||
if let Some(sender) = self.sender.take() {
|
||||
for _ in &self.workers {
|
||||
let _ = sender.send(Message::Terminate);
|
||||
}
|
||||
}
|
||||
|
||||
// Ignore join errors: a worker whose job panicked returns Err here, but
|
||||
// Drop must never itself panic (that would abort the process).
|
||||
for worker in &mut self.workers {
|
||||
if let Some(thread) = worker.thread.take() {
|
||||
let _ = thread.join();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
struct Worker {
|
||||
thread: Option<thread::JoinHandle<()>>,
|
||||
}
|
||||
|
||||
impl Worker {
|
||||
fn new(receiver: Arc<Mutex<mpsc::Receiver<Message>>>) -> Self {
|
||||
let thread = thread::spawn(move || {
|
||||
loop {
|
||||
let message = match receiver.lock_safe() {
|
||||
Ok(rx) => rx.recv(),
|
||||
Err(_) => break,
|
||||
};
|
||||
match message {
|
||||
// Contain a panicking job so it kills only that job, not the
|
||||
// worker: the pool has a fixed number of workers and a lost
|
||||
// one is never replaced. The lock is already released here,
|
||||
// so a panic cannot poison the shared receiver.
|
||||
Ok(Message::NewJob(job)) => {
|
||||
let _ = std::panic::catch_unwind(
|
||||
std::panic::AssertUnwindSafe(job),
|
||||
);
|
||||
}
|
||||
Ok(Message::Terminate) | Err(_) => break,
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
Self {
|
||||
thread: Some(thread),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::ThreadPool;
|
||||
use std::sync::mpsc;
|
||||
use std::time::Duration;
|
||||
|
||||
// A single panicking job must not permanently kill its worker: subsequent
|
||||
// jobs still need to run. Otherwise one malformed network message could
|
||||
// leak a pool worker until the whole pool is exhausted (a DoS).
|
||||
#[test]
|
||||
fn pool_survives_a_panicking_job() {
|
||||
let pool = ThreadPool::new(1);
|
||||
pool.execute(|| panic!("boom"));
|
||||
|
||||
let (tx, rx) = mpsc::channel();
|
||||
pool.execute(move || {
|
||||
let _ = tx.send(42);
|
||||
});
|
||||
|
||||
assert_eq!(
|
||||
rx.recv_timeout(Duration::from_secs(5)).ok(),
|
||||
Some(42),
|
||||
"worker died after a panicking job"
|
||||
);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user