Currently there are methods to get the length of the queue, and the number of jobs being processed. It's not currently possible to simultaneously get both of these.
In the autoscaler, we get one and then the other. This may not be an accurate total, since items could move in between calls, though realistically it's more than accurate enough. It does mean there are multiple backwards and forwards to the redis server, instead of a single pipeline though.
This becomes more of a problem when you're trying to shut down a queue when it's empty, since you can't really be 100% sure that it definitely is empty, it opens the door for a very occasional failure.
We should implement a length method that returns both these lengths atomically.
Currently there are methods to get the length of the queue, and the number of jobs being processed. It's not currently possible to simultaneously get both of these.
In the autoscaler, we get one and then the other. This may not be an accurate total, since items could move in between calls, though realistically it's more than accurate enough. It does mean there are multiple backwards and forwards to the redis server, instead of a single pipeline though.
This becomes more of a problem when you're trying to shut down a queue when it's empty, since you can't really be 100% sure that it definitely is empty, it opens the door for a very occasional failure.
We should implement a length method that returns both these lengths atomically.