How to dynamically determine the number of goroutines to process tasks
I wrote a virtual code to prove the purpose
There are two channels and three goroutines in the code
1 goroutine generates numbers according to whether they can be divided by 100 without remainder:
>If the number can be divided by 100, push it to the first channel. > Otherwise, push it to the second channel
2 goroutines are consumers of these channels:
>1 goroutine is responsible for consuming figures 1... 99 – 101... 199, etc. > other goroutines are responsible for 100200300, etc
Obviously, one goroutine does 99 times more work than another How to deal with it in go? If goroutine does more work than others, will this goroutine have more CPU time? Or should I deal with this situation, such as creating 99 goroutines for more resource consuming channels? (for the sake of argument, these works are considered to be the same)
func main() { ch1 := make(chan int) ch2 := make(chan int) go generator(ch1,ch2) go handler(ch1) go handler2(ch2) time.Sleep(1*time.Second) } func generator(chan1,chan2 chan int){ for i:=0 ; ; i++{ if i%100 == 0{ chan1 <- i }else{ chan2 <- i } } } func handler(number chan int){ for _ = range number{ num := <- number fmt.Println("Number divided by 100 is 0. ",num) } } func handler2(number chan int){ for _ = range number{ num := <- number fmt.Println("Number divided by 100 is not 0. ",num) } }
Solution
How much CPU resources goroutine gets depends on many things
We can usually say that goroutines that deal only with numbers that can be scored 100 are likely to wait more than other numbers You don't have to worry about this. The elements on the waiting channel don't need CPU resources, so if you have enough "other" goroutines to execute jobs, they can use your CPU
For obvious reasons, your example is simple, but in real life, it is more profitable to abstract a task into a regular task (for example, processing any number may be a task), create and use a regular work pool, and send all tasks to the pool In this way, no matter how many goroutines there are in the pool, if there is work to do and there is a free (waiting) goroutine, it will undertake the task and make the best use of your CPU resources The job processor (performer) should know how to handle numbers of 100 or 101
For an example of how to implement such a goroutine pool, see is this an idiomatic worker thread pool in go?