Multithreading – how to avoid threading?

I've recently read a lot about how writing multithreaded applications is a great pain, and I've learned enough about the topic, at least to some extent, why

I've read that using functional programming techniques can help ease some pain, but I've never seen a simple example of concurrent function code So what are the alternatives to using threads? At least, there are ways to abstract them so that you don't have to think about locking and whether the objects of a specific library are thread safe

I know Google's MapReduce should solve the problem, but I haven't seen its concise explanation yet

Although I give a specific example below, I am more interested in general technology than solving this specific problem (using this example to help illustrate other technologies will help)

I asked this question when I wrote a simple web crawler as a learning exercise It works well, but it's slow Most bottlenecks come from downloading pages It is currently single threaded, so only one page is downloaded at a time Therefore, if you can download pages at the same time, even if the crawler runs on a single processor computer, it will greatly speed up I considered using threads to solve the problem, but they scared me Any suggestions on how to add concurrency to such problems without releasing terrible thread nightmares?

Solution

I'll add an example of how to use functional code to safely make code concurrent

Here are some of the code you might want to execute in parallel, so you don't have to wait for one file to complete to start downloading the next file:

void DownloadHTMLFiles(List<string> urls)
{
    foreach(string url in urls)
    {
         DownlaodOneFile(url);  //download html and save it to a file with a name based on the url - perhaps used for caching.
    }
}

If you have multiple files, users may spend a minute or more waiting for all files We can rewrite this code functionally like this. It is basically the same:

urls.ForEach(DownloadOneFile);

Note that this still runs in sequence However, it is not only shorter, we have gained an important advantage here Since each call to the downloadonefile function is completely isolated from other functions (available bandwidth is not a problem for our purposes), you can easily replace the foreach function with another very similar function: a function that starts each call to downloadonefile from a separate thread in the thread pool

It turns out Net can achieve this function by using parallel extensions Therefore, by using functional programming, you can change a line of code and suddenly have something running in parallel for sequential running This is very powerful