Have you ever needed to call a server-side script that processes 1,000s upon 1,000s of documents or rows of data? If you're doing this from a browser you're going to need to take a couple of things in to account:
- Web connections, Agents and scripts can timeout.
- Users need re-assuring feedback that something is actually happening.
If you make a website with a button, which, when clicked, calls a script that loops 10,000 plus document the likelihood is the browser will timeout or the user will get sick of looking at the "Waiting for server..." message, assume it's broken, and refresh the page or close the browser.
What would be better is having some visual way of showing progress to the user. Couple this with an approach that only processes a set amount of data per execution and you have a safe and user-friendly way of doing it.
The approach I came up with was to use a round-robin method of calling the back-end script multiple times, processing a set number of document each time, until the server reported back that all were processed.
I'll post a demo and some code examples later this week. For now I'll leave you with a grab of what I ended up with:
As you can probably tell, this is a Flex app. But, it doesn't really matter what technologies you use, either on the front or the backend. The approach can apply to anything.
Is it an approach commonly used though? I can't remember seeing it discussed or used anywhere else, but find it hard to believe I can be the first to discuss it.
Perhaps there are some pitfalls to doing it this way?