Not that I get tapped for referee work all the time, but it happens often enough that I decided to figure out a strategy for ensuring that whenever I do get tapped, I get the work done in a timely manner. Since the strategy I've developed has been at least somewhat effective so far, I figured I would share it in case anyone else is interested.
The problem with the work I need to do as a referee is that, pretty much perpetually, I am going to have a lot of other academic work of my own on my plate, much of which can legitimately be prioritized over the task of refereeing. What this means is that finding time to referee is a project doomed to failure. My other work will usually just expand to fill up all of my work hours. Instead, I realized I need to plan time to referee. So, on the model of office hours, I've designated two hours a week (10:00 am to Noon on Thursdays, if you are curious) as referee hours. And, much like office hours, this is simply a block of time during which I impose a temporary shift in my priorities. If there is refereeing work on my plate, it is (barring unusual circumstances and major deadlines), the primary thing to do during those two hours. And, just as I find myself using my office hours to do my own work if no students come in to see me, I can similarly spend the referee hours doing my own work if I have no outstanding referee requests.
So far, I have been surprised at how effective of a way this is to keep the refereeing I've agreed to do from falling off my radar.