I have been reading a lot online about this (and there is a lot out there) and have been struck by what appears to be 2 common threads/assumptions. The first is that all of these articles seem to regard a data center as being some uniform ‘thing’ as in a data center is a data center is a data center. So the first point I want to make is that this is patently ridiculous and a uniform approach is therefore ridiculous as well. Someone decommissions NSA data centers and I can assure you with 100% certainty that this warrants a completely different process to, say, shutting down a ‘pod’ within a data center that is no longer needed because the corporate owner has migrated to the cloud. Admittedly this is a rather extreme example but it does provide a backdrop to illustrate the 7 main variables you should consider at the outset. These are all on a sliding scale as opposed to a yes or a no. Also, although these may appear obvious they need stating because they are used later in this article.
The second common thread/assumption seems to be that there can be a single ‘Plan of Action’, a checklist that can be applicable to all decommissioning projects. Well, yes there is, it’s the one used when decommissioning NSA or CIA server farms holding secret data. Here are the 7 S’s.
- Security. As in Data Security. This is always a major consideration but not every decommissioning project needs to be treated as though it’s the NSA. If you fall under HIPPA or Sarbanes Oxley though you’ll be right up there in certain aspects.
- Size. Clearly decommissioning an entire 2 million square foot secure facility is different from a ‘pod’ within a secure facility which is different to a server room at a corporate HQ. Note that the word ‘secure’ is mentioned twice under the heading of ‘size’ and yes, there is a correlation.
- Sensitivity. Is this the server room at corporate HQ that has already been 100% migrated to the cloud or are the assets to be decommissioned mixed in with assets still in production?
- Sophistication. This is actually 2 considerations. How reliable is your documentation on current assets going in (Physical and Logical data models, for example) and what level of documentation do you need during and after completion? An important note here is that in my experience there is a large disparity in what is perceived as ‘good’ documentation.
- Sequence. If the last step is turn off the lights and shut the door on the way out, what goes on before that, when and in what order? You have to have a plan, the complexity of which is driven by the bullet points above.
- Saleability. Once decommissioned is this equipment worth money and if so how much? Someone somewhere is going to have to fund the project and if it can be revenue from remarketing that’s a clear win. Note to the wise** In my experience even a small pod containing about 3 year old equipment can be worth around $70K in resale value.
- Software. Depending on the sophistication of the plan you are going to need at least 2 software components, some sort of GANTT chart and something for identifying and tracking the assets (and components) through to final disposition whether that be dismantling or resale. GANTT software is easily available but software specifically designed for data center decommissioning is pretty well limited to a single application, ours. This has been in development over many cycles for nearly 20 years and it covers everything and I mean everything. It is called ROMS (Recycling Operations Management System), it is web based and it is free to use for all of our customers. Call us any time and we’d be happy to walk you through a demonstration.
So, I’ve covered the high level considerations now we move on to a discussion of the various stages. This is going to be a 30,000’ high view because the possible range of project scope is literally infinite. We’ll take a single pod in a large 2MM sq ft secure facility. Some of the assets will remain in production, others are to be decommissioned to make space for newer units. You fall under HIPPA so you need to be able to prove compliance. The retiring equipment is estimated to have considerable but as yet undetermined value. You do have limited IT personnel resources nearby.
- Select a Vendor. Could you do this ‘in-house’? Probably yes but the real question is should you? Firstly, you fall under HIPPA; do you really want to retain 100% of the potential liability here just to save a few bucks? A better idea is to pass 100% of that potential liability to someone else who is R2 certified and insured up to the hilt. Certainly supervise and monitor but that should be it. The equipment has considerable potential value as well. As you fall under HIPPA the chances are that a core competence of yours does not include refurbishing, testing and remarketing used equipment. Do you want your IT people to be running an Ebay or Amazon or RecycleSoft account? Probably not so select an R2 certified Ewaste processor who has years of experience in this. Look for a vendor who can provide references for projects similar to yours. Following an initial inquiry you should ask for an ‘Upstream Information Packet’ which should contain, among other things, all certifications and insurance docs. You should not be considering costs at this point, you should shortlist 2 candidate downstream vendors who you regard as broadly capable. Now compare things like reporting capabilities, references and yes, costs too now.
- Project Plan. Working with the vendor put a project plan together. The chances are that your vendor has done a similar project at some point so they should do most of the work here but you’ll need to give final approval. It is important to note that the project plan is a live document and is subject to change even during implementation. Things come up. I once had a project where we noticed cat5 cables going into a wall and on the other end into a switch. We followed the cable which led to a single Pentium 3 era 6U server, still humming away in a locked and hot cupboard but nobody knew what it did and it wasn’t on any documentation. We turned it off and found that everyone’s access cards had stopped working. It was an old DOS security system that was used by only 2 people.
- Identify Assets. You need to have a clear understanding of everything in that pod in a logical and physical data model. It may turn out that one of the items you were thinking of retiring actually is hosting some middleware used by something that was to remain in production. When you start unplugging things is not the time to learn what you’re really unplugging. Remember the ‘Sophistication’ bullet point? If I had a dollar for every time I was given a spreadsheet with a list of assets which turned out to be very different to the actual assets it would easily pay for an expensive dinner for two. Our software ROMS is a very good tool for hosting a complete inventory if you don’t have one already. The project plan should clearly state to what category of equipment should be inventoried and with what information. For instance you may not need to include battery backups and cables but need serial, manufacturer, model number of everything else.
- Identify Assets Again. Before you start tagging units for removal (bright red & green labels work well) it never hurts to recheck.
- Power Down. Don’t do this on the same day as removal, wait at least a day. This way, if suddenly someone starts complaining that their application is not working you still have the option of turning everything back on.
- Inventory Hard Drives. Capture the serials, size and manufacturer of all hard drives AS THEY ARE REMOVED. ROMS is a good tool for this. The project plan will need to specify the destruction method which can range from onsite physical destruction to NAID compliant data erasure. ROMS will generate the appropriate certificates of data destruction.
- Security. Now is a good time to mention the other security consideration. Data Centers are very secure environments in a physical sense. Many have multiple security layers; some even go as far as requiring background checks on everyone entering the building.The project plan should clearly identify all security checkboxes are checked otherwise you may not even be granted entry on day one.
- Pack & Ship. Your role is almost complete but this is where your vendor is going to really start adding value. If they know what they’re doing their sort process will have already begun. Each E waste processor has their own ‘Tech-cut’ which is a value below which an item is not worth more than scrap value. Take a 7th generation Dell 1U server. The tech-cut for a server may be, say, $100. The tech-cut for that Cisco Switch over there may be, say, $1000 if grade A. This means no scratches, damaged ports etc so packing is important. It also means it devalues faster so it needs to be worked on before lesser value items.
- Inventory (2). It is sometimes not possible to have a complete inventory prior to removal for various reasons. Serial numbers are often not visible before the equipment is removed from it’s housing. Once back at your vendor’s facility a full inventory should be completed consistent with the guidelines in the project plan. If done using ROMS you will have complete online access to this.
- Test or Dismantle. Most people can sell something as-is on a platform such as Ebay. An R2 Certified recycler cannot. We have to be able to prove to an auditor what item was tested, by whom, when, what exact test criteria we used and if the item passed or failed in these. Depending on the test results ROMS then assigns an R2 category to that item. Let’s take that Dell 1U server from earlier. We might know from experience that it’s value as a whole unit may be just above the tech-cut of $100 but shipping could eat up $60 of that leaving $40. However, it might have 16 x 16GB sticks of memory and twin CPU’s which are inexpensive to ship. So it is likely in this instance that just the memory & cpu’s would be taken into stock and tested whilst the server body would be sold to a downstream processor.
- Sell. We have ready buyers for 95% of everything we process. This is important because used electronics devalue extremely quickly. The other 5% is remarketed with a warranty and accompanied by the test criteria and results. That, along with our 100% positive feedback means we get the best possible price for your equipment. Regardless of the sales channel, through ROMS you are able to see what was sold, when, to whom and the sale amount. If sold online there is even a link to the listing itself. We believe in complete transparency.
- A thing to be wary of. Depending on your choice of vendor, the project plan (or another document) may contain a section that clearly describes the costs associated with each aspect of the above in detail. For instance, the charge to generate each data destruction certificate (remember, the charge includes the fact that your recycler is taking on 100% of the potential liability should there be a HIPPA violation). There may be an hourly labor charge, trucking, adding items to inventory and a host of other nickel and dime type items. I have seen thousands upon thousands of dollars eaten up by this widely used technique. At R3Ewaste we use a different approach.
- Share.We will do any size data center decommissioning project with no line item charges whatsoever. What we do is to assign a third of whatever revenue we generate through resale to costs; this covers everything from data certs to labor, trucking, reports, everything. Then there is an agreed formula on the distribution of the balance, the percentage of which goes up for you the more revenue is generated.
Decommissioning a Data center is not rocket surgery, it merely requires planning to a level of detail commensurate with the size and nature of the project. Always keep in mind that this is your project with someone else’s help, not the other way round.
Please feel free to reach out should you have any questions or need to discuss a specific project. My cell is 602-317-3981 or by email at Graham@R3Ewaste.com.