• Don’t Forget The Human Factor

    Date: 2009.09.07 | Category: netbooks, setup, tech•ed 2009 | Tags:

    Netbooks ready for collection

    I woke up at 4:00am this morning in a cold sweat. And I haven’t slept since. In the wee hours of the morning my brain, as is its want, was mulling a worst-case scenario: “What if we got the image or imaging wrong. How are we ever going to re-image these 2575 machines?”

    In our planning, the imaging process was always date constrained. Whilst David Haysom had planned an extra 2 days, just in case, as there are always strange things that could bite you on the bum.

    First day of Netbook handout. Today. 9am. Within an hour, we had handed out about 10% of the total.

    And a problem appeared. Thankfully not hardware; the opening and recharging of each machine tickled out any weird hardware issues. In fact, our out-of-box hardware failure rate was half of the planned amount. HP repaired these quickly, and all was well.

    Boot to nothing. Ouch!

    The problem: missing images on a low, single percentage number of machines.

    After reviewing our WDS/MDT2010 imaging server and technique by local and Redmond experts: this was deemed not the root cause for the missing images. Another possibility was Netbooks booting in the box, running out of power, and getting into a weird state. Again, this was discounted as there was just no data on the drives.

    The Current Theory

    The imaging process ends with the Netbook shutting down. From F12 to network boot to shutdown is one keypress: F12 to “boot from network”. At the completion of the multi-cast and unpacking in the WDS/MDT process, the Netbook shuts itself down to a black screen.

    On the first two days of imaging last week, the imaging area had power issues: essentially, fluctuating power at the setup trestle tables. This caused machines to be left powered off. To a black screen.

    Our thinking is that the black screen at the end of this process led to false-positives. Netbooks that had power issues, left on a black screen, were accidentally deemed “imaged” during the first few days and packed up.

    Learning 1:  Leave setup machines with a static, on-state saying “OK” would be our recommendation if you are planning such a large, time constrained install.

    The QA process we had during the imaging week was to test every 80 to 100 machines: that is, go through the out of box experience to test build quality.

    Learning 2: factor in a slightly higher rate of QA checks than you think necessary.

    How are we Fixing this?

    Handing over a known good, working Windows 7 Netbook to each attendee is a must. No compromise. Therefore, today, we instituted a quickly developed solution to freshly image these machines.

    After re-evaluation this evening, we are dedicating a team to process any machines with missing images. There is also another, faster process that is being investigated that may be implemented (via USB imaging)

    So, worst case: we have upset attendees whilst we fix their machines. All I can say is “Sorry, and we are making it right” and I am confident this process will work tomorrow. With a little patience, all will be well.

    Best case: our current glass-half-full thinking is that the machines that were handed out today were from the early in the week batch: where we had power issues, and black screen false-positives.

    Tomorrow will tell.

    Thanks for your patience and understanding.

    And if you want to complain or vent at someone: find me.