Company wide reboots-how to handle?

[Russ Smith]

Long story short due to Sarbanes Oxley we have to do fairly frequent forced password changes, think it's every 6 months now. We have large numbers of computers in labs that are used by one user, so we'll have the case where john lab user will be logged in to 15-20 machines with the same login and password. When we do the forced password change, john lab user can't remember how many machines he's logged into, so we get constant problems where they screw up when they change their password and get locked out of numerous lab machines.

their solution is never log out of the lab machines, but apparently this causes IT another problem as they get bombarded with error messages related to a user being logged in with multiple passwords to the same account.

So we have given up trying to get the lab guys to do it themselves and we're being told(by our VP) to just go through and do a forced reboot so all those machines have to be logged back in.

Going forward I'm just wondering if anybody else has a similar issue and has come up with a better solution that avoids this whole mess? Otherwise in 6 months I'm going to be doing the same thing going lab bench to lab bench flipping the switch on the bench and then dealing with all the irate engineers who had some critical project delayed and insisting they never saw the repeated warnings that this was going to take place.

God I hate Sarbanes Oxley.

 

[Covert Rain]

Long story short due to Sarbanes Oxley we have to do fairly frequent forced password changes, think it's every 6 months now. We have large numbers of computers in labs that are used by one user, so we'll have the case where john lab user will be logged in to 15-20 machines with the same login and password. When we do the forced password change, john lab user can't remember how many machines he's logged into, so we get constant problems where they screw up when they change their password and get locked out of numerous lab machines.

their solution is never log out of the lab machines, but apparently this causes IT another problem as they get bombarded with error messages related to a user being logged in with multiple passwords to the same account.

So we have given up trying to get the lab guys to do it themselves and we're being told(by our VP) to just go through and do a forced reboot so all those machines have to be logged back in.

Going forward I'm just wondering if anybody else has a similar issue and has come up with a better solution that avoids this whole mess? Otherwise in 6 months I'm going to be doing the same thing going lab bench to lab bench flipping the switch on the bench and then dealing with all the irate engineers who had some critical project delayed and insisting they never saw the repeated warnings that this was going to take place.

God I hate Sarbanes Oxley.

When I was in support we were set up with profiles based on the type of users we were. IT created Power Users profile that had unlimited access in terms of how many machines they could be signed into at once. Anybody who was a Power User could sign onto as many machines as needed.

This was a huge help because sometimes we had to manually apply a patch on the floor and needed to sign into each machine to reimage. If not for that, we would have gone through something similar that your experiencing now.

 

[abomb]

When I was in support we were set up with profiles based on the type of users we were. IT created Power Users profile that had unlimited access in terms of how many machines they could be signed into at once. Anybody who was a Power User could sign onto as many machines as needed.

This was a huge help because sometime we had to manually apply a patch on the floor and needed to sign into each machine to reimage. If not for that, we would have gone through something similar that your experiencing now.

Best practices would tell you to create seperate test accounts for this clown to use. Logging into multiple machines with your primary id is stoopid.

 

[Ryanwb]

Does your company use active directory?

 

[Treefiddy]

Does your company use active directory?

Along the same lines, does your company use Microsoft products? If so and you have to go the restart route, you can setup a scheduled task on every computer to restart itself at the same time everyday. That would make it so you don't have to touch each computer everyday.

Just make sure that when you set it up, that you're able to use an administrator account that has a password that will never expire. Otherwise when their 6 months are up, the computers will stop restarting because the password will be different.

 

[Russ Smith]

Along the same lines, does your company use Microsoft products? If so and you have to go the restart route, you can setup a scheduled task on every computer to restart itself at the same time everyday. That would make it so you don't have to touch each computer everyday.

Just make sure that when you set it up, that you're able to use an administrator account that has a password that will never expire. Otherwise when their 6 months are up, the computers will stop restarting because the password will be different.

Don't know if we use active directory or not. Regarding auto reboots, the problem is they're doing long term testing (DSL chips mainly ) where doing that would literally disable our company, many of these tests have to run uninterrupted. We've given them a full weeks notice of the forced reboots upcoming and I'd bet anything we'll end up rebooting less than 50% of the machines. By then we'll have high level people screaming at our boss and giving him a list of machines that can't be rebooted under any circumstances.

That's why we end up we're we are. There's one guy in engineering who keeps complaining about all sorts of password issues with his machine in his cube and it's all connected to his being logged in with so many machines in the lab. He keeps complaining, IT keeps saying you have to reboot all those machines to resolve it, and he finally went to his boss who went to our boss and thus we are here.

It's a continuous loop, the guy causing the problem is the one complaining the loudest about it, he's the one refusing to reboot his machines, and he'll be the one screaming bloody murder when we actually try to reboot.

IT has the same issues with windows updates and antivirus updates the engineers disable those things on their lab pc's and that's why lab PC's account for damn near 100% of our internal viruses.

 

[Ryanwb]

I can't imagine a corporate environment not on AD unless you are in a Novel environment, but then you would have NDS. Any competant administrator could set account limitations to a single log in at a time.

 

[Russ Smith]

So the Friday at 6PM reboot has already been moved to next Tuesday at 8am to give them more time to prepare. I suggested the time change so I won't get stuck staying late for nothing.

We've already received 3 emails from engineering managers saying they're going to have several machines that can't go down under any circumstance, so it ought to be interesting since you can guess the machines causing the problems probably haven't been rebooted in ages and are the very ones they refuse to take down next Tuesday.

I'd be tempted to just throw the main breaker for the first floor in the electrical room but I'm always hesitant to do something like that you just never know if it will really come back on, and what equipment could be damaged when it does come back on.

 

[abomb]

We've already received 3 emails from engineering managers saying they're going to have several machines that can't go down under any circumstance, so it ought to be interesting since you can guess the machines causing the problems probably haven't been rebooted in ages and are the very ones they refuse to take down next Tuesday.

Just keep escalating it up the chain. You will eventually find someone who understands the importance of SOX compliance and will quite the engineering managers.

Why dont they build their test environment on a separate (non-Internet connected) network, with their own AD? Shoot, as long as they have documented why they dont allow for logout or password expiration, the SOX auditor shouldnt care, reguardless it if is in the current environment or a new one.

 

[Treefiddy]

Your network must be a hackers dream. No Anti-virus on computers that are never rebooted (leading me to believe they are behind in updates as well).

Stress the importance of computer security and possibly point out how the computers are wide open.

 

[abomb]

Your network must be a hackers dream. No Anti-virus on computers that are never rebooted (leading me to believe they are behind in updates as well).

Stress the importance of computer security and possibly point out how the computers are wide open.

Not to mention the mission-critical development being done on said machines.

 

[NavyVet]

Shoot, as long as they have documented why they dont allow for logout or password expiration, the SOX auditor shouldnt care, reguardless it if is in the current environment or a new one.

You are correct. I'm an auditor and when the exposure is documented, risk-assessed, and excepted, it becomes a non-issue. An auditor will ask for the overriding documentation and pass. Not everything can be cut and dry. Engineering labs are always the exception because they run long compiles, tests, simulators, etc. that can fall out of the blanket password change interval requirements. Typically this test equipment is not internet-facing, so there is limited or no risk.

 

[Russ Smith]

You are correct. I'm an auditor and when the exposure is documented, risk-assessed, and excepted, it becomes a non-issue. An auditor will ask for the overriding documentation and pass. Not everything can be cut and dry. Engineering labs are always the exception because they run long compiles, tests, simulators, etc. that can fall out of the blanket password change interval requirements. Typically this test equipment is not internet-facing, so there is limited or no risk.

Well we did the reboot this morning, essentially bench by bench. I guess there's lots of scopes and other equipment that are also windows based.

What is happening is when one guy logs in to multiple machines in the labs, he never logs out. Months later he is forced to change his password on his desktop due to Sox. When he does, all the other machines that he's logged into have a conflict. Apparently they essentially keep generating errors until he's locked out of his desktop, because they're still logged in under his old password.

So to stop him from getting locked out,we had to reboot, adn since he refused to do the reboot himself, my boss(the VP of Operations) just said reboot them give them warning, and reboot. I did it at 8am today.

We're getting hit with a few complaints so far, this didn't come up, this test was killed etc, but we're just referring them all to their VP who had multiple chances to kill the plan, and never spoke up.

Ironically I have to change my own password in 4 days. apparently the password expires are not all on the same day so that does not mean the guy causing all the problems will also expire in 4 days.

For the most part we've been pretty virus free, the only real problems we've ever had have always started not surprisingly, in the labs.

 

[bratwurst]

Windows machines with AD? You shouldn't need to reboot. Lock the machine, hit control-alt-del, put the new password in and re-authenticate to the domain.

If these guys are turning off AV/and not running updates, you absolutely should look into an IPS solution with multiple network segments. They analyze the packets going across the network and if they are malware, viruses, what have you, they just drop the traffic. You can go a long time without patches and the like if you keep the packet signatures up to date on the IPS.

Like what this company offers http://www.tippingpoint.com/