sysadmin

From Bloomberg: How China Used a Tiny Chip to Infiltrate Amazon and Apple

From Bloomberg: How China Used a Tiny Chip to Infiltrate Amazon and Apple

Time to check who manufactured your server motherboards.

​

The Big Hack: How China Used a Tiny Chip to Infiltrate Amazon and Apple

"Elemental servers sold for as much as $100,000 each, at profit margins of as high as 70 percent, according to a former adviser to the company. Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."

Made me laugh

I'm astounded that SoC technology has come so far that a chip of that size can be capable of anything like this. It says a lot about the lump of outdated parts that I work on.

Edit:

In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached,

Holy fuck

AWS’s response:

“We've found no evidence to support claims of malicious chips or hardware modifications.”

Source

Apple’s response:

Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple. Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. We have repeatedly and consistently offered factual responses, on the record, refuting virtually every aspect of Bloomberg’s story relating to Apple.

On this we can be very clear: Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement.

In response to Bloomberg’s latest version of the narrative, we present the following facts: Siri and Topsy never shared servers; Siri has never been deployed on servers sold to us by Super Micro; and Topsy data was limited to approximately 2,000 Super Micro servers, not 7,000. None of those servers has ever been found to hold malicious chips.

As a matter of practice, before servers are put into production at Apple they are inspected for security vulnerabilities and we update all firmware and software with the latest protections. We did not uncover any unusual vulnerabilities in the servers we purchased from Super Micro when we updated the firmware and software according to our standard procedures.

We are deeply disappointed that in their dealings with us, Bloomberg’s reporters have not been open to the possibility that they or their sources might be wrong or misinformed. Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple.

While there has been no claim that customer data was involved, we take these allegations seriously and we want users to know that we do everything possible to safeguard the personal information they entrust to us. We also want them to know that what Bloomberg is reporting about Apple is inaccurate.

Apple has always believed in being transparent about the ways we handle and protect data. If there were ever such an event as Bloomberg News has claimed, we would be forthcoming about it and we would work closely with law enforcement. Apple engineers conduct regular and rigorous security screenings to ensure that our systems are safe. We know that security is an endless race and that’s why we constantly fortify our systems against increasingly sophisticated hackers and cybercriminals who want to steal our data.

Source

Bloomberg on the denials:

The companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation. One of those officials and two people inside AWS provided extensive information on how the attack played out at Elemental and Amazon; the official and one of the insiders also described Amazon’s cooperation with the government investigation. In addition to the three Apple insiders, four of the six U.S. officials confirmed that Apple was a victim. In all, 17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information.

Source

How US Used a Tiny Chip to Infiltrate Companies Worldwide: They installed Intel ME on all their chipsets and CPUs.

Why do people not understand that working outside of business hours essentially makes other people work outside of business hours?

Why do people not understand that working outside of business hours essentially makes other people work outside of business hours?

As I enter into the latter half of 30, I notice, more and more, a certain type of toxicity in the American work environment. The person who works after hours and feels good about it.

I just can’t understand how people can’t see that their email sent at 8pm or their project updates at 2pm on a Saturday is essentially putting it on other people to work after hours as well.

I was talking to a coworker who said he was coming into work over the weekend. This is an older guy who is divorced, a lot of his value is in his job. He had no outside hobbies and will be completely destroyed when he is forced to retire and realizes that he is just a cog in the machine of business.

What is this American obsession with working too much and seeing it as a positive thing? Why is work/life balance so hard of a concept for people to try to push and strive for? Do people not like their hobbies? Do they not feel valued at home? Do they feel that work is the end all and be all to life?

Sorry for the rant post, just frustrated to see this same old song and dance year after year, job after job.

All I ask is for people to remember those after hours emails that you didn’t have time to send during the day, are essentially forcing someone else to put in more time than they have in their day.

Edit: My word, I’ve hit a pythons nest. It really is interesting to here how divisive this subject is.

Edit2: There has been a huge mix of thought out replies, some insults, some defending the right to work for free as much as one wants and everything in between. I’ve always found us in /sub/sysadmin to be a salty bunch, often arguing just to argue, it seems to be in our nature.

Edit3: I woke up to 101 replies, I read them all but it’s just too much to respond to. Most of them are around me being lazy or people arguing that they like their job and should be able to work as much as they like...to make it more simple, I’ll just add these few points

-It’s fine to like your job and enjoy your work. It’s dangerous to derive your personal value from your job. After 30-40 years of working nonstop, you retire. They have a party for you....and no one thinks about you ever again.

Do something useful with that time instead of making someone else money. Spend time with your family. If you don’t have a family, volunteer (I volunteer at the animal shelter and parks services, I beautify my neighborhood and help abandoned dogs find homes. This is a better use of time instead of making my boss more money).

-My post was only about salaried workers. If your hourly, work and make those dollars, more power to you. I wish we were all hourly as companies would force people to work less because...guess what? They don’t want to actually pay for all the free working hours you’re currently giving them.

-I guess the final thing I have to say is that people working more makes other people work more, or at least sets the expectation. If several people start working all the time, send data after hours and work on projects after hours, that sets a bad expectation for everyone else. We don’t live in a vacuum, things we do in our lives affect others. Those that want to work after hours a lot, negatively impact those of us that want to work the hours we were hired for.

I put in my 40-45 and that’s the way it should be. Working more for free is not something to be proud of.

I just can’t understand how people can’t see that their email sent at 8pm or their project updates at 2pm on a Saturday is essentially putting it on other people to work after hours as well.

Why are you reading your work email outside of work hours?

I like having flexible hours and working some later at night after the gym or running. I don't send emails out that late though out of respect for others.

I came into work Tuesday morning this week to a meeting invite in my inbox. It was "urgent" sent out at 9:30pm Monday for an 8am Tuesday meeting. I came in at 9am that day. WTF?!

Who says you must respond to the requests outside of normal business hours? If\When I send an email after business hours, I don’t expect a response until the following day. With that said, I do agree with your sentiments regarding how some people expect a response. I guess we can thank technological advancements for keeping us connected at all times.

Sometimes i run out of shit to read on the toilet.

Just something to cheer you up

Just something to cheer you upI got a new printer today from my mother's new wife. It was a Brother from another mother.

Edit: Oh hey I got gilded, thanks!
Just something to cheer you up

I got a new printer today from my mother's new wife. It was a Brother from another mother.

Edit: Oh hey I got , thanks!

Have your damn upvote

You mean /sub/sysdadminjokes right?

You think you can come on here and tell terrible jokes to get upvotes? Well buster let me tell you YOU'RE ABSOLUTELY RIGHT YOU CAN.

We need a /sub/sysadmindadjokes sub.

I'm at Microsoft Ignite and...

I'm at Microsoft Ignite and...

About 1-5% of you here need to start showering EVERY day. PLEASE.

Game cons need signs about hygiene but I expect more of our technical brethren.

Even at DefCon we practice 3-2-1 (three hours of sleep, two meals. and a shower a day). What is wrong with you savages?

1-5% is fairly vague, can you start keeping more accurate logs please.

three hours of sleep

you on some good shit

Please run sfc (shower for conference) tool to resolve issue

RCN stores your online passwords as security phrases in plaintext and their reps can see them when you call. and they think its okay to do that.

RCN stores your online passwords as security phrases in plaintext and their reps can see them when you call. and they think its okay to do that.This is dumb. my jaw damn near hit the floor.

I set up my online account and set my password to something generated by KeePass, 26 chars, nice 120bit strength.

noticed that during initial account creation their rep got my email wrong. so i called them.

Their rep without any validation (strike 1), was able to see my password that i had just set online, 5 minutes earlier, in plaintext (strike 2) and then straight up READ IT BACK TO ME, OVER THE PHONE, asking "the password looks very long and odd, are you sure this is what you want?" (strike 3, yer out!)

I had to interrupt her halfway through reading the string because my jaw hit the floor so hard, i may have caused an earthquake in china.

And their explanation, after i escalated the call, and then tweeted them? "we need to see it so we can validate identity of users calling in, its cool though, we dont have any access to your account and cant see anything"

I legit don't know how to handle this. I asked the shift supervisor to escalate the issue, stating that this approach, in the current climate where security of PII is paramount, is a complete breakdown of practices and at most got your general, "someone will get back in touch with you in about 24 hours. i dont have a case number to give you, but i proooomise they will reach out."

https://twitter.com/RCNconnects/status/1043616436843945985
https://i.imgur.com/NhobZxn.png

so.
how is your weekend?
RCN stores your online passwords as security phrases in plaintext and their reps can see them when you call. and they think its okay to do that.

This is dumb. my jaw damn near hit the floor.

I set up my online account and set my password to something generated by KeePass, 26 chars, nice 120bit strength.

noticed that during initial account creation their rep got my email wrong. so i called them.

Their rep without any validation (strike 1), was able to see my password that i had just set online, 5 minutes earlier, in plaintext (strike 2) and then straight up READ IT BACK TO ME, OVER THE PHONE, asking "the password looks very long and odd, are you sure this is what you want?" (strike 3, yer out!)

I had to interrupt her halfway through reading the string because my jaw hit the floor so hard, i may have caused an earthquake in china.

And their explanation, after i escalated the call, and then tweeted them? "we need to see it so we can validate identity of users calling in, its cool though, we dont have any access to your account and cant see anything"

I legit don't know how to handle this. I asked the shift supervisor to escalate the issue, stating that this approach, in the current climate where security of PII is paramount, is a complete breakdown of practices and at most got your general, "someone will get back in touch with you in about 24 hours. i dont have a case number to give you, but i proooomise they will reach out."

https://twitter.com/RCNconnects/status/1043616436843945985

so. how is your weekend?

Six months later — “Our systems were compromised by hackers through no fault of our own. All your data has been stolen, but it was all hackers. Hackers bad, mmmmkay?”

If they're doing this, they're probably not compliant with PCI security standards, either. I hope this gets enough traction to catch the attention of one or more credit card companies.

Followed by the typical boilerplate "We take security seriously..." In their breach notification message.

Yeah, no.

Hopefully You don't have a credit card on file or anything you care about getting out. You'd think people would be smarter about this stuff since there seems to be an article at least once a month about data breaches.

It's 2018, nobody should be buying a laptop with a HDD!

It's 2018, nobody should be buying a laptop with a HDD!

I'm an IT apprentice, so my opinion in terms of purchasing new devices doesn't carry much weight. My boss for some reason thinks a 2TB HDD on a laptop is better than a 256/500GB SSD. Considering all of our files are on the network and don't need to be saved locally, this makes little to no sense. Am I right in my thinking here? Just feeling incredibly frustrated.

Spinning rust is the biggest bottleneck in any modern PC as far as I'm concerned. If mass on board storage isn't a requirement, then it's always better to go solid state. Of course, if there is an M.2 slot and SATA, then go for both!

Yes, your boss is an idiot...

Brand new 2018 iMacs for $1200+ still ship with with 5400RPM HDDs by default. It's a nightmare trying to convince Purchasing that the $200 upgrade to an SSD isn't just a benefit, but a necessity at that point.

Correct. But /u/UTFR_TOM is now at a Junction in her/his career. You can either be "that person" which makes a song and dance about it. Your manager will dislike and possibly shoebox you. Or you shake your head, and move on. I was "that person" and even with all the benchmarks and proof in the world, some people just never learn. In IT you're going to find a lot of people like this, most of the time they will outrank you.

Hold the fuck up, you can rack servers by setting them on the back screws and swinging them up?

Hold the fuck up, you can rack servers by setting them on the back screws and swinging them up?I found this video on an ad on Facebook, Gif of the ad. 

You can just pull out the ready rails and set the back two screws then swing them up? You don’t need to hold the entire thing up while balancing and trying to get the screws in? You don’t need to get busted knuckles while scraping against the rails? What the fuck, man?
Hold the fuck up, you can rack servers by setting them on the back screws and swinging them up?

I found this video on an ad on Facebook,

You can just pull out the ready rails and set the back two screws then swing them up? You don’t need to hold the entire thing up while balancing and trying to get the screws in? You don’t need to get busted knuckles while scraping against the rails? What the fuck, man?

Every time I have to rack a server, the rails have a new design. I have to figure them out each time.

Wuuuuut? Holy crap. I've been doing it wrong all these years.

Every time there is a few months between having to rack up the same type of servers i have to re-remember the trick

Sun boxes always slid into retracted rails. They were easy as hell to rack.

Our dell servers were horrible and always took two people.

I have no idea what any of them are like anymore (AWS) ;)

So this is the end...

So this is the end...

I finally announced my retirement last Thursday and it looked like that I told my boss that his mother died.

For the past 3 years or so I was the only qualified IT guy in the whole dept. And it was reaffirmed when they hired a new director recently and told him, "forget about the rest, thisdodobird is the man you need to keep"

Fuck that nobody owns me.

With nothing to show for my efforts, not to mention my stress. I've had enough, so I took the plunge.

I've got plans, I want to spend more time with my wife and finally pursue the dreams that we've had.

My fellow sysadmins and geeks: don't let them trod on you.

PS: Seeing that I won't change my mind, they mentioned having me on as a consulting contractor. I said no.

I feel much lighter this weekend. 😊

Edit: posted using wrong user shrugs doesn't matter. And thank you everyone for your support it really means a lot!

Edit #2: RIP inbox, thank you again everyone for wishing me well on this next chapter. Some assume I'm old, but I'm still a spring chicken! I'll be 42 end of this month!

Due to special circumstances from our Social Security (I meet certain requirements as a disabled citizen - am deaf btw), I can retire with full benefits and a pension that's quite higher than what I made at my job.

So with the encouragement of my friends & family, not to mention the negativity I've been facing from work. I decided to take the leap, which I should have done a couple of years ago.

So financially, yeah you can call it "fuck you" money. Which will be sunk into my high tech dev "goat" farm.

I feel the same, brother. I just put my 2 weeks in after being told that I should work while on my honeymoon. There's almost a euphoria associated to leaving.

In a previous job, they called me on my wedding day. And they actually filed a complaint against me for not being on call that day as per contract.

A previous boss once flipped out into a full on screaming rage because a employee didn't answer their personal phone on their day off. That person wasn't on call, they had a day off to have an eye operation that they booked months in advance. They were literally being operated on at that exact time.

That's a new low.

Last week I handed in my notice, yesterday 1/3 of IT got walked out the door.

Last week I handed in my notice, yesterday 1/3 of IT got walked out the door.

New IT leadership came in externally, I checked their history and they have impressive records in 'turning IT around'.

Well, I know exactly what that means. Do more with less and it's something I pre empted which resulted in my getting a better paid position with better benefits.

About 70 (so far) friends and colleagues go no notice, shown the door, some with 40 years service.

Not just people but a fucking truckload of knowledge left with them, I'm sure you all have your knowledge fully detailed in KBs right???? Yeah us neither.

It's going to be an absolute shit show when they realize stuff breaking is due to undocumented maintenance people used to do or how to fix the mainframe was generally done by the guy you frog marched out the door.

I feel very quite content about the foresight I had but it's a small comfort when you see so many lose their livelhoods. Many over 50 all hitting the job market simultaneously.

Do more with less

This is the biggest fraud in the history of work. When you hear it, you know that the well is poisoned and you should go.

I was laid off 2 years ago. But the way we do things, is we give people 2 months notice, restrict their access, and tell them they can ride it out at home if they want to.

I'm pretty sure this is done to avoid paying severance. If you find a job in those 2 months, they don't have to pay it, because, technically you have not been laid off yet. We had one guy get a new job and take laptop into the new job and VPN in and continue to respond to email and Skype from the new job to make it look like was still looking for work, so he could still get the severance package. And he almost got away with it. I don't know how he got caught, but he did.

Well, this happened to me.

The day they told me, I spent about 30 minutes with a manager and some woman in HR. When they were done their spiel, I politely asked if I needed to sign anything, and then told them to email me the details, because I needed to go and troubleshoot an Symantec issue and needed to be on a call in 5 minutes. As I was leaving, I heard the HR lady say 'What the fuck was that? Why is he not going home all pissed off?"

I get on my call, and immediately realize I no longer have rights to remote control workstations. So, I tell the people on the call what happened. After a minute of stunned silence, the manager on the phone tells me to hold on. 2 min later, I can suddenly remote control machines again. We troubleshoot the issue, gather logs, and I submit a ticket to Symantec.

I worked on an international team. My WHOLE team, including my manager AND his manager was let go. I was the only US resource on the team. All of my coworkers were told to finish their time from home after a week. They were allowed in the building in order to talk to HR and do job searches, and that's it.

So, a week goes buy, and once again the Symantec issue surfaces. And I'm on an all day call troubleshooting the issue. While on the call, someone starts rattling off the names of a bunch of my coworkers that were sent home, and says we need them on this bridge line now. And the manager on the call says "We can't do that. These people were let go, and calling them in now is an admission we made a mistake." So, I jokingly say "Well then, I'm hanging up." to which she immediately replied "Except for you. You're not going anywhere."

Right after that call is over, I get a call from some manager in I don't even know what country telling me that basically I seem to be very important, and that he has restored all my previous access, because they're not 100% sure what I need to do my job.

So, for the next 8 weeks, I was an island. No coworkers, no manager. I'm working 10-12 hour days, because I need to do the work of a half dozen people. They give me all the time off I need during the day to go to interviews, and I interview for some internal positions as well. But when the 8 weeks was up, after working insanely long hours and pretty much proving I was needed, they were still going to lay me off.

Luckily, about 3 days before my last day, I was offered another position internally, which I accepted, because I didn't want to lose the seniority, and I REALLY liked the new manager. So, my last day was Friday. I have a 6:00 PM meeting with the "survivors" of the layoff to discuss new management structure, etc.

At the end of the call, I ask the executive on the call what's going to happen to our teams ticket queue. There are over 100 tickets in there now. He says the manager will deal with it. I tell him the manager was let go over 6 weeks ago. He then replies the manager's manager will deal with it. Then I tell him he was laid off also. And the guys asks me "What the fuck have you been doing unsupervised for the last 2 months?" To which I reply "Um... Working 10-12 hour days and picking up the pieces of the layoff. I'm looking forward to getting my evenings back." And the guy says "Why the fuck were we laying you off? That's insane!"

Monday comes around, and my new boss starts my training, and politely says that he really doesn't want me doing any of my old work, since, technically, I should have been laid off on Friday. I completely agree.

I start declining meetings for my old jobs and set up an auto-reply that I am in a new role and an unable to assist them with tasks from my previous job.

People go APESHIT that day. My phone is ringing off the hook with people asking what they're supposed to do now.

There was a HUGE gap in support and admin for probably about 6 months before they finally decided to create a new team to replace my old team. But the new team was all contractors with no knowledge of how things operate. Probably took them a good 6 months to get up to speed. At one point they asked me to cross-train the new team, which I was willing to do. But my boss absolutely rejected the idea, saying I was far too busy in my new role, and they should have thought of that before they laid me off.

2 years in the new role, and I'm quite happy. The team is awesome. My manager is great. Work-life balance is a little skewed towards work. But my manager makes up for it in other ways. I work from home a lot. if I am going to work an evening, my boss lets me take my family out to dinner and expense it. We then get back, and I VPN in and do work. My kids are 15 and 17 now, so they're on autopilot, going over friends' houses or playing on Xbox. When they want me, they come down to the basement and we talk.

First off, good on you for moving on.

Second, let this be a reminder to all of /sub/sysadmin. No matter how good your situation is, no matter how good the relationship with your boss or your team, always be looking for your next gig. Keep in touch with friends that leave. Know what companies in your area are doing and who are the "good ones" to work for. And when a recruiter reaches out, if its a role you'd actually be interested in, talk with them, even if you aren't actively looking.

Always be looking for your next role. Never ever show loyalty to a company. They will never show it back. And, at some point, the people you love working with? They'll leave. Your boss who treats you well? They are going to leave. Or get transferred. Or get promoted. And your situation will change.

I'm certainly not advocating constantly blasting your resume or switching jobs every 6 months. But what I am saying is - keep your eyes open. Never refuse to consider an opportunity because you are happy where you are at. Always have an up to date resume and be ready to move on if the opportunity comes.

Rehired back as consultants in 6 months after the new management leaves for a new opportunity

"Throttled" Message in Outlook

"Throttled" Message in Outlook

[deleted]

"Throttled" Message in Outlook

A few users running Outlook 2016 (O365) have reported the following error and then get stuck in a password loop...

​

Has anyone seen this before?

​

https://imgur.com/a/6ARDMTu

I'm experiencing absolute schadenfreude knowing that Microsoft is experiencing a significant outage due to one of their updates going sideways.

Am I a bad person?

Microsoft aware of the issue:

EX147785 - Unable to access Outlook or Skype

Title: Unable to access Outlook or Skype User Impact: Users may receive a message indicating they are being throttled when accessing Outlook or Skype. Current status: As part of our follow-up remediation actions stemming from the issue reported under service incident MO147606, an update was introduced to the components that manage authentication. We've determined that this update has resulted in users receiving a message indicating they are being throttled when attempting to access Outlook and Skype. We're reverting the update to remediate the problem. Scope of impact: This issue could potentially affect any of your users intermittently if they are routed through the affected infrastructure. Start time: Wednesday, September 5, 2018, at 5:30 PM UTC Preliminary root cause: A recent update to components that manage authentication requests has resulted in users receiving a message indicating that they are being throttled when accessing some Office 365 services. Next update by: Wednesday, September 5, 2018, at 6:30 PM UTC

Same here. We're up to about 25 users. Central IL.

Try one of these subthreads