sysadmin

Just a quick reminder to take care of yourselves..

Just a quick reminder to take care of yourselves..

At one point in my career I was a sysadmin. I've since moved on to other things; however, I feel like this belongs here. Over the past few years, I've been having health issues. Occasional migraines which I dismissed. I assumed it was from looking at a screen all day long. Occasional sinus infections I ignored because everyone gets these, right? Occasional nosebleeds that HAD to be because of the dry climate I'm working in. Over the last few months, these symptoms have gotten progressively worse. My nosebleeds were happening frequently enough that I would keep pieces of tampons in my desk (I'm a male BTW) to shove them up my nose so I could continue working through them.

Today, I finally went to the doctor. When the diagnosis came in, I can't say I'm surprised. The symptoms had been there for years. Ironically, I now work in the medical field and run into this thing all the time. I've seen this happen to hundreds of people over the years. I just never thought it would happen to me. I have surgery scheduled for next week. Wish me luck. On the bright side, I finally know what was causing my symptoms. In a weeks time, I should finally be free from my deviated nasal septum. I should have figured it out a lot earlier to be honest. It's always DNS.

EDIT: Because I've been getting a few PMs: Yes. I do have to have surgery for my deviated septum. I tried telling this joke to my current coworkers, but it fell on deaf ears. I figured you would enjoy it.

EDIT 2: There's a number of people in the comments below who are using this as a reminder to take care of themselves physically. If you're struggling physically or mentally, please take the time to take care of yourself. A career isn't worth sacrificing your retirement health for. Please do. 10 years ago is the best time to start, today is the second best time to start.

EDIT 3: My first reddit silver! Thanks!.

EDIT 4: MY FIRST REDDIT GOLD. THANKS! THIS IS TOO MUCH.

Thought it was weird you were delaying your diagnosis till the end of the story.

it's because we already knew deep down it was DNS.

> my understanding of human physiology is that leaking red coolant is urgent.

Of course. How else can you be a good parent? Speaking of that, how's your wife and my kids?

Here is why Ex-Military have weird salary ideas

Here is why Ex-Military have weird salary ideas

So disclaimer, currently in the military. Getting IT certs on the Gov Dime to make a career change.

First off is DoD culture, soldiers in any occupation have been given a set of standards and a base level of qualifications to be considered “fully trained”. These requirements have created a culture where getting the certificate is the goal. As long as you have a cert you are a “Subject Matter Expert”, and looked upon as skilled service member. I hate it and it’s completely backwards. So when you get an ex-military guy in an interview and he thinks he’s IT god because he has A+, Net+ and CCENT, it’s not cause he has an ego, the military makes him think so.

This is disheartening and makes it difficult to hire veterans. But from the other side of the table I suggest including more detail in your job postings. Look at their resumé objectively and if it looks like fluff it probably is. That’s a double edge sword though; Army IT guy A could have just done help desk work for 5 years, but Army IT guy B could have been designing, deploying and maintaining NIPR and SIPR networks. This is where good interview questions can really sort out the less experienced guys.

For the salary estimations I blame the training soldiers receive as they are exiting the service. They use salary estimator tools that take in account our food/housing allowances. Our free healthcare and tax advantage. For example I make about 65,000 a year, only 32,000 is taxable income so my take home cash is about 4900 a month. Using those calculators, I would need a salary of about 85,000-95,000 to maintain my current standard of living.

So yea the salary requests are way out of scope for industry average. But it’s a byproduct of the training and culture they experienced . So when a dude puts 80k on his application don’t immediately think he’s crazy, be upfront and counter offer with an appropriate amount that is within industry norms.

Finally for those of us still in, we have plenty of opportunities to control our career and where we end up. So giving feedback here on reddit will help service members gain the experience you’re looking for. Even if you disagree with my points, extending an olive branch and offering advice to current military members now, will make it easier to hire them in the future. Plus your HR department, and marketing department will get the warm and fuzzy about hiring veterans.

I do have to say, we work our ass off and will are used to working late, and getting the job done. Guys that had the opportunity to be officers or NCOs(sergeants) make excellent leaders or managers. Imagine managing an IT department and you literally cannot fire anyone, you are responsible for each persons well being, and are their career mentor, trainer and disciplinarian.

TL;DR : Veterans are a byproduct of the DoD culture, don’t immediately discount veterans, be straight forward and make the interview an open discussion.

Edit: Did not expect this response. But I hope I cleared some stuff up for some, maybe made it muddier for others.

Here is the link for DoD pay tables DFAS Website

Here is the calculator to find your Regular Military Compensation MilitaryPay.Defence.gov

An article explaining and comparing military to civilian benefits

and i used this calculator as an example: Civilian Pay Equal To Military Pay

We also get the Personal Statement Of Military Compensation yearly. We can find it on our pay website and it is a work sheet that helps soldiers calculate what they really make. Its incredibly comprehensive and includes tax advantage, child care costs, insurance premiums, legal counsel, pension, clothing allowances, housing allowances , food allowances etc. I wont link mine here cause it does contain some private info but this website has a good write up explaining it

I hope this helped some ex-military applying for IT jobs, and hiring managers understand how complicated the military is. Please look at every ex-military applicant differently, we're all human. Being a Soldier doesn't make a person good, being a good person makes a good Soldier.

As for me, I have shit ton of work to do, thanks for all the input!

P.S Obligatory Gold edit

Thank you for that write-up and explanation. The part about salary calculations to compensated for what the Government was providing is a great piece of info to discuss in interviews.

he thinks he’s IT god because he has A+, Net+ and CCENT, it’s not cause he has an ego, the military makes him think so.

Also when they're fed propaganda on what they could expect to make in the private sector, they're fed cherrypicked DC-area salaries that are fairly inflated due to pork spending, corruption, and the very high cost of living in the DC area.

So they end up back in Kansas and think $120,000 for a jr sysadmin role is appropriate. Yeah maybe some DC-area hotshot contractor with the right connections and clearances can get that, but you won't.

They dont realize they got conned by recruiters and the DoD until later.

Also the DC-Area guy will pay $1,200-$1,400 a month for a studio apartment. Meanwhile the guy in Kansas pulling $70,000 will pay a fraction of that to own a home. The Kansas guy with the "low" salary doesn't know how easy he's got it. On paper its less money but from a housing perspective and cost of living perspective, its a bargain compared to competitive urban areas with a glut of talent.

I currently have a marine as a boss (I would say former Marine but I have been corrected before because apparently there is no such thing as a "former" marine).

Working for him took a lot of getting used to. But it's pretty good when you do. I have more structure than I've ever had in a job and I generally think that's good for IT.

Plus, you can curse and joke all you want because there is no way he hasn't heard worse. So there's that.

A little tip is if he is hard on you specifically he likes you.

And if you’re having problems at home, or stress that’s causing you to not perform do not hesitate to tell him. He will give you the shirt off his back I guarantee it.

We come from a culture where if some dude calls you at 2am 100 miles away cause he’s drunk and can’t drive home. You bet your ass we’d get in a car and pick him up.

Or if you’re tight on money and you’re struggling to feed your kids, I guarantee he’d send groceries to your house.

We are a pretty Type A breed of people, but have the most thoughtful and caring attributes when you break away that tough guy ego

Ok, did I miss something, when did TeamViewer become all about BDSM and sexual fetishes?

Ok, did I miss something, when did TeamViewer become all about BDSM and sexual fetishes?

Long story short, I decided not to renew my companies Teamviewer subscription this year as it just wasnt any sort of value for money. I just discovered they are trying to take me to collections for the next years payment despite my account being closed.

So I do what any techy person does in this day in age, I take to twitter to vent, and WTF all the tweets mentioning Teamviewer are BDSM, Sexual Fetishes and Sex chat type stuff.

Literally as far as I can tell here, 95% of tweets are people talking about booking in sessions or wanting sessions to do some sort of BDSM over Teamviewer.

See here: https://twitter.com/search?f=tweets&vertical=default&q=teamviewer&src=typd

Did I miss something? When has this been the main use of TeamViewer?

Sorry if this is only vaguely /sub/sysadmin stuff, but here I was all this time thinking Teamviewer was just a really expensive remote support tool.

Well, that's got my 'WTF' quota of the day.

I'm not sure that's so much 'major use' as 'more likely to post on Twitter'.

The whole 'control my PC and give me orders' thing I guess is a bit kinky, but ... I don't think I'm ever going to be able to give 'remote support' again without thinking of this.

..are you kink-shaming Teamviewer?

I will now wonder everytime I support remotely if the person on the other side gets off on me using their cursor with them unable to do anything

https://www.gimletmedia.com/reply-all/116-the-process

HARLEY: So let's say Mohammad, living in Kuwait, he finds my website, and he sends me an email and says, “Mistress Harley, I love the idea of being controlled by a strong powerful woman. Can I book a computer session with you?” And then I would say, “Yes. Booking a computer session is 100 dollars. Go to my website MistressHarley.com. Pay me, and then we will set up a time for the session, if it's not immediate.”

And then he would give me... whatever remote desktop sharing software he's using, he would give me the login information for that. We would discuss limits. You know, if he says, “Look you can do anything you want, but do not email my wife.” Like, ok. Great.

SRUTHI: Got it.

HARLEY: Now I know. I won't email your wife. And then once we start, it's like any BDSM play session where now we've negotiated. So now I'm not going to be polite anymore. And now I'm to do whatever I want, within the realms of what we discussed. The guy I’m thinking of in particular who I'm thinking of, who is Mohammed from Kuwait, uh, he likes to be exposed. And so when I go on his computer, I open up his camera on his computer and start taking pictures and videos of him. And then, I start posting those pictures and videos to my Twitter where I have like 50,000 followers. Or I might post them on my Loser Hall of Shame on my website. And then, you know, now his computer is locked down, and he is exposed. And this is all very exciting to him.

SRUTHI: So when you say he's excited, like how is he showing it?

HARLEY: Oh I mean, usually there's a little text chat in the corner, where he's like “Oh my god! What are you doing?” And me like, “Oh well I'm- You didn't say I couldn't go on your Facebook. And so now I'm in your Facebook, and now I see all your friends, and now I see where you work.” And “Oh my god this is so- this is so- you're so powerful. You’re- this is so exciting.” They'll say to me. And I'm like, “Yeah, of course I'm very powerful. I own all your shit now.”

Whats a nice way to tell the owner not to text me @ 7am?

Whats a nice way to tell the owner not to text me @ 7am?

I work at a very small company, about 100 total users. On occasion, the owner or his wife will message me at ridiculous hour, or on a weekend, to ask a question that can A - Wait until I have started my day @ 8am, or B - Can wait until its NOT a weekend.

Today I got a message from the owners wife @ 7:15, while I was on my way to work, because she couldn't connect to her Remote Access VM that was setup for her. When I responded via voice assistant asking what was she trying to connect to, and informed her I was in the car driving, her response was "Not an emergency". If it isn't an emergency, then why are you contacting me?

Does everyone have to deal with this shit, or are some of you lucky?

Unless it seems like 'server on fire' emergency, don't answer it until you get to work.

Stop answering at 17:01 PM

Start answering at 8:01 AM.

They'll get the memo.

If you consider it's an emergency, act on it right away. Everything else, CAN. WAIT.

EDIT: BTW, It's ok to text you at any hours. Not OK to wait/expect an answer right away.

If your owner or wife is a reasonable human being, I would approach it directly: "hey, if it's outside of normal business hours and it's not an emergency, it'd be great if you could send me an email instead of a text. I usually assume anything outside of business hours sent in a text is really important, and I want to make sure that I can clearly understand when there's a critical emergency and when it can wait."

They may forget once or twice, just remind them. Be clear that it's in their best interest to follow this rule so that you clearly understand when they need important after-hours responses.

If the owners are not reasonable people, you train them just like you train animals. You don't respond to the text until you arrive at work.

My boss(es) are generally amenable to the former. I have taught a few of my coworkers with the latter approach, though. If they abuse texting privileges, they stop getting responses to text messages.

Turn on Do Not Disturb while Driving (or the equivalent for Android.) To quote my iPhone... (Not quite the default message...)

I’m driving with Do Not Disturb While Driving turned on. If it’s really important call me, my car has a built in speakerphone.

(I’m not receiving notifications. If this is urgent, reply “urgent” to send a notification through with your original message.)

This is why you should always lock your computer before you leave your desk.

This is why you should always lock your computer before you leave your desk.

There is nothing better than your IT boss passing your desk and noticing you left you computer unlocked. Especially if you are logged on to half a dozen websites including Reddit. I eat my poop!!!

hah, standard around here is an email to the team generously offering to shout everyone lunch at the pub

We have a “Hipaa-potamus” background easily accessible on the company share drive for this very reason.

EDIT: RIP My inbox...so young...we hardly knew thee.

http://fakeupdate.net/ is a good one for unlocked PCs ;)

We always used the Cake'd method. Leaving your computer unlocked leads to an office wide "Hey guys, i will be bringing in cake on monday stop by my desk for a slice"

You are 100% expected to have cake

The sender of this email could not be validated and may not match the person in the "From" field.

The sender of this email could not be validated and may not match the person in the "From" field.

Can anyone tell me why the recipient is receiving this warning in Outlook? At least I would like to know if the problem is on our side or at the recipient server side. We have an Exchange 2016 server. I checked(https://mxtoolbox.com/spf.aspx) the SPF record and it looks fine.

Perhaps sender us using a gmail account through another account? Google insists in sending On behalf of, which kinda defeats the purpose.

Exchange and DNS. Unfortunately, IIRC, Exchange does not support DKIM

The recipient might have SMTP Authentication enforced on their mail server, which makes the mail server check to see if the name in the from field, matches the name in the email header. If it does not match, it will normally hold the email as bad.

Did they send the email to an externally hosted distribution list? We typically comes across the fraud flag on these types of emails.

Announcing our new companion sub-reddit: /sub/SysAdminBlogs

Announcing our new companion sub-reddit: /r/SysAdminBlogs

Our no Advertising policy here was born of a desire for absolute equality among content submitters, even if it hurt.

Well, we've removed some really good content in the past year or two because of that policy, and it has never sat well with us.

So, instead of modifying the rules here, we're giving you all a new place to submit that content where it can be seen and appreciated by your peers.

Those that want to see that sort of thing, can subscribe.

Those that don't want to see the same news or whatever they are already seeing via some other media feed, don't have to do anything any differently.

As of this moment there are TWO subscribers to that sub. Today is the day you get to be part of something NEW.

Got a dead-sexy Power Shell script to show us?

Did you build a better mousetrap in Python? Let's see it !

Did you find the perfect How to Love Linux article? Share it!

/sub/sysadminblogs

Probably worth reminding people like me that forget; you can just put '+' inbetween subreddits and browse them all in one page. So: https://www.reddit.com/r/SysAdminBlogs+sysadmin gets you both these subreddits in one page.

Also that adding +gonewild on the end of that string may make it more fun to browse, but could also get you fired...

Some favourite combinations:

/r/nottheonion+TheOnion

/r/WeWantPlates+CrappyDesign+DelusionalArtists

/r/DC_Cinematic+InsanePeopleFacebook

The real LPT is in the comments.

Not to be contrarian for the sake of contrarian...ness...but, why not just change the rule?

Also, where do the mods stand on "Blog-esque" posts, like everyone's favorite opinionated sysadmin?

Meltdown & Spectre Megathread

Meltdown & Spectre Megathread

Due to the magnitude of this patch, we're putting together a megathread on the subject. Please direct your questions, answers, and other comments here instead of making yet another thread on the subject. I will try to keep this updated when major information comes available.

If an existing thread has gained traction and a suitable amount of discussion, we will leave it as to not interrupt existing conversations on the subject. Otherwise, we will be locking and/or removing new threads that could easily be discussed here.

Thank you for your patience.

A CPU predicts you will walk into a bar, you do not. Your wallet has been stolen.

Not all AVs play nicely with the latest windows patches that fix the CPU Flaw.

You can track which ones using this google doc

And here is the official MS piece about AV support

Microsoft have released a powershell module that checks if their patch as well as if firmware patches have been applied: https://support.microsoft.com/en-us/help/4073119/windows-client-guidance-for-it-pros-to-protect-a...

PowerShell Verification

Install the PowerShell module

PS > Install-Module SpeculationControl

Run the PowerShell module to validate protections are enabled

PS > Get-SpeculationControlSettings

It will vary depending on what the machines are doing and how they are configured, but 30% sounds like it's the high end.

Redhat's benchmarks from another thread. Essentially 1-20% depending, with particular applications listed as between 2% and 12%.

EDIT: Reportedly Microsoft are not seeing any performance penalty on Azure after patching.

Intel bug incoming

Intel bug incoming

http://pythonsweetness.tumblr.com/post/169166980422/the-mysterious-case-of-the-linux-page-table

https://www.reddit.com/sub/Amd/comments/7nkza3/massive_intel_hardware_bug_might_be_incoming_up/

TLDR;

Copying from the thread on 4chan

There is evidence of a massive Intel CPU hardware bug (currently under embargo) that directly affects big cloud providers like Amazon and Google. The fix will introduce notable performance penalties on Intel machines (30-35%).

People have noticed a recent development in the Linux kernel: a rather massive, important redesign (page table isolation) is being introduced very fast for kernel standards... and being backported! The "official" reason is to incorporate a mitigation called KASLR... which most security experts consider almost useless. There's also some unusual, suspicious stuff going on: the documentation is missing, some of the comments are redacted (https://twitter.com/grsecurity/status/947147105684123649) and people with Intel, Amazon and Google emails are CC'd.

According to one of the people working on it, PTI is only needed for Intel CPUs, AMD is not affected by whatever it protects against (https://lkml.org/lkml/2017/12/27/2). PTI affects a core low-level feature (virtual memory) and has severe performance penalties: 29% for an i7-6700 and 34% for an i7-3770S, according to Brad Spengler from grsecurity. PTI is simply not active for AMD CPUs. The kernel flag is named X86_BUG_CPU_INSECURE and its description is "CPU is insecure and needs kernel page table isolation".

Microsoft has been silently working on a similar feature since November: https://twitter.com/aionescu/status/930412525111296000

People are speculating on a possible massive Intel CPU hardware bug that directly opens up serious vulnerabilities on big cloud providers which offer shared hosting (several VMs on a single host), for example by letting a VM read from or write to another one.

Edit: the examples of the i7 series, are just examples. This affects all Intel platforms as far as I can tell.

So let me get this straight, not only is this a massive security bug that unpatched could let a VM write to another VM, but patched it will incur a 30+% performance hit?

Goddamnit 2018 you were supposed to be better than 2017.

Only if you use Intel (99% of the market)

https://media.giphy.com/media/RHiD0K65NxxLO/giphy.gif

What an EPYC opportunity!

I'm sorry, I know where the door is.

Turnkey Linux / Bitnami, anyone using them in production?

Turnkey Linux / Bitnami, anyone using them in production?

We're a tiny team and are looking to deploy a handful of open source web apps. Being able to deploy these pre-packaged VMs (or simple to install stacks) quickly is very appealing. We don't have strong Linux skills(we're a windows shop) and this has been a deterrent in the past but the quick and easy set-up seems to minimise time cost.

We're also looking at simply paying for decent webhosting and using something like installation or solicitous to manage them but we want some of these applications to be internal only so this is not as ideal.

Has anyone had much experience using these VM appliances in the past?

Here's the fundamental problem with appliance platforms like TurnKey Linux:

The software that it packages has an array of features and settings, and configuring them correctly is complex.

In order to hide that complexity with an appliance-like wrapper, the developers of the appliance necessarily need to make decisions on your behalf on how things should be configured, based on what they think their users are most likely going to need.

Different environments are, well... different, and have their own unique configuration requirements. Since these appliances have already made their configuration decisions for you, there's going to be some cases where the appliances aren't going to be optimal (and may even catastrophically fail).

If you don't have experience configuring the software wrapped in the appliance, then you're not in a position to determine if the configuration of the appliance is suitable for your environment.

This approach is frequently used by developers doing their own operations. It lets them get off the ground quickly, and is a great choice when you just need to produce an MVP. However, it's also how you wind up accidentally exposing your databases (or other sensitive systems) to the Internet, using products that have unexpected limitations that can't easily be worked around without a redesign, or ending up with an unexpectedly high bill.

While I can give developers a pass since their focus is on building software, professional admins should know better.

Learning to manage Linux is not difficult, and in 2018, I'd expect anyone who claims to be a sysadmin to have at least a basic ability to function in a Linux environment. If your team does not, then the professional thing to do is to admit that you don't have the expertise to manage those applications, and either hire someone that does, or outsource it to a service provider that can manage it for you.

EDIT: Perfect example: many of the TurnKey Linux appliances are designed to be managed with Webmin, a web control panel interface. While this works for simple configuration tasks, it doesn't scale well, and there's configuration functionality that Webmin doesn't expose. You can drop down to the command line and configure things directly, but creating config files that have a mix of webmin-managed and hand-managed configurations is extremely error prone, and can leave your services in a broken or undefined state that's difficult to get out of if you don't know what you're doing. Lastly, Webmin has a long history of security problems.

You still need to be comfortable with the basics of working with Linux in order to configure, run, and diagnose containers.

and want to minimise time required to set some of these up.

The majority of your time and effort should be on the long term maintenance. This is something that will be harder, and take longer, if noone on your team was involved in the original setup.

I wouldn't recommend it myself. They end up being black boxes that are quite difficult to fix and modify.

The installation process is often the best time to get your head round the configuration format, logging, permissions, file structure, etc.

Try one of these subthreads