sysadmin

Whats a nice way to tell the owner not to text me @ 7am?

Whats a nice way to tell the owner not to text me @ 7am?

I work at a very small company, about 100 total users. On occasion, the owner or his wife will message me at ridiculous hour, or on a weekend, to ask a question that can A - Wait until I have started my day @ 8am, or B - Can wait until its NOT a weekend.

Today I got a message from the owners wife @ 7:15, while I was on my way to work, because she couldn't connect to her Remote Access VM that was setup for her. When I responded via voice assistant asking what was she trying to connect to, and informed her I was in the car driving, her response was "Not an emergency". If it isn't an emergency, then why are you contacting me?

Does everyone have to deal with this shit, or are some of you lucky?

Unless it seems like 'server on fire' emergency, don't answer it until you get to work.

Stop answering at 17:01 PM

Start answering at 8:01 AM.

They'll get the memo.

If you consider it's an emergency, act on it right away. Everything else, CAN. WAIT.

EDIT: BTW, It's ok to text you at any hours. Not OK to wait/expect an answer right away.

If your owner or wife is a reasonable human being, I would approach it directly: "hey, if it's outside of normal business hours and it's not an emergency, it'd be great if you could send me an email instead of a text. I usually assume anything outside of business hours sent in a text is really important, and I want to make sure that I can clearly understand when there's a critical emergency and when it can wait."

They may forget once or twice, just remind them. Be clear that it's in their best interest to follow this rule so that you clearly understand when they need important after-hours responses.

If the owners are not reasonable people, you train them just like you train animals. You don't respond to the text until you arrive at work.

My boss(es) are generally amenable to the former. I have taught a few of my coworkers with the latter approach, though. If they abuse texting privileges, they stop getting responses to text messages.

Turn on Do Not Disturb while Driving (or the equivalent for Android.) To quote my iPhone... (Not quite the default message...)

I’m driving with Do Not Disturb While Driving turned on. If it’s really important call me, my car has a built in speakerphone.

(I’m not receiving notifications. If this is urgent, reply “urgent” to send a notification through with your original message.)

This is why you should always lock your computer before you leave your desk.

This is why you should always lock your computer before you leave your desk.

There is nothing better than your IT boss passing your desk and noticing you left you computer unlocked. Especially if you are logged on to half a dozen websites including Reddit. I eat my poop!!!

hah, standard around here is an email to the team generously offering to shout everyone lunch at the pub

We have a “Hipaa-potamus” background easily accessible on the company share drive for this very reason.

EDIT: RIP My inbox...so young...we hardly knew thee.

http://fakeupdate.net/ is a good one for unlocked PCs ;)

We always used the Cake'd method. Leaving your computer unlocked leads to an office wide "Hey guys, i will be bringing in cake on monday stop by my desk for a slice"

You are 100% expected to have cake

The sender of this email could not be validated and may not match the person in the "From" field.

The sender of this email could not be validated and may not match the person in the "From" field.

Can anyone tell me why the recipient is receiving this warning in Outlook? At least I would like to know if the problem is on our side or at the recipient server side. We have an Exchange 2016 server. I checked(https://mxtoolbox.com/spf.aspx) the SPF record and it looks fine.

Perhaps sender us using a gmail account through another account? Google insists in sending On behalf of, which kinda defeats the purpose.

Exchange and DNS. Unfortunately, IIRC, Exchange does not support DKIM

The recipient might have SMTP Authentication enforced on their mail server, which makes the mail server check to see if the name in the from field, matches the name in the email header. If it does not match, it will normally hold the email as bad.

Did they send the email to an externally hosted distribution list? We typically comes across the fraud flag on these types of emails.

Announcing our new companion sub-reddit: /sub/SysAdminBlogs

Announcing our new companion sub-reddit: /r/SysAdminBlogs

Our no Advertising policy here was born of a desire for absolute equality among content submitters, even if it hurt.

Well, we've removed some really good content in the past year or two because of that policy, and it has never sat well with us.

So, instead of modifying the rules here, we're giving you all a new place to submit that content where it can be seen and appreciated by your peers.

Those that want to see that sort of thing, can subscribe.

Those that don't want to see the same news or whatever they are already seeing via some other media feed, don't have to do anything any differently.

As of this moment there are TWO subscribers to that sub. Today is the day you get to be part of something NEW.

Got a dead-sexy Power Shell script to show us?

Did you build a better mousetrap in Python? Let's see it !

Did you find the perfect How to Love Linux article? Share it!

/sub/sysadminblogs

Probably worth reminding people like me that forget; you can just put '+' inbetween subreddits and browse them all in one page. So: https://www.reddit.com/r/SysAdminBlogs+sysadmin gets you both these subreddits in one page.

Also that adding +gonewild on the end of that string may make it more fun to browse, but could also get you fired...

Some favourite combinations:

/r/nottheonion+TheOnion

/r/WeWantPlates+CrappyDesign+DelusionalArtists

/r/DC_Cinematic+InsanePeopleFacebook

The real LPT is in the comments.

Not to be contrarian for the sake of contrarian...ness...but, why not just change the rule?

Also, where do the mods stand on "Blog-esque" posts, like everyone's favorite opinionated sysadmin?

Meltdown & Spectre Megathread

Meltdown & Spectre Megathread

Due to the magnitude of this patch, we're putting together a megathread on the subject. Please direct your questions, answers, and other comments here instead of making yet another thread on the subject. I will try to keep this updated when major information comes available.

If an existing thread has gained traction and a suitable amount of discussion, we will leave it as to not interrupt existing conversations on the subject. Otherwise, we will be locking and/or removing new threads that could easily be discussed here.

Thank you for your patience.

A CPU predicts you will walk into a bar, you do not. Your wallet has been stolen.

Not all AVs play nicely with the latest windows patches that fix the CPU Flaw.

You can track which ones using this google doc

And here is the official MS piece about AV support

Microsoft have released a powershell module that checks if their patch as well as if firmware patches have been applied: https://support.microsoft.com/en-us/help/4073119/windows-client-guidance-for-it-pros-to-protect-a...

PowerShell Verification

Install the PowerShell module

PS > Install-Module SpeculationControl

Run the PowerShell module to validate protections are enabled

PS > Get-SpeculationControlSettings

It will vary depending on what the machines are doing and how they are configured, but 30% sounds like it's the high end.

Redhat's benchmarks from another thread. Essentially 1-20% depending, with particular applications listed as between 2% and 12%.

EDIT: Reportedly Microsoft are not seeing any performance penalty on Azure after patching.

Intel bug incoming

Intel bug incoming

http://pythonsweetness.tumblr.com/post/169166980422/the-mysterious-case-of-the-linux-page-table

https://www.reddit.com/sub/Amd/comments/7nkza3/massive_intel_hardware_bug_might_be_incoming_up/

TLDR;

Copying from the thread on 4chan

There is evidence of a massive Intel CPU hardware bug (currently under embargo) that directly affects big cloud providers like Amazon and Google. The fix will introduce notable performance penalties on Intel machines (30-35%).

People have noticed a recent development in the Linux kernel: a rather massive, important redesign (page table isolation) is being introduced very fast for kernel standards... and being backported! The "official" reason is to incorporate a mitigation called KASLR... which most security experts consider almost useless. There's also some unusual, suspicious stuff going on: the documentation is missing, some of the comments are redacted (https://twitter.com/grsecurity/status/947147105684123649) and people with Intel, Amazon and Google emails are CC'd.

According to one of the people working on it, PTI is only needed for Intel CPUs, AMD is not affected by whatever it protects against (https://lkml.org/lkml/2017/12/27/2). PTI affects a core low-level feature (virtual memory) and has severe performance penalties: 29% for an i7-6700 and 34% for an i7-3770S, according to Brad Spengler from grsecurity. PTI is simply not active for AMD CPUs. The kernel flag is named X86_BUG_CPU_INSECURE and its description is "CPU is insecure and needs kernel page table isolation".

Microsoft has been silently working on a similar feature since November: https://twitter.com/aionescu/status/930412525111296000

People are speculating on a possible massive Intel CPU hardware bug that directly opens up serious vulnerabilities on big cloud providers which offer shared hosting (several VMs on a single host), for example by letting a VM read from or write to another one.

Edit: the examples of the i7 series, are just examples. This affects all Intel platforms as far as I can tell.

So let me get this straight, not only is this a massive security bug that unpatched could let a VM write to another VM, but patched it will incur a 30+% performance hit?

Goddamnit 2018 you were supposed to be better than 2017.

Only if you use Intel (99% of the market)

https://media.giphy.com/media/RHiD0K65NxxLO/giphy.gif

What an EPYC opportunity!

I'm sorry, I know where the door is.

Turnkey Linux / Bitnami, anyone using them in production?

Turnkey Linux / Bitnami, anyone using them in production?

We're a tiny team and are looking to deploy a handful of open source web apps. Being able to deploy these pre-packaged VMs (or simple to install stacks) quickly is very appealing. We don't have strong Linux skills(we're a windows shop) and this has been a deterrent in the past but the quick and easy set-up seems to minimise time cost.

We're also looking at simply paying for decent webhosting and using something like installation or solicitous to manage them but we want some of these applications to be internal only so this is not as ideal.

Has anyone had much experience using these VM appliances in the past?

Here's the fundamental problem with appliance platforms like TurnKey Linux:

The software that it packages has an array of features and settings, and configuring them correctly is complex.

In order to hide that complexity with an appliance-like wrapper, the developers of the appliance necessarily need to make decisions on your behalf on how things should be configured, based on what they think their users are most likely going to need.

Different environments are, well... different, and have their own unique configuration requirements. Since these appliances have already made their configuration decisions for you, there's going to be some cases where the appliances aren't going to be optimal (and may even catastrophically fail).

If you don't have experience configuring the software wrapped in the appliance, then you're not in a position to determine if the configuration of the appliance is suitable for your environment.

This approach is frequently used by developers doing their own operations. It lets them get off the ground quickly, and is a great choice when you just need to produce an MVP. However, it's also how you wind up accidentally exposing your databases (or other sensitive systems) to the Internet, using products that have unexpected limitations that can't easily be worked around without a redesign, or ending up with an unexpectedly high bill.

While I can give developers a pass since their focus is on building software, professional admins should know better.

Learning to manage Linux is not difficult, and in 2018, I'd expect anyone who claims to be a sysadmin to have at least a basic ability to function in a Linux environment. If your team does not, then the professional thing to do is to admit that you don't have the expertise to manage those applications, and either hire someone that does, or outsource it to a service provider that can manage it for you.

EDIT: Perfect example: many of the TurnKey Linux appliances are designed to be managed with Webmin, a web control panel interface. While this works for simple configuration tasks, it doesn't scale well, and there's configuration functionality that Webmin doesn't expose. You can drop down to the command line and configure things directly, but creating config files that have a mix of webmin-managed and hand-managed configurations is extremely error prone, and can leave your services in a broken or undefined state that's difficult to get out of if you don't know what you're doing. Lastly, Webmin has a long history of security problems.

You still need to be comfortable with the basics of working with Linux in order to configure, run, and diagnose containers.

and want to minimise time required to set some of these up.

The majority of your time and effort should be on the long term maintenance. This is something that will be harder, and take longer, if noone on your team was involved in the original setup.

I wouldn't recommend it myself. They end up being black boxes that are quite difficult to fix and modify.

The installation process is often the best time to get your head round the configuration format, logging, permissions, file structure, etc.

On-Call over Christmas and New Year

On-Call over Christmas and New Year

So I'm on-call over Christmas and New Year again, which is fine as it's always pretty quiet. That's good because I'm suffering from Flu and desperately need to just relax and hopefully recover before Monday so I don't act like Typhoid Mary to the rest of the family.

Naturally at 01:30 this morning I get woken by a raft of alerts pinging my mobile and find myself sat in bed, shivering and sweating next to my sleeping wife, while working my way through remote connections to resolve problems that really shouldn't exist.

The best part - and why I always have respect for people who work on-call - nobody will ever really know, because it's been resolved now.

So if you're also on-call over the holiday season - I wanted to raise my glass to you (except mine has hot lemon!)

Happy holidays.

The best part - and why I always have respect for people who work on-call - nobody will ever really know, because it's been resolved now.

Write up a “for your awareness” email for your boss. Just let them know what happened and the steps taken to resolve it. Not only does it give you recognition, but it keeps them in the loop of recurring or urgent issues.

I always volunteer to be on call this week because I'm Jewish and this is my Christmas gift to you guys lol

Long story but the short version is that there are only three of us on call. I was due to be on call next week and put in my notice to quit. I have no loyalty to the company but I like my colleagues and they both have young families so rather than screw up their Christmas I added a week to my notice period and am still covering it.

Yeah this should really always happen if there was something serious enough that you had to work it in the middle of the night.

Windows: Paste password into UAC prompt broken

Windows: Paste password into UAC prompt broken

With Fall Creators Update, MS managed to introduce an awful bug in the Win 10 "Run as Administrator" UAC prompt: any attempt to paste an admin password in there plain causes a variety of strange error messages (see below). Happens on both Home and Enterprise. Does anyone have a fix?

https://imgur.com/a/5Fcki

It’s simple. Make all your passwords, admin or general service accounts, short, easy to remember and type passwords.

Four letters should do it. And to make it extra easy, 2 of the letters in a row should be the same. Play along with MS’s disregard for case and just make it all lower.

root

Once all your passwords have been changed to root, you won’t need copy+paste. Hell, you might not even need UAC because now your computer isn’t even owned by you anymore.

In Secure Desktop mode you should not be able to do that. The update might have raised the uac level. I am not sure I would want a copy paste in UAC boxes just in general.

*not that i use it, but from what i read, Keeppass has away around this

more info on secure desktop mode

haha too many people didn't read to your last paragraph.

Here's your /s tag, since too many people missed it.

Who says UAC is pretty crap? This is a comment from the Vista era. Yes, exploits come out for privileged elevation but you minimize risk GREATLY by doing so.

US Government Bans Kaspersky

US Government Bans Kaspersky

I like how, in this debate, everyone is worried about a Russian company doing malware analysis in their home country, and everyone is overlooking the fact that someone took top-secret data home to further work on it on a private system.

Someone at the CIA NSA took top-secret tools home and put them on their home PC.

Kaspersky did it's job and marked it as malware

Someone at Kaspersky analyzed the tools, recognized what they were and handed them to the Russian Mob resulting a mass wave of Cryptolocker attacks.

In my experience of government IT work, it will remain on computers for years to come.

I don't think the prosecutors are overlooking that fact.

Try one of these subthreads