deeb.me

Teaching Claude Code to message my colleagues in my own voice

Fri, 06 Nov 2026 13:13:38 +0000

I'm part of a small infrastructure team, and a surprising amount of my day is short messages: "can you look at this MR", "is staging free", "did the migration finish". I already work with Claude Code most of the day, so I wanted it to send those messages for me.

The first drafts were the problem. I'd ask for a one-line question, and the agent would write five paragraphs: what the MR changes, why, which files it touches, the risks, a bulleted list of "key considerations", and a closing "let me know if you have any questions!" Every word was accurate, and nobody read it. It was too long and too technical, and it looked like AI slop, so people skimmed past it or left it for later. A question my colleagues would have answered in a minute sat there for hours.

What I needed was the message I would have typed myself: "can you check the staging plan on !885 when you get a minute? no rush".

So I wrote mattermost-dm, an agent skill and a small Python script that sends Mattermost DMs as you. Sending is the easy part. Most of the skill is about making the messages sound like the person who sends them.

A style guide for one person

The skill reads a file called voice.md before it drafts anything. It's a short style guide, written about you, for the agent. The template covers the habits that make a message recognisably yours:

Length: one line, or two or three short ones.
Capitals and punctuation: sentence case or all lowercase, a full stop at the end or not.
Asks: "can you … please?", "when you have a minute, …".
Warmth: "thanks!", a smiley now and then, or none at all.
Words you use often: "btw", "FYI".
Links and tickets: a bare URL, or a linked ticket number followed by its title.
Don't: the things you never do, like formal sign-offs, long explanations, or bold.

The template also asks for one example of a first message of the day, because a single example shows the shape faster than any list:

good morning

can you check the staging plan on !885 when you get a minute? no rush

The "Don't" list does the most work against slop. Agents add things: background nobody asked for, bullet points, bold, a summary of what they just asked, a "let me know if you have any questions". The guide says what you never write, so the agent knows what to leave out.

The skill also has a few rules of its own, whatever your voice: lead with what you need, then add one or two lines of context at most. Don't invent dates or promises, and don't use review-speak like "the part worth your eyes". Always link a ticket or MR, so the reader can click through for the detail.

Git ignores voice.md, so it stays on your machine. Without it, the skill falls back to short, friendly, plain messages, written like a note to a teammate.

Learning the voice from your own messages

Writing a style guide about yourself is hard, because you don't notice your own habits. The faster route is to let Claude Code read how you actually write.

mm-dm --history <person> --days 90 prints your DM with one person. The README suggests this prompt:

Run mm-dm --history <name> --days 90 for each of these people: <names>.
Learn only from messages I wrote. Draft voice.md from voice.example.md. Describe my habits for
length, capitals, punctuation, greetings, asks, warmth, common words and links, and what I never
do. Copy my habits, not my typos. Don't include real names, projects, clients or quoted messages.
Show me the draft before you save it.

A few lines in that prompt carry most of the weight:

Three to five people, across different kinds of work. You write differently to a close teammate than to someone in another department, and the guide should cover both.
"Copy my habits, not my typos." Without this line, the agent faithfully reproduces your mistakes.
No real names or quoted messages. The guide describes patterns with invented examples, so it doesn't turn into a copy of your private conversations.

Then read the draft and fix anything that doesn't sound like you. Ask for a few test messages and adjust the file until you'd send them without editing.

There's a trade-off to know about: --history prints both sides of each conversation, so your colleagues' messages reach your AI provider while the guide is drafted. Check your company's policy and choose conversations with nothing confidential in them. If that isn't possible, write voice.md by hand from the template.

Greetings are part of the voice

Voice isn't only wording; it's also rhythm. Nobody says "good morning" to someone they messaged ten minutes ago, and nobody skips a greeting after a week of silence.

Messages start with a {hi} placeholder, and the script fills it in from your real history. It asks Mattermost when you last posted in that DM. If you haven't spoken today, or it's been more than 4 hours, {hi} becomes a greeting. Otherwise it disappears, along with the blank line after it. In a batch, this happens separately for each person.

The greetings themselves are yours too. Add your own lists to ~/.mattermost, one for the morning and one for the afternoon:

MM_DM_GREETINGS_AM="good morning|morning :)"
MM_DM_GREETINGS_PM="hi there|hey|good afternoon"

The script picks one per person per day and keeps it, so you don't greet the same person three different ways in one morning.

Batches use {first_name}, so a question to five reviewers still reads as five personal messages:

{hi} {first_name}

can you review the runbook change before Friday's maintenance?

You still see every message

A good voice makes approval quick, but it doesn't replace it. Every message goes through a preview first: mm-dm --dry-run renders each message, greeting and name included, and Claude shows them to you. When you say yes, mm-dm --send-pending sends exactly what the preview printed. It doesn't take new text, so nothing gets reworded after you've read it. Each preview sends once, and it expires after an hour.

The token stays out of the conversation as well. The script reads it from ~/.mattermost and never prints it, and the skill tells Claude not to open that file.

Install

git clone https://github.com/abdallah/mattermost-dm.git ~/.claude/skills/mattermost-dm
touch ~/.mattermost && chmod 600 ~/.mattermost
cp ~/.claude/skills/mattermost-dm/voice.example.md ~/.claude/skills/mattermost-dm/voice.md

Add your server and a personal access token (Profile > Security > Personal Access Tokens) to ~/.mattermost:

MM_URL=https://chat.example.com
MM_TOKEN=your-token

Then edit voice.md, or ask Claude Code to draft it from your history. It needs Python 3.9 or later and has no other dependencies. If you use more than one machine, copy voice.md to each one yourself, because Git doesn't sync it.

Beyond Mattermost

The script is specific to Mattermost, but the idea isn't. Any agent that writes as you, whether it's emails, ticket comments, or commit messages, benefits from a short, private file that describes how you write and, more importantly, what you never write. Draft it from your own messages, then correct it until the test messages sound right.

The code is MIT licensed: github.com/abdallah/mattermost-dm.

I built an inbox for my Claude Code sessions. Then I never checked it.

Sun, 25 Oct 2026 15:27:12 +0000

Three weeks ago I wrote about sessionhub: one registry and one inbox for every Claude Code session on every machine I use. The hub itself took off. I have it open all day to see what's running where, and sessions register, report and move between machines without me thinking about it.

The inbox is a different story. I stopped looking at it.

Not because it broke. Because nothing in it told me what to do. A list of sessions marked "blocked" or "done" is information, not an action. If I still had to SSH into the machine to do anything about it, I'd go straight to the terminal and skip the inbox.

This post is about the changes that turned the inbox from something I could check into something I act from.

The question I couldn't answer

I went through the database to see what the inbox had actually caught. Since October 3, Claude Code had raised 171 permission prompts across my machines. 120 of them were not permissions at all. They were Claude asking me a question (the AskUserQuestion tool: "Which approach?", "Should I also update the docs?").

The hub could show those questions. It could not answer them. "Allow" from the dashboard approved the tool call and answered nothing, so the dialog stayed open in the terminal. All 120 were answered locally.

Claude Code hooks can answer a question on your behalf: return allow together with updatedInput, which is the original tool input plus an answers map from question text to the chosen label. sessionhub's permission hook was already sitting in that exact spot, waiting for a decision from the server. So the change was small:

The dashboard shows each question's options as buttons, plus an Other box.
Send answer posts the choice to the server with the decision.
The hook hands it back to Claude Code as updatedInput.

{"hookSpecificOutput": {"hookEventName": "PermissionRequest",
 "decision": {"behavior": "allow",
 "updatedInput": {"questions": [...], "answers": {"Which colour?": "Blue"}}}}}

The first live test failed. My phone tapped the old Allow button, because it was still running the page from before the deploy. So now every API response carries a fingerprint of the page, and an open tab shows a Reload bar when it changes. The server also refuses a plain allow on a question, so a stale page fails loudly instead of quietly doing nothing.

The second test worked: I tapped "Blue" on my phone and the session carried on.

Replying without opening a terminal

The other half of "act from the inbox" is answering a session that finished and asked something in its last message. Each inbox row now has a reply box. sessionhub already had a way to type a message into a session's pane when it's idle (the watcher or the Claude Code mod delivers it), so the reply box reuses that. The draft survives the page's 30-second refresh.

Inbox rows also show the start of Claude's last reply now, Markdown stripped and line breaks kept, instead of the word "Finished". That one change made rows readable at a glance.

Moving a session when its machine is off

sessionhub can move a session between machines: pack the transcript and the uncommitted work, seal it for the target, and resume it there. Yesterday I tried to move one between machines while the source was unreachable, so nothing could pack it, and the move failed. The only sign was a line under the session's details.

Two fixes:

Failed moves are an inbox group of their own, with a retry button, and they show up in the alerts.
Snapshots. Each machine's watcher snapshots its idle sessions in the background, at most every 10 minutes and only when the transcript changed. Each snapshot is sealed with X25519 for each other machine, so the server stores blobs it cannot read. If the source machine hasn't polled for 2 minutes, a move uses the latest snapshot instead of waiting. When the source comes back, it ends the leftover pane.

Snapshots are removed when the session ends or moves, after 14 days, or oldest first past 1 GiB.

The dashboard, from a human's point of view

I asked Claude Code for a UX review of the live dashboard. It took screenshots at phone and desktop width with headless Chrome through Playwright, read them, and listed what a person would trip over. The fixes:

One colour and one icon per state: ✋ needs you, 🚚 move failed, 💬 waiting, ⚠️ machine silent, ⏳ working, ✅ finished, 💤 idle.
Sorted by state, so whatever needs me is on top.
Rename a session from the web, and fall back to the task's name when the title is the default "Claude Code".
Machine headers say "online" or "silent 8m", which is how I found out one of my machines had dropped off the network.

Tasks: a standup and a place to park ideas

sessionhub turns worklog notes into a daily task list. I never used that tab. When I thought about why, I found two jobs it should be doing:

The standup I write. The Tasks tab now has a Standup button: yesterday (the last day with work, so Monday covers Friday), today, and blockers, as plain text with a Copy button. sessionhub task standup prints the same thing.
Parking ideas. I switch contexts many times a day, and ideas that arrive mid-task get lost. Now there's a "Park an idea" box, a sessionhub idea "…" command, and any Claude session will park one if I ask. Ideas wait in their own list until I start them or drop them, and they stay out of the standup.

The tab itself went from a wall of cards to one line per task: open work first, then ideas, then done.

What I learned

A dashboard you read is not a dashboard you use. Every change above removed one trip to a terminal.
Look at the data before the design. "Most prompts are questions" was not something I'd have guessed.
Test on the device you'll use. The bug that mattered was a stale page on my phone, not anything a unit test would catch.

sessionhub is open source (MIT), a single Go binary with SQLite: github.com/abdallah/session-hub. If you run Claude Code on more than one machine, I'd like to hear what you'd want from it.

One inbox for every Claude Code session on every machine

Sun, 04 Oct 2026 10:40:28 +0000

On a normal day I have somewhere between five and fifteen Claude Code sessions running. Some are on my laptop, some on a desktop, a few on a home server that never sleeps. They run inside herdr, an agent-aware terminal multiplexer, and I attach to them over SSH from whatever I'm holding at the time, which is often a phone.

That setup works well until you step away. Then three questions start nagging:

Which sessions exist right now, and which are still alive?
What has each one done, what is it doing, and is it waiting on me?
How do I get back into the one that needs me, from wherever I am?

None of the tools I already had answered all three. So I built a small thing that does. It's called sessionhub, it's open source (MIT), and this post is about why it exists, how it works, and how you can run it.

The problem: every tool knows one machine, or one session

I want to be fair to the tools I use every day, because they're good at what they do:

herdr knows every agent on its machine and whether it's working, blocked, or done. It doesn't know about my other machines, and it knows state, not meaning: "blocked" doesn't tell me blocked on what.
Claude Code's own session list is per machine too. Great for resuming on the box you're sitting at; no help for the session on the server in the other room.
Remote Control gives a single session a link I can open in the Claude app. Excellent, but I have to turn it on per session, and I first need to know which session to turn it on for.
SSH plus tmux-style multiplexers get me into a machine. Then I'm hunting through panes.

What was missing wasn't another terminal. It was a registry: one place that knows every session on every machine, what each one is working on in plain words, and what each one needs from me.

What sessionhub does

sessionhub is one static Go binary with a SQLite database. The same binary is the server and every client. You run the server on one always-on machine and "join" each machine that runs Claude Code.

From there:

Registry. Every session on every joined machine shows up with its machine, directory, branch, title, and status: live, blocked, stale, or ended.
Inbox. The sessions that need you, across all machines: blocked on a permission prompt, waiting on an answer, or finished a turn you haven't looked at.
Remote approvals. Allow or deny a waiting permission prompt from the dashboard or the CLI. The prompt still shows in the terminal, and whichever answer comes first wins. It never adds an "always allow" rule.
Progress reports. Each session gets an MCP tool to report what it has done, what's in flight, and what it's waiting on.
Get back in. sessionhub resume <id> focuses the pane, restarts the session, or prints the exact ssh command for the machine it's on. Or turn on Remote Control and continue in the Claude app.
Standing rules and messages. Add a rule once and every session sees it with every prompt. Type a message into one session, or into all of them.
Move sessions. Hand a session, with its branch and uncommitted changes, to another machine or to a Claude Code cloud session.
Start new sessions. Pick a machine and a directory on the dashboard, and a new Claude session starts there with Remote Control on.
Alerts. An optional Telegram message when a session has been blocked or waiting on you for 30 seconds.

Here's the inbox on a phone, with a session asking to run npm install:

How it fits together

There are three ways a session talks to the hub, and you can use any of them:

1. Claude Code hooks. sessionhub install-hooks adds entries to ~/.claude/settings.json (it backs the file up first, and only ever touches its own entries). Most are async, so they never slow a prompt down (paths shortened):

{
 "hooks": {
 "SessionStart": [{
 "matcher": "*",
 "hooks": [
 { "type": "command", "command": "~/.local/bin/sessionhub hook session-start", "async": true },
 { "type": "command", "command": "~/.local/bin/sessionhub hook context", "timeout": 5 }
 ]
 }],
 "UserPromptSubmit": [{ "hooks": [
 { "type": "command", "command": "~/.local/bin/sessionhub hook prompt", "async": true },
 { "type": "command", "command": "~/.local/bin/sessionhub hook context", "timeout": 5 }
 ]}]
 }
}

The context hook is synchronous on purpose: its output is how the standing rules reach the prompt. It reads a local copy of the rules and never touches the network, so a prompt never waits on the hub.

2. An MCP server. sessionhub mcp gives each session report_progress and set_title. A short snippet in ~/.claude/CLAUDE.md tells Claude when to use them:

## Reporting progress to sessionhub

- Call `report_progress` when you finish a task, when you start a task that
 will take a while, and when you are blocked waiting on me or on something
 external. Fill `done`, `in_flight`, and `waiting_on` with short one-line items.
- Call `set_title` once the session has a clear purpose, and again if the
 purpose changes.

That's what turns "blocked" into "waiting on: permission to run npm install".

3. A herdr plugin (optional). If you use herdr, its plugin registers the sessions in your panes, ends them when the pane closes, and runs a small watcher. The watcher is what lets the hub do things on a machine: type a message, turn on Remote Control, move a session, or start a new one.

The rule I cared about most: fail open

A tool that watches your agents must never become the reason they stall. So every client fails open. Hooks always exit 0. Every call from a hook or the MCP server has a two-second limit. If the server is down, writes go to a local queue (~/.local/state/sessionhub/queue.jsonl) and drain later. Claude Code and herdr never wait on sessionhub.

The server never connects to your machines

Machines poll the server; the server never reaches into them. When you tap Allow once on your phone, the PermissionRequest hook on that machine is holding a long poll open, and it picks up your answer. When you start a new session, the machine's watcher is long-polling for work and claims the request. That keeps the firewall story simple: the only open port is the hub's.

Using it

The CLI is where I spend most of my time:

$ sessionhub ls
MACHINE ID TITLE STATUS AGE REPORT
tower e5b07a1c postgres 16 upgrade plan live 1s doing: writing the rollback steps
bluebox c18e5f43 rewrite the quickstart live 1s waiting: your review of the wording
bluebox a72d4b90 dark mode for settings blocked 1s doing: settings page
tower 3f9c1e27 fix CI token rotation live 1s doing: re-running the pipeline

$ sessionhub inbox
Blocked (1)
 a72d4b90 dark mode for settings bluebox 1s asks to use Bash: npm install

Waiting on you (1)
 c18e5f43 rewrite the quickstart bluebox 1s your review of the wording

sessionhub show gives the full picture for one session, including its report history:

$ sessionhub show a72d
title: dark mode for settings
machine: bluebox
state: blocked
branch: dark-mode
resume: ssh -t bluebox '~/.local/bin/sessionhub resume a72d4b90-...'

reports (1, newest first)
 done theme tokens
 in flight settings page
 waiting on permission to run npm install

A few more I use daily:

sessionhub approve a72d # allow that npm install once (asks first)
sessionhub rules add "Never push to main." # every session sees it, on every prompt
sessionhub send --machine tower -m "Wrap up and report progress."
sessionhub move c18e bluebox # conversation, branch and uncommitted changes
sessionhub start tower --dir ~/Code/api -m "Fix the flaky test in ci.yml"

That last one is the newest feature, and it's the one I wanted most. On the dashboard it's a New session button: pick a machine and a directory (it suggests the ones you've used), optionally type a first prompt, and you get an Open in Claude link a few seconds later:

Now I can start work on my home server from a train.

Trying it in five minutes

You can run the server and a client on one machine to see what it does:

# Install the latest release (Linux or macOS, amd64 or arm64) into ~/.local/bin:
curl -fsSL https://raw.githubusercontent.com/abdallah/session-hub/main/install.sh | sh

# This machine holds the server:
mkdir -p ~/.config/sessionhub
install -m 600 /dev/null ~/.config/sessionhub/server.toml
sessionhub server &

# Join it: creates a token, installs the hooks and the MCP server.
sessionhub join localhost --name "$(hostname -s)"

# Start claude anywhere, send a prompt, then:
sessionhub ls
sessionhub login --name my-phone # a one-time sign-in link for the dashboard

For more machines, run the server on an always-on box (there's a systemd unit and a docker-compose.yml), put it behind a tunnel or reverse proxy, and sessionhub join each machine. The self-hosting guide walks through it.

A word on security

Be honest with yourself about what this server can do. A machine token or a signed-in browser can approve tool calls, add rules that every session reads, type messages into sessions, and start new ones. That's the point of the tool, and it also means a leaked credential can get Claude to run commands on your machines.

So: keep the server on a private network if you can, use HTTPS if you expose it, turn on the Telegram alerts (they also fire whenever someone adds a rule), and use the per-machine opt-outs (remote_permissions = false, remote_start = false) on machines you want to keep hands-off. SECURITY.md has a table of what each credential can do.

How it was built

I built sessionhub with Claude Code itself, which turned out to be a good test of the tool and of the process.

Every feature started as a short spec, then a plan broken into tasks, and each task had to meet a written definition of done before it counted. Two rules from that list did most of the work:

Don't invent APIs. Claude Code hook payloads and herdr events come from captured real payloads in testdata/, never from memory. The spec said it bluntly: verify against --help and the docs, and quote what you found.
Tests walk the whole path. Each one checks the start state, acts, asserts what must not have happened as well as what must, and reads the result back the way you would see it.

The early plan was to host it on Cloud Run. An always-on instance came out at about $45–50 a month, so it moved to a systemd user service on a home server behind a Cloudflare Tunnel, which costs nothing extra.

What's next

sessionhub is a personal, single-user tool: one person, their machines, no accounts or teams. That's deliberate, and I don't plan to change it. Things I do want:

a Claude Code plugin, so the hooks, MCP server and CLAUDE.md snippet install in one step;
cloud sessions reporting in, so the claude.ai/code sessions show up next to the local ones.

If you run more than a couple of Claude Code sessions across more than one machine, I'd love to hear whether this matches your workflow, and where it doesn't.

Repo: https://github.com/abdallah/session-hub (MIT, Linux and macOS binaries on the releases page)

Why I Built a New Memory Plugin for Hermes Agent

Sun, 14 Jun 2026 18:17:11 +0000

Hermes Agent already has memory, and that matters.

It keeps local context, it improves over time, and it works without forcing you into a cloud service. It also supports several external memory providers.

I still built hermes-mempalace, because none of the existing options fit my setup quite right.

I wanted something:

local-first
isolated by Hermes profile
verbatim, not just extracted facts
easy to inspect on disk
simple enough to trust over time

That last part is the important one. I did not want a memory layer that turns conversations into an opaque pile of embeddings or summaries you cannot really audit. I wanted actual transcripts, mined into a readable structure, with no hidden server in the middle.

Why the existing options were not enough

Hermes already gives you a few paths:

built-in memory and session context
external providers for different use cases
enough flexibility to adapt, if you are willing to bend your workflow around them

And to be clear, some of those options are good.

But ... I run Hermes on a headless machine at home. And I use separate profiles for different contexts. And I do not want conversation content depending on a cloud API or a separate service unless there is a very good reason.

So, the best fit had to check a few boxes:

no API key
no external server
no extra runtime I did not already want/install
storage isolated by HERMES_HOME
memory we can actually read later

That let to MemPalace, or https://mempalaceofficial.com/ (hopefully, that's the right one!)

What hermes-mempalace does

hermes-mempalace wires MemPalace into the Hermes memory provider interface.

It follows the same lifecycle as the rest of Hermes memory providers:

system_prompt_block() adds a short memory reminder to the prompt.
prefetch() can run a MemPalace search before the first model call.
sync_turn() buffers completed turns without slowing the chat loop.
on_session_end() writes buffered turns to markdown and mines them into the palace.
shutdown() flushes anything still waiting.

During a session, Hermes gets three tools:

mempalace_search
mempalace_wake_up
mempalace_status

That gives the agent a simple pattern: find relevant past context, surface it early, and keep the full record available if you need to inspect it.

Storage is explicit

The plugin stores data under the current Hermes profile, not in some shared global bucket.

$HERMES_HOME/mempalace/palace/ # palace index
$HERMES_HOME/mempalace/conversations/ # mined transcripts as markdown

That was important to mw. Profile isolation should be structural, not something you hope the config handles correctly.

Install

If you already use Hermes:

`
hermes memory setup

Install the MemPalace CLI:

uv tool install mempalace

Then verify everything is in place:

hermes memory status
hermes plugins list

What it feels like in practice

The main difference is not flashy.

It is that mempalace_wake_up gives Hermes useful context at the start of a session without turning the whole memory layer into a black box.

You can inspect the transcripts. You can search them. You can open the directory and see what happened. If you want to prune or review something, you are not fighting the system.

That is the part I cared about most.

Hermes still keeps its built-in memory. hermes-mempalace does not replace that. It adds an archival layer on top of it. One that feels a lot more like a real notebook than a hidden model feature.

If this sounds useful... let me know :)

If you are already happy with Hermes memory, you probably do not need this.

But if you want:

local-first memory
profile-scoped storage
verbatim transcript mining
a readable on-disk layout
no extra server to babysit

then hermes-mempalace may be a good fit.

The plugin is MIT licensed, and the repo is here:

https://github.com/abdallah/hermes-mempalace

Quick WHM/cPanel All email passwords reset

Fri, 07 May 2021 08:42:14 +0000

You generally don’t want to use this script. This is a last resort, when all mailbox passwords for the entire account need to be changed. For this script to work, you need root access to your WHM/cPanel server. This could ruin everything, be warned!

Here be dragons!

#!/bin/bash
newpass='SomeStr0ngPasswordGoesHere!'
accountname='example'
domainname='example.com'

uapi --user=$accountname Email list_pops \
 | grep email \
 | awk '{print$2}' | awk -F\@ '{print$1}' | \
 while read m; do
 echo changing password for $m
 uapi --user=@accountname Email passwd_pop domain=$domainname email=$m password=$newpass
 done
echo done

Here are the links to the official API docs:

Is there a better way to do this? probably

Can I randomize the password? sure, but how will you send the new passwords to your users?

Hope it helps!

Quick ElasticSearch Index Cleanup

Wed, 07 Apr 2021 08:50:38 +0000

There are many solutions using curator and other libraries. I’ll try to add links here later (if I remember)

However, most of the time, all that’s needed is a quick curl call

curl -X DELETE "https://${ES-Endpoint}/${INDEX_NAME}"

Get the ES endpoint from the domain overview

List the index names using

curl "https://${ES-Endpoint}/_cat/indices"

So for a quick cleanup session for everything that matches a grep

curl "https://${ES-Endpoint}/_cat/indices" | grep 'something' | awk '{print$3}' \
while read INDEX_NAME; do
 echo "Deleting ${INDEX_NAME}"
 curl -X DELETE "https://${ES-Endpoint}/${INDEX_NAME}"
done

My Ansible AWS EC2 Dynamic Inventory

Tue, 11 Aug 2020 11:22:35 +0000

Start with the the Ansible configuration. This can be set in /etc/ansible/ansible.cfg or ~/.ansible.cfg (in the home directory) or ansible.cfg (in the current directory)

My suggestion is use one of the first 2 (ie. /etc/ or ~/.ansible.cfg if you’re going to be managing instances from your machine. Update the configuration as needed.

[defaults]
inventory = ./ansible_plugins
enable_plugins = aws_ec2
host_key_checking = False
pipelining = True
log_path = /var/log/ansible

You may need other plugins, this one is for aws_ec2. In the /etc/ansible/ansible_plugins directory, create the *_aws_ec2.yml configuration file for your inventory

# /etc/ansible/ansible_plugins/testing_aws_ec2.yml
---
plugin: aws_ec2
aws_profile: testing
regions:
 - us-east-1
 - us-east-2
filters:
 tag:Team: testing
 instance-state-name : running
hostnames:
 - instance-id
 - dns-name
keyed_groups:
 - prefix: team
 key: tags['Team']

You'll notice, I’m filtering using a tag:Team == testing and showing only running instances.

I’m also using the instance-id and dns-name attributes as hostname

And I’m using the tag['Team'] as a grouping.

So now, I can do the following from any directory (since my configuration is global in /etc/ansible)

$ ansible-inventory --list --yaml
all:
 children:
 aws_ec2:
 hosts:
 i-xxxxxxxxxxxxxxx:
 ami_launch_index: 0
 architecture: x86_64
 block_device_mappings:
 - device_name: /dev/sda1
 ebs:
 attach_time: 2020-08-10 15:20:58+00:00
 delete_on_termination: true
 status: attached
 volume_id: vol-xxxxxxxxxxxxxx
...
 team_testing:
 hosts:
 i-xyxyxyxyxyyxyxyy: {}
 i-xyxyxy2321yxyxyy: {}
 i-xyxyxyxyxy89yxyy: {}
 i-xyxy1210xyyxyxyy: {}
 i-xyxy999999yxyxyy: {}
 i-xyxyxy44xyyxyxyy: {}
 i-xyx2323yxyyxyxyy: {}
 i-xyxyxyxyxy9977yy: {}
 ungrouped: {}

I can also use the team_testing or the individual instance_id in my Ansible hosts calls.

Get daily cost alerts on AWS

Thu, 18 Jun 2020 15:42:00 +0000

So I wanted to have a better alarm system for when AWS hits us with unexpected costs. It’s better to know there’s something wrong rather quickly and not suffer hundreds of dollars costs for something you don’t really need or want.

The AWS provided alarm checks for hikes on a monthly basis. Here’s the doc they published. So that’s an alarm that sounds when your estimated bill is going to be higher than the budgeted amount, or what you had in mind in the first place. Not very useful honestly in our case. It will just be too late.

The only alternative I found was creating a daily check, that will compare yesterday’s costs against a max_amount set by default. Let’s say you want to have your daily bill not exceed 5$US.

For ease of use and maintainability, I’m using a lambda function triggered by a cron (EventBridge rule) for the daily checks. And I’m sending the Alarm using an SNS topic, this way I can subscribe to it by email, or send it to our Mattermost channel, etc.

Here’s the code:

import os
import json
import boto3
from datetime import datetime, timedelta


def lambda_handler(event, context):
 yesterday = datetime.strftime(datetime.now() - timedelta(1), '%Y-%m-%d')
 twodaysago = datetime.strftime(datetime.now() - timedelta(2), '%Y-%m-%d')
 cost_metric = os.environ.get('cost_metric')
 max_amount = os.environ.get('max_amount')
 
 ce = boto3.client('ce')
 sns = boto3.client('sns')
 
 result = ce.get_cost_and_usage(
 TimePeriod={'Start': twodaysago, 'End': yesterday}, 
 Granularity='DAILY', 
 Metrics=[cost_metric]
 )
 total_amount = result['ResultsByTime'][0].get('Total').get(cost_metric).get('Amount')
 
 if total_amount > max_amount:
 sns_topic = sns.create_topic(Name='BillingAlert')
 sns.publish(
 TopicArn=sns_topic.get('TopicArn'), 
 Message='Total cost "{} USD" exceeded max_amount rate: {}'.format(total_amount, max) 
 )
 
 return {
 'statusCode': 200,
 'body': json.dumps('cost check: {}'.format(total_amount))
 }

Note that you will need to add a couple of environment variables to Lambda: cost_metric and max_amount
And the following permissions to the role used by the lambda function: ce:GetCostAndUsage, sns:Publish and sns:CreateTopic

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "VisualEditor0",
 "Effect": "Allow",
 "Action": "ce:GetCostAndUsage",
 "Resource": "*"
 },
 {
 "Sid": "VisualEditor1",
 "Effect": "Allow",
 "Action": [
 "logs:CreateLogStream",
 "sns:Publish",
 "sns:CreateTopic",
 "logs:PutLogEvents"
 ],
 "Resource": [
 "arn:aws:logs:us-east-1:*:log-group:/aws/lambda/checkDailyCosts:*",
 "arn:aws:sns:*:*:*"
 ]
 },
 {
 "Sid": "VisualEditor2",
 "Effect": "Allow",
 "Action": "logs:CreateLogGroup",
 "Resource": "arn:aws:logs:us-east-1:*:*"
 }
 ]
}

After that’s set up, go to your SNS topic (created by Lambda if it doesn’t exist) and subscribe to it. There you go, daily checks and an alarm if the bill is higher than expected.

Sync S3 objects across AWS account?

Tue, 12 May 2020 09:42:00 +0000

How do I copy S3 objects/files/etc from another AWS account? How do I migrate an S3 bucket to another AWS account? [playbook below + extra preliminary step, create the bucket]

Well, you can do it manually following the notes in the AWS KB article I linked at the end of the article. Honestly though, I am always confused by those KB articles. They are concise ans usually well written. I guess it's my dislexya ;) So I have to read them a few times before I’m able to apply the steps correctly. That's not a bad thing really ... better than blindly copy/paste commands (which I NEVER DO, right).

Recently, I’ve been going the other way around: whenever there are steps to follow, I try to translate them into an Ansible playbook. This way I make sure I got it right, AND I get to reuse the playbook, which makes things a lot faster the second time around. Oh and as an added bonus, I can share the playbook and have someone else take care of the task!

Enough introductions, here’s what you came here for: an ansible playbook for syncing files between S3 buckets in different accounts:

# copy-bucket.yml
- hosts: localhost
 vars:
 source_bucket: my-source-bucket 
 dest_bucket: my-dest-bucket
 dest_user_arn: arn:aws:iam::ACCOUNTID:user/USERNAME
 dest_user_name: USERNAME
 source_profile: src_profile
 dest_profile: dest_profile
 tasks:
 - name: Attach bucket policy to source bucket
 s3_bucket:
 name: "{{ source_bucket }}"
 policy: "{{ lookup('template','bucket-policy.json.j2') }}"
 profile: "{{ source_profile }}"

 - name: Attach an IAM policy to a user in dest account
 iam_policy:
 iam_type: user
 iam_name: "{{ dest_user_name }}"
 policy_name: "s3_move_access"
 state: present
 policy_json: "{{ lookup( 'template', 'user-policy.json.j2') }}"
 profile: "{{ dest_profile }}"
 register: user_policy

 - name: Sync the files 
 shell: aws s3 sync s3://{{ source_bucket }}/ s3://{{ dest_bucket }}/ --profile {{ dest_profile }}

You will also need the following json templates

bucket-policy.json.j2

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "s3:ListBucket",
 "s3:GetObject"
 ],
 "Resource": [
 "arn:aws:s3:::{{ source_bucket }}",
 "arn:aws:s3:::{{ source_bucket }}/*"
 ]
 },
 {
 "Effect": "Allow",
 "Action": [
 "s3:ListBucket",
 "s3:PutObject",
 "s3:PutObjectAcl"
 ],
 "Resource": [
 "arn:aws:s3:::{{ dest_bucket }}",
 "arn:aws:s3:::{{ dest_bucket }}/*"
 ]
 }
 ]
}

user-policy.json.j2

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "DelegateS3Access",
 "Effect": "Allow",
 "Principal": {"AWS": "{{ dest_user_arn }}"},
 "Action": ["s3:ListBucket","s3:GetObject"],
 "Resource": [
 "arn:aws:s3:::{{ source_bucket }}/*",
 "arn:aws:s3:::{{ source_bucket }}"
 ]
 }
 ]
}

run: ansible-playbook -v copy-bucket.yml

make sure the profile names are setup on your machine correctly, and the IAM user is there.

Ref: AWS KB: copy-s3-objects-account

ChatOps with Mattermost and AWS Lambda

Fri, 14 Jun 2019 18:09:07 +0000

I’ve been working towards making things simpler when managing distributed resources at work. And since we spend most of our day in the chat room (was Slack, now Mattermost) I thought it’s best to get started with ChatOps

It’s just a fancy word for doing stuff right from the chat window. And there’s so much one can do, especially with simple Slash Commands.

Here’s a lambda function I setup yesterday for invalidating CloudFront distributions.

from time import time
import boto3
 
import json
import os
import re
 
 
EXPECTED_TOKEN = os.environ['mmToken']
ALLOWED_USERS = re.split('[, ]', os.environ['allowedUsers'])
DISTRIBUTIONS = {
 'site-name': 'DISTRIBUTIONID',
 'another.site': 'DISTRIBUTIONID05'
}
 
def parse_command_text(command_text):
 pattern = r"({})\s+(.*)".format('|'.join(DISTRIBUTIONS.keys()))
 m = re.match(pattern, command_text)
 if m:
 return { 'site': m.group(1), 'path': path}
 else: 
 return False
 
def lambda_handler(event, context):
 # Parse the request
 try:
 request_data = event["queryStringParameters"]
 except:
 return {
 "statusCode": 400,
 "headers": {"Content-Type": "application/json"},
 "body": '{ "message": "Use GET for setting up mattermost slash command" }'
 }
 
 # Check the token matches.
 if request_data.get("token", "") != EXPECTED_TOKEN:
 print('Wrong Token!')
 return {
 "statusCode": 401,
 "headers": {"Content-Type": "application/json"},
 "body": '{ "message": "Mattermost token does not match" }'
 }
 
 # Check the user is allowed to run the command
 if request_data.get("user_name", "") not in ALLOWED_USERS:
 print('Wrong User! {} not in {}'.format(request_data['user_name'], ALLOWED_USERS))
 return {
 "statusCode": 401,
 "headers": {"Content-Type": "application/json"},
 "body": '{ "message": "User not allowed to perform action" }'
 }
 
 # parse the command
 command_text = request_data.get("text", "")
 if not command_text:
 print('Nothing to do, bailing out')
 return {
 "statusCode": 404,
 "headers": {"Content-Type": "application/json"},
 "body": '{ "message": "No command text sent" }'
 }
 parts = parse_command_text(command_text)
 if not parts: 
 print('Bad formatting - command: {}'.format(command_text))
 return {
 "statusCode": 402,
 "headers": {"Content-Type": "application/json"},
 "body": '{ "message": "Wrong pattern" }'
 }
 
 
 # Do the actual work
 cf_client = boto3.client('cloudfront')
 
 # Invalidate
 boto_response = cf_client.create_invalidation(
 DistributionId=DISTRIBUTIONS[parts['site']],
 InvalidationBatch={
 'Paths': {
 'Quantity': len(parts['path']),
 'Items': parts['path'] 
 },
 'CallerReference': str(time()).replace(".", "")
 }
 )['Invalidation']
 
 # Build the response message text.
 text = """##### Executing invalidation
| Key | Info |
| --- | ---- |
| Site | {} |
| Path | {} |
| ID | {} |
| Status | {} |""".format(
 parts['site'], 
 parts['path'],
 boto_response['Id'], 
 boto_response['Status']
 )
 
 # Build the response object.
 response = {
 "response_type": "in_channel",
 "text": text,
 }
 
 # Return the response as JSON
 return {
 "body": json.dumps(response),
 "headers": {"Content-Type": "application/json"},
 "statusCode": 200,
 }

Note that you need to hook that up with an API Gateway in AWS. Once that’s done, you will have a URL endpoint ready for deployment.

Next, I created the slash command in mattermost with the following:

slash command configuration

That’s pretty much it. Rinse and repeat for a different command, different usage.

On my list next is to have more interaction with the user in mattermost per https://docs.mattermost.com/developer/interactive-messages.html
Weekend Project, Yay!

MFA from command line

Thu, 02 May 2019 18:00:33 +0000

MFA, 2FA, 2 step validation, etc. are everywhere these days. And it’s a good thing.

Problem with using the phone to get the authentication code is that you need to have it handy at all times (when you want to login at least) and that you have to read the code then type it in (too many steps)

One possible alternative is to use the command line oathtool

Here’s my snippet, I added the following line in my .bashrc

function mfa () { oathtool --base32 --totp "$(cat ~/.mfa/$1.mfa)" | xclip -sel clip ;}

Some preparation work:

sudo apt install oathtool xclip<br>mkdir ~/.mfa<br>

If you want to use the same account with both a phone based MFA generator and the shell, set them up at the same time. Simply use the generated string for setting up the account in the Google Authenticator (as an example) and the add it to the ~/.mfa/account_name.mfa

The use of xclip automatically copies the 6 digit authentication code to the clipboard. You can go ahead and paste it.

The above setup works in Ubuntu. Didn’t try it on other systems.

VaultPress SSH Access to Site Behind AWS Load Balancer

Thu, 21 Mar 2019 13:32:35 +0000

I recently worked on migrating a large WordPress site from a dedicated server to AWS following the reference architecture as outlined in https://github.com/aws-samples/aws-refarch-wordpress

One of the upsides of this architecture is that the site is entirely behind the load balancer and the instances running the web server are never accessible from the internet (not even via ssh or http). So any updates to the site need to be deployed using Systems Manager. I will post about that later … However due to this restriction, VaultPress is only able to access the site (for backups) over https, which causes extra load on the web instances.

To fix that, we need to allow VaultPress SSH access directly to the web instances (at least to 1 of them). And the only way in via SSH is through a bastion instance.

On this bastion instance, I installed Nginx with the following configuration in /etc/nginx/nginx.conf

stream {
 server {
 listen 1234;
 proxy_pass 10.0.1.12:22;
 }
}

where 10.0.1.12 is the internal IP of one web instances behind the load balancer.

At this stage (after reloading Nginx), I added the vaultpress user to the web instances and set their SSH public key in the .ssh/authorized_keys for that user.

To test manually, I added my own public key and connected to:

ssh -l vaultpress -p 1234 bastion.mydomain.com

After that works, I tested from VaultPress dashboard. No surprises: testing the connection works (SSH/SFTP), all green! Great, both the SSH and the SFTP setups should work properly.

<shrug>One issue I faced here was that after a few hours/the next day, VaultPress somehow forgot the setup and I had to re-add the connection information. Why? No idea! Just make sure to check on it the next day to make sure it's still there! </shrug>

Since the IPs may change in case there’s an auto scaling event, I added a cron job that runs every 5 minutes on the bastion and updates the nginx configuration to point to one of the web instances internal IPs. A simplistic version looks like this:

#!/bin/bash
IP="$(list-ips.sh | awk '{ print $1 }')"
sed -i -E "s/proxy_pass (.+):22/proxy_pass ${IP}:22/" /etc/nginx/nginx.conf
nginx -t && service nginx reload

You can also use CloudWatch Events to add a rule, that runs when AutoScaling happens. Pointing that to a Lambda function that calls Systems Manager to change the IP in Nginx. Here’s a reference for that https://docs.aws.amazon.com/autoscaling/ec2/userguide/cloud-watch-events.html#create-lambda-function

Oh, and don’t forget to lock down the bastion instance. In my case, I set the Security group in the EC2 console to allow only the following:

Type: Custom TCP Rule
Protocol: TCP
Port Range: 1234
Source: 192.0.64.0/18
Description: VaultPress SSH Access

Hope it helps! If you have any thoughts on improving the setup, I'm open for discussions.

List IPs from CloudTrail events

Fri, 01 Feb 2019 16:17:54 +0000

A quick command to list the IPs from AWS CloudTrail events.

#!/bin/bash ACCESS\_KEY\_ID=AKIASMOETHINGHERE MAX\_ITEMS=100 aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=${ACCESS\_KEY\_ID} --max-items ${MAX\_ITEMS} \ | jq -r '.Events[].CloudTrailEvent' \ | jq '.sourceIPAddress' \ | sort | uniq

This of course can be extended to include more information, for example:

#!/bin/bash ACCESS\_KEY\_ID=AKIASMOETHINGHERE MAX\_ITEMS=100 aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=${ACCESS\_KEY\_ID} --max-items ${MAX\_ITEMS} \ | jq -r '.Events[].CloudTrailEvent' \ | jq '{ User: .userIdentity.userName, IP: .sourceIPAddress, Event: .eventName }'

Change profiles automatically in Tilix when connecting to SSH hosts

Wed, 16 Jan 2019 16:57:46 +0000

I use Tilix as my main terminal app on Ubuntu. I like it mainly because it’s easy to use and shows multiple sessions in tiles and tabs, so it’s easy to switch between multiple servers without too much fuss

Tilix is also pretty customizable. I just needed to have the “profile” automatically switch according to the server/machine I switch to. This can be done according to the documentation per https://gnunn1.github.io/tilix-web/manual/profileswitch/ but I wanted something configured locally, not sent to the remote system.

So I just added a function that wraps my ssh command. Here’s how it looks in my .bashrc

ssh() { SSHAPP=`which ssh`; ARGS=$@; echo "switching to $ARGS"; printf "\033]7;file://%s/\007" "$ARGS"; $SSHAPP $ARGS; }

This sets up the hostname of machine you’re logging into as the title. That’s your trigger. What remains is to create a profile and assign that profile to automatically switch when the trigger (hostname:directory, user can also be added) is set.

Go to Profiles > Edit Profile > Advanced (tab) and under Match add the hostname.

That’s about it. I’m going to add a new profile with a RED background now for my production machines! too much?

packet_write_wait: Connection to xxx port 22: Broken pipe

Tue, 15 Jan 2019 20:33:56 +0000

SSH connections started dropping left and right a few days ago. I thought at first it was a problem with my connection, our DSL connection in Beirut, Lebanon is getting a lot better this last year. But it still had its quirks now and then and I blamed the wires and bad weather.

But it happened again the second day and the third, and it was becoming really annoying with jobs killed in the middle of execution when I forget to start a screen.

Long story short, it seems that some settings were removed or reset after an upgrade in Ubuntu on my home machine. This is what I added in /etc/ssh/ssh_config (not sshd_config!)

Host \* ServerAliveInterval 30 ServerAliveCountMax 5

And that’s it!

If you don’t want to edit the system-wide configuration, you can always edit ~/.ssh/config with the same for similar effect.

Handling AWS Simple Email Service notifications

Wed, 27 Jun 2018 19:06:51 +0000

Apparently it’s not enough to gather information on your email sending, bounces and complaints. You also need to act when there’s a problem otherwise Amazon will put your account under probation and might even shut it completely if you don’t fix the problem.

And it’s pretty easy to implement per their sample code and the explanation in the aws blog

Here are the steps I use to set it up:

Create an SNS topic. Let’s call it SES-BOUNCES. Or create 2 SNS topics (1 for bounces, another for complaints). I will go with just one.
Create a service on your site to handle the bounces. I used some simple PHP code (pasted below) for that.
Create a subscription in your SNS topic (created first)

The service should handle 2 things:

the subscription (that will happen once!): call the SubscribeURL and you’re done.
the bounces/complaints: Remove the email addresses form your mailing list, or blacklist them, or … The main idea is to stop sending emails to those who complain. And stop trying to send emails to addresses that bounce.

Here’s the code for the PHP service, it’s pretty straight forward

if ( 'POST' !== $\_SERVER['REQUEST\_METHOD'] ) { http\_response\_code( 405 ); die; } require 'vendor/autoload.php'; use Aws\Sns\Message; use Aws\Sns\MessageValidator; function parse\_bounce( $message ) { $recipients = array(); foreach ( $message['bounce']['bouncedRecipients'] as $recipient ) { $recipients[] = $recipient['emailAddress']; } switch ( $message['bounce']['bounceType'] ) { case 'Transient': error\_log( "BouncesController - Soft bounces occurred for " . join( ', ', $recipients ) ); return array(); break; case 'Permanent': error\_log( "BouncesController - Hard bounces occurred for " . join( ', ', $recipients ) ); return $recipients; break; } } function parse\_complaint( $message ) { $recipients = array(); foreach ( $message['complaint']['complainedRecipients'] as $recipient ) { $recipients[] = $recipient['emailAddress']; } return $recipients; } function blacklist\_recipients( $recipients ) { foreach ( $recipients as $email\_address ) { // blacklist those emails } } try { $sns\_message = Message::fromRawPostData(); $validator = new MessageValidator(); $validator-\>validate( $sns\_message ); if ( $validator-\>isValid( $sns\_message ) ) { if ( in\_array( $sns\_message['Type'], ['SubscriptionConfirmation', 'UnsubscribeConfirmation'] ) ) { file\_get\_contents( $sns\_message['SubscribeURL'] ); error\_log( 'Subscribed to ' . $sns\_message['SubscribeURL'] ); } if ( $sns\_message['Type'] == 'Notification' ) { $message = $sns\_message['Message']; $notification\_type = $message['notificationType']; if ( $notification\_type == 'Bounce' ) { blacklist\_recipients( parse\_bounce( $message ) ); } if ( $notification\_type == 'Complaint' ) { blacklist\_recipients( parse\_complaint( $message ) ); } } } } catch ( Exception $e ) { error\_log( 'Error: ' . $e-\>getMessage() ); http\_response\_code( 404 ); die; }

Don’t forget to run composer to get those dependencies:

{ "require": { "aws/aws-sdk-php": "3.\*", "aws/aws-php-sns-message-validator": "1.4.0" } }

And make sure you don’t send unsolicited emails! it’s just not nice.

Stop un-tagged instances

Thu, 03 May 2018 15:33:17 +0000

After setting up a bastion and getting GitLab runner to autoscale on EC2 spot instances, I noticed that some instances are being started but left un-tagged and probably unused. Those seem to slip through the cracks somehow .. I’m still investigating why that’s happening.
Meanwhile, to avoid paying for those instances, I set up a cron to check for non tagged instances on EC2 and terminate them.

Here’s my bash code using aws-cli

#!/bin/bash INSTANCES=$(aws ec2 describe-instances \ --filters "Name=key-name,Values=runner-\*" \ "Name=instance-state-name,Values=running" \ --query 'Reservations[].Instances[?!not\_null(Tags[?Key == `Name`].Value)] | [].[InstanceId]' \ --output text) if [-n "$INSTANCES"]; then aws ec2 terminate-instances --instance-ids $INSTANCES fi

This looks for running instances, with the Key Name runner-* and where the tag Name is not not_null (so null!)

It’s working so far. Will keep on looking for a more permanent solution.

35.5138298
24.01803670000004